model benchmark
Classical and machine-learned forecasts of aftershock counts, scored on identical forward windows (origins from 1 to 240 days, horizons of 1, 7 and 30 days). The reference is the Omori–Utsu law; every score is a per-event difference in log predictive probability.
pooled over sequences
14 sequences, each held out in turn. Intervals resample whole sequences, the unit the generalisation question is about.
modelestimate [95% interval]
ETAS (fixed μ, subcritical)14 of 14 sequences above Omori · 612 windows+0.171 [+0.096, +0.307]
ETAS (free)14 of 14 sequences above Omori · 612 windows+0.138 [+0.073, +0.253]
GBM · cross/pooled13 of 14 sequences above Omori · 615 windows+0.130 [+0.055, +0.256]
Poisson GLM · cross/pooled13 of 14 sequences above Omori · 615 windows+0.129 [+0.048, +0.270]
Poisson MLP · cross/pooled8 of 14 sequences above Omori · 615 windows+0.059 [−0.061, +0.201]
Omori–Utsu + NB spread14 of 14 sequences above Omori · 615 windows+0.019 [+0.012, +0.030]
GBM-hybrid · cross/pooled + NB9 of 14 sequences above Omori · 615 windows+0.013 [−0.029, +0.049]
GBM-hybrid · cross/chrono7 of 13 sequences above Omori · 570 windows−0.003 [−0.043, +0.036]
GBM-hybrid · cross/pooled9 of 14 sequences above Omori · 615 windows−0.007 [−0.043, +0.024]
GBM-hybrid · within + NB5 of 14 sequences above Omori · 405 windows−0.008 [−0.032, +0.020]
GBM-hybrid · cross/pooled+within8 of 14 sequences above Omori · 615 windows−0.010 [−0.047, +0.021]
GBM-hybrid · cross/other_region7 of 14 sequences above Omori · 615 windows−0.013 [−0.055, +0.018]
GBM-hybrid · within2 of 14 sequences above Omori · 405 windows−0.056 [−0.078, −0.030]
All 38 model variants
modelestimate [95% interval]
ETAS (fixed μ, subcritical)14 of 14 sequences above Omori · 612 windows+0.171 [+0.096, +0.307]
GBM · cross/pooled + NB13 of 14 sequences above Omori · 615 windows+0.147 [+0.075, +0.283]
Poisson GLM · cross/pooled + NB13 of 14 sequences above Omori · 615 windows+0.145 [+0.065, +0.274]
GBM · cross/chrono + NB12 of 13 sequences above Omori · 570 windows+0.139 [+0.045, +0.293]
ETAS (free)14 of 14 sequences above Omori · 612 windows+0.138 [+0.073, +0.253]
GBM · cross/other_region + NB13 of 14 sequences above Omori · 615 windows+0.135 [+0.060, +0.285]
GBM · cross/pooled13 of 14 sequences above Omori · 615 windows+0.130 [+0.055, +0.256]
Poisson GLM · cross/pooled13 of 14 sequences above Omori · 615 windows+0.129 [+0.048, +0.270]
GBM · cross/chrono12 of 13 sequences above Omori · 570 windows+0.116 [+0.010, +0.272]
GBM · cross/other_region13 of 14 sequences above Omori · 615 windows+0.115 [+0.035, +0.261]
Poisson MLP · cross/pooled + NB10 of 14 sequences above Omori · 615 windows+0.102 [+0.006, +0.241]
Poisson GLM · cross/chrono + NB10 of 13 sequences above Omori · 570 windows+0.087 [−0.047, +0.258]
Poisson MLP · cross/pooled8 of 14 sequences above Omori · 615 windows+0.059 [−0.061, +0.201]
Poisson MLP · cross/chrono + NB8 of 13 sequences above Omori · 570 windows+0.057 [−0.071, +0.228]
Poisson MLP · cross/other_region + NB7 of 14 sequences above Omori · 615 windows+0.055 [−0.045, +0.213]
Poisson GLM · cross/chrono9 of 13 sequences above Omori · 570 windows+0.050 [−0.108, +0.241]
GBM · within + NB9 of 14 sequences above Omori · 405 windows+0.022 [−0.000, +0.045]
GBM-hybrid · cross/chrono + NB8 of 13 sequences above Omori · 570 windows+0.019 [−0.030, +0.067]
Omori–Utsu + NB spread14 of 14 sequences above Omori · 615 windows+0.019 [+0.012, +0.030]
GBM-hybrid · cross/other_region + NB9 of 14 sequences above Omori · 615 windows+0.016 [−0.024, +0.047]
GBM-hybrid · cross/pooled + NB9 of 14 sequences above Omori · 615 windows+0.013 [−0.029, +0.049]
GBM-hybrid · cross/pooled+within + NB8 of 14 sequences above Omori · 615 windows+0.012 [−0.030, +0.049]
Poisson GLM · within + NB7 of 14 sequences above Omori · 405 windows+0.004 [−0.019, +0.029]
Poisson MLP · cross/other_region4 of 14 sequences above Omori · 615 windows+0.001 [−0.107, +0.170]
GBM-hybrid · cross/chrono7 of 13 sequences above Omori · 570 windows−0.003 [−0.043, +0.036]
GBM-hybrid · cross/pooled9 of 14 sequences above Omori · 615 windows−0.007 [−0.043, +0.024]
GBM-hybrid · within + NB5 of 14 sequences above Omori · 405 windows−0.008 [−0.032, +0.020]
GBM-hybrid · cross/pooled+within8 of 14 sequences above Omori · 615 windows−0.010 [−0.047, +0.021]
GBM-hybrid · cross/other_region7 of 14 sequences above Omori · 615 windows−0.013 [−0.055, +0.018]
GBM · within3 of 14 sequences above Omori · 405 windows−0.023 [−0.043, −0.001]
Poisson MLP · cross/chrono5 of 13 sequences above Omori · 570 windows−0.029 [−0.212, +0.174]
Poisson GLM · within3 of 14 sequences above Omori · 405 windows−0.042 [−0.067, −0.017]
GBM-hybrid · within2 of 14 sequences above Omori · 405 windows−0.056 [−0.078, −0.030]
Poisson GLM · cross/other_region + NB8 of 14 sequences above Omori · 615 windows−0.406 [−1.366, +0.189]
Poisson GLM · cross/other_region7 of 14 sequences above Omori · 615 windows−0.744 [−2.289, +0.126]
1-day windows
modelestimate [95% interval]
ETAS (fixed μ, subcritical)9 of 14 sequences above Omori · 205 windows+0.017 [+0.005, +0.027]
Omori–Utsu + NB spread11 of 14 sequences above Omori · 206 windows+0.005 [+0.002, +0.010]
GBM-hybrid · within7 of 14 sequences above Omori · 136 windows−0.007 [−0.026, +0.007]
GBM-hybrid · cross/pooled3 of 14 sequences above Omori · 206 windows−0.020 [−0.036, −0.009]
Poisson MLP · cross/pooled1 of 14 sequences above Omori · 206 windows−0.076 [−0.126, −0.038]
7-day windows
modelestimate [95% interval]
ETAS (fixed μ, subcritical)14 of 14 sequences above Omori · 204 windows+0.095 [+0.062, +0.146]
Omori–Utsu + NB spread14 of 14 sequences above Omori · 205 windows+0.012 [+0.006, +0.019]
GBM-hybrid · cross/pooled9 of 14 sequences above Omori · 205 windows−0.001 [−0.025, +0.020]
Poisson MLP · cross/pooled7 of 14 sequences above Omori · 205 windows−0.003 [−0.087, +0.076]
GBM-hybrid · within3 of 14 sequences above Omori · 135 windows−0.026 [−0.043, −0.008]
30-day windows
modelestimate [95% interval]
ETAS (fixed μ, subcritical)14 of 14 sequences above Omori · 203 windows+0.223 [+0.122, +0.406]
Poisson MLP · cross/pooled8 of 14 sequences above Omori · 204 windows+0.103 [−0.058, +0.315]
Omori–Utsu + NB spread14 of 14 sequences above Omori · 204 windows+0.024 [+0.014, +0.038]
GBM-hybrid · cross/pooled8 of 14 sequences above Omori · 204 windows−0.008 [−0.053, +0.029]
GBM-hybrid · within2 of 14 sequences above Omori · 134 windows−0.069 [−0.102, −0.037]
same spread, better mean?
Clustered counts are over-dispersed, so any Poisson forecast gains simply by widening its intervals. Here the reference is Omori–Utsu with the same negative-binomial spread fitted to its own track record, and the learned models carry the same treatment. What remains above zero is information in the mean forecast itself.
modelestimate [95% interval]
ETAS (fixed μ, subcritical)14 of 14 sequences above Omori + NB · 612 windows+0.152 [+0.080, +0.277]
GBM · cross/pooled + NB13 of 14 sequences above Omori + NB · 615 windows+0.128 [+0.059, +0.256]
Poisson GLM · cross/pooled + NB13 of 14 sequences above Omori + NB · 615 windows+0.126 [+0.050, +0.246]
ETAS (free)14 of 14 sequences above Omori + NB · 612 windows+0.119 [+0.056, +0.225]
Poisson MLP · cross/pooled + NB10 of 14 sequences above Omori + NB · 615 windows+0.082 [−0.009, +0.212]
GBM-hybrid · cross/chrono + NB8 of 13 sequences above Omori + NB · 570 windows+0.000 [−0.044, +0.039]
GBM-hybrid · cross/other_region + NB7 of 14 sequences above Omori + NB · 615 windows−0.003 [−0.039, +0.024]
GBM-hybrid · cross/pooled + NB9 of 14 sequences above Omori + NB · 615 windows−0.006 [−0.044, +0.025]
GBM-hybrid · cross/pooled+within + NB7 of 14 sequences above Omori + NB · 615 windows−0.007 [−0.047, +0.023]
GBM-hybrid · within + NB0 of 14 sequences above Omori + NB · 405 windows−0.058 [−0.094, −0.027]
per sequence
Information gain over Omori for each held-out sequence. Stability is the question: a method that wins on average but loses on most sequences is not the better method.
| sequence | region | ETAS (fixed μ, subcritical) | Omori–Utsu + NB spread | GBM-hybrid · within | GBM-hybrid · cross/pooled | Poisson MLP · cross/pooled | GBM-hybrid · cross/other_region |
|---|---|---|---|---|---|---|---|
| Landers 1992 | California | +0.135 | +0.016 | −0.019 | −0.016 | +0.094 | +0.004 |
| Hector Mine 1999 | California | +0.165 | +0.020 | +0.018 | +0.013 | −0.008 | −0.103 |
| Ridgecrest 2019 | California | +0.078 | +0.009 | −0.092 | +0.000 | +0.034 | −0.036 |
| Northridge 1994 | California | +0.088 | +0.007 | −0.087 | −0.025 | −0.608 | −0.018 |
| El Mayor–Cucapah 2010 | California | +0.073 | +0.011 | +0.019 | +0.018 | +0.025 | +0.032 |
| Loma Prieta 1989 | California | +0.070 | +0.017 | −0.001 | −0.110 | +0.034 | −0.036 |
| San Simeon 2003 | California | +0.263 | +0.004 | −0.035 | −0.118 | +0.248 | −0.190 |
| South Napa 2014 | California | +0.061 | +0.034 | −0.035 | +0.012 | −0.069 | +0.062 |
| Parkfield 2004 | California | +0.071 | +0.009 | −0.009 | +0.018 | −0.155 | −0.024 |
| L’Aquila 2009 | Central Italy | +0.111 | +0.021 | −0.091 | +0.051 | +0.064 | +0.017 |
| Amatrice 2016 | Central Italy | +0.133 | +0.012 | −0.083 | +0.030 | −0.132 | +0.051 |
| Norcia 2016 | Central Italy | +0.095 | +0.028 | −0.086 | +0.016 | +0.055 | +0.014 |
| Darfield 2010 | New Zealand | +0.810 | +0.070 | −0.114 | +0.084 | +0.771 | +0.049 |
| Kaikōura 2016 | New Zealand | +0.160 | +0.007 | −0.068 | −0.119 | −0.114 | −0.112 |
| sequences above Omori | 14 / 14 | 14 / 14 | 2 / 14 | 9 / 14 | 8 / 14 | 7 / 14 |
across regions
Learned models trained on every other sequence (pooled), only on earlier sequences (chronological), or only on other regions. If the regional column falls well below the pooled one, what the model learned does not transfer across tectonic settings and catalogs.
| region | sequences | GBM-hybrid · cross/pooled | GBM-hybrid · cross/chrono | GBM-hybrid · cross/other_region | ETAS (fixed μ, subcritical) |
|---|---|---|---|---|---|
| California | 9 | −0.033 | −0.003 | −0.041 | +0.114 |
| Central Italy | 3 | +0.031 | −0.025 | +0.032 | +0.116 |
| New Zealand | 2 | +0.033 | +0.061 | +0.008 | +0.645 |
Event-weighted mean of per-sequence information gain over Omori. Region and catalog are confounded here: each region comes from a different agency with its own network and magnitude practice.
beyond the log score
Mean across sequences, on each sequence’s windows where every model forecast. Error is in log₁₀(count + 1); coverage should be near 90%; the number test rejects a forecast whose total is implausible.
| model | abs. error | bias | 90% coverage | number-test rejections | sequences |
|---|---|---|---|---|---|
| Omori–Utsu | 0.138 | −0.003 | 56% | 39% | 14 |
| Omori–Utsu + NB spread | 0.138 | −0.003 | 94% | 3% | 14 |
| ETAS (fixed μ, subcritical) | 0.264 | +0.228 | 95% | 2% | 14 |
| ETAS (free) | 0.269 | +0.241 | 90% | 8% | 14 |
| Persistence | 0.241 | +0.216 | 36% | 59% | 14 |
| Poisson GLM · within | 0.167 | +0.048 | 52% | 42% | 14 |
| GBM-hybrid · within | 0.164 | +0.040 | 49% | 43% | 14 |
| Poisson GLM · cross/pooled | 0.151 | +0.072 | 53% | 40% | 14 |
| GBM · cross/pooled | 0.224 | +0.163 | 40% | 54% | 14 |
| GBM-hybrid · cross/pooled | 0.147 | +0.043 | 55% | 39% | 14 |
| Poisson MLP · cross/pooled | 0.216 | +0.048 | 42% | 50% | 14 |
| GBM-hybrid · cross/chrono | 0.157 | +0.038 | 52% | 42% | 13 |
| GBM-hybrid · cross/other_region | 0.163 | +0.097 | 52% | 41% | 14 |