failures
Where the models — classical and learned — go wrong, and where the benchmark’s own design choices are fragile. Nothing on this page is filtered for flattery.
fits on their bounds
Origins at which a fitted decay model finished on its search bound. Early in a sequence the catalog is still incomplete, the observed rate looks flat, and the likelihood does not identify the decay parameters — for Omori–Utsu as much as for ETAS. Forecasts from those fits can still be serviceable; the parameters are not.
| sequence | origins | Omori on a bound | primary ETAS on a bound | parameters | free ETAS on a bound | free ETAS supercritical |
|---|---|---|---|---|---|---|
| Landers 1992 | 15 | 4 | 3 | p, alpha | 0 | 8 |
| Hector Mine 1999 | 15 | 1 | 0 | — | 2 | 1 |
| Ridgecrest 2019 | 15 | 0 | 0 | — | 1 | 2 |
| Northridge 1994 | 15 | 1 | 0 | — | 1 | 1 |
| El Mayor–Cucapah 2010 | 15 | 2 | 2 | p | 3 | 0 |
| Loma Prieta 1989 | 15 | 0 | 0 | — | 0 | 1 |
| San Simeon 2003 | 15 | 1 | 1 | p | 1 | 6 |
| South Napa 2014 | 15 | 5 | 0 | — | 0 | 0 |
| Parkfield 2004 | 15 | 0 | 1 | alpha | 3 | 0 |
| L’Aquila 2009 | 15 | 1 | 0 | — | 1 | 0 |
| Amatrice 2016 | 11 | 1 | 0 | — | 0 | 1 |
| Norcia 2016 | 15 | 0 | 0 | — | 0 | 0 |
| Darfield 2010 | 15 | 3 | 5 | c, p | 7 | 7 |
| Kaikōura 2016 | 15 | 2 | 1 | p | 1 | 0 |
overconfidence
Share of windows where the observed count fell outside the 90% interval, and where the number test rejected the forecast, averaged across sequences. Poisson intervals are too narrow for clustered counts; the “+ NB spread” variants fit a negative-binomial spread to each model’s own closed forecasts.
| model | outside 90% interval | number-test rejections |
|---|---|---|
| Omori–Utsu | 44% | 39% |
| Omori–Utsu + NB spread | 6% | 3% |
| ETAS (fixed μ, subcritical) | 5% | 2% |
| ETAS (free) | 10% | 8% |
| Persistence | 64% | 59% |
| GBM-hybrid · within | 51% | 43% |
| GBM-hybrid · within + NB | 24% | 20% |
| GBM-hybrid · cross/pooled | 45% | 39% |
| GBM-hybrid · cross/pooled + NB | 7% | 3% |
the worst windows
The fifteen largest errors in log₁₀(count + 1) across all sequences for Omori, ETAS and the learned hybrids.
| sequence | model | origin (d) | horizon (d) | observed | forecast mean | log₁₀ error |
|---|---|---|---|---|---|---|
| San Simeon 2003 | GBM-hybrid · cross/pooled | 1 | 30 | 1,178 | 55.7 | −1.32 |
| San Simeon 2003 | Omori–Utsu | 1 | 30 | 1,178 | 129.2 | −0.96 |
| El Mayor–Cucapah 2010 | ETAS (fixed μ, subcritical) | 240 | 7 | 2 | 25.7 | +0.95 |
| El Mayor–Cucapah 2010 | ETAS (fixed μ, subcritical) | 240 | 1 | 0 | 7.5 | +0.93 |
| Ridgecrest 2019 | ETAS (fixed μ, subcritical) | 60 | 30 | 174 | 1,471.9 | +0.93 |
| San Simeon 2003 | GBM-hybrid · cross/pooled | 1 | 7 | 681 | 83.1 | −0.91 |
| Ridgecrest 2019 | ETAS (fixed μ, subcritical) | 90 | 30 | 125 | 991.7 | +0.90 |
| San Simeon 2003 | ETAS (fixed μ, subcritical) | 1 | 30 | 1,178 | 159.4 | −0.87 |
| Hector Mine 1999 | ETAS (fixed μ, subcritical) | 240 | 7 | 0 | 5.9 | +0.84 |
| Ridgecrest 2019 | ETAS (fixed μ, subcritical) | 45 | 30 | 277 | 1,910.9 | +0.84 |
| Kaikōura 2016 | GBM-hybrid · cross/pooled | 1 | 30 | 583 | 89.4 | −0.81 |
| Darfield 2010 | GBM-hybrid · within | 10 | 30 | 641 | 99.7 | −0.80 |
| Ridgecrest 2019 | ETAS (fixed μ, subcritical) | 21 | 30 | 551 | 3,374.5 | +0.79 |
| Darfield 2010 | GBM-hybrid · cross/pooled | 1 | 30 | 1,398 | 8,060.1 | +0.76 |
| Northridge 1994 | ETAS (fixed μ, subcritical) | 21 | 30 | 91 | 525.5 | +0.76 |
design choices under pressure
Each robustness variant changes one choice — completeness cutoff, window start, zone radius or rule, window length, truncation, foreshock history — and reruns the benchmark at lighter settings.
| sequence | variant | Mc | events | ETAS vs Omori | GBM-hybrid vs Omori | windows |
|---|---|---|---|---|---|---|
| Amatrice 2016 | mc+0.3 | 2.1 | 1,652 | +0.001 | −0.015 | 7 |
| Amatrice 2016 | mc+0.6 | 2.4 | 730 | +0.003 | −0.018 | 7 |
| Amatrice 2016 | no_lookback | 1.8 | 3,912 | +0.030 | −0.149 | 7 |
| Amatrice 2016 | primary_light | 1.8 | 3,912 | +0.030 | −0.149 | 7 |
| Amatrice 2016 | r0.5 | 1.8 | 3,845 | +0.035 | −0.098 | 7 |
| Amatrice 2016 | r1.5 | 1.8 | 3,922 | +0.030 | −0.127 | 7 |
| Amatrice 2016 | r2.0 | 1.8 | 3,936 | +0.030 | −0.085 | 7 |
| Amatrice 2016 | t0=1 | 1.6 | 5,883 | +0.031 | −0.212 | 7 |
| Amatrice 2016 | t0=2 | 1.8 | 3,065 | +0.029 | −0.113 | 4 |
| Amatrice 2016 | t_end=180 | 1.8 | 3,912 | +0.030 | −0.149 | 7 |
| Amatrice 2016 | wc_radius | 1.8 | 2,616 | +0.040 | +0.001 | 8 |
| Darfield 2010 | mc+0.3 | 3.1 | 2,441 | −0.011 | −0.128 | 15 |
| Darfield 2010 | mc+0.6 | 3.4 | 1,242 | +0.005 | −0.088 | 15 |
| Darfield 2010 | no_lookback | 2.8 | 4,641 | +0.006 | −0.102 | 15 |
| Darfield 2010 | no_truncation | 2.8 | 4,641 | +0.005 | −0.102 | 15 |
| Darfield 2010 | primary_light | 2.8 | 4,641 | +0.005 | −0.102 | 15 |
| Darfield 2010 | r0.5 | 2.8 | 2,945 | −0.005 | −0.116 | 15 |
| Darfield 2010 | r1.5 | 2.8 | 4,712 | +0.004 | −0.087 | 15 |
| Darfield 2010 | r2.0 | 2.8 | 4,769 | +0.004 | −0.100 | 15 |
| Darfield 2010 | t0=1 | 2.8 | 4,507 | +0.004 | −0.093 | 15 |
| Darfield 2010 | t0=2 | 2.8 | 4,295 | +0.011 | −0.087 | 12 |
| Darfield 2010 | t_end=180 | 2.8 | 3,217 | −0.002 | −0.114 | 12 |
| Darfield 2010 | wc_radius | 2.8 | 4,645 | +0.006 | −0.093 | 15 |
| Kaikōura 2016 | primary_light | 3.2 | 941 | −0.029 | −0.023 | 15 |
| Kaikōura 2016 | r1.5 | 3.3 | 1,021 | −0.046 | −0.014 | 15 |
| Kaikōura 2016 | r2.0 | 3.3 | 1,544 | −0.023 | −0.001 | 15 |
| Kaikōura 2016 | wc_radius | 3.4 | 802 | −0.048 | −0.027 | 15 |
| Landers 1992 | mc+0.3 | 2.8 | 1,644 | +0.011 | +0.013 | 15 |
| Landers 1992 | mc+0.6 | 3.1 | 680 | +0.047 | +0.038 | 15 |
| Landers 1992 | no_lookback | 2.5 | 3,679 | +0.003 | −0.039 | 15 |
| Landers 1992 | no_truncation | 2.5 | 3,679 | +0.002 | −0.039 | 15 |
| Landers 1992 | primary_light | 2.5 | 3,679 | +0.002 | −0.039 | 15 |
| Landers 1992 | r0.5 | 2.5 | 2,353 | −0.019 | −0.023 | 15 |
| Landers 1992 | r1.5 | 2.5 | 4,403 | +0.005 | +0.010 | 15 |
| Landers 1992 | r2.0 | 2.5 | 4,487 | +0.011 | −0.011 | 15 |
| Landers 1992 | t0=1 | 2.5 | 3,465 | +0.002 | +0.017 | 15 |
| Landers 1992 | t0=2 | 2.5 | 3,153 | −0.012 | +0.006 | 12 |
| Landers 1992 | t_end=180 | 3.0 | 844 | +0.021 | +0.031 | 12 |
| Landers 1992 | wc_radius | 2.5 | 3,795 | +0.009 | −0.023 | 15 |
| Parkfield 2004 | mc+0.3 | 1.5 | 1,408 | +0.059 | +0.029 | 15 |
| Parkfield 2004 | mc+0.6 | 1.8 | 760 | +0.060 | −0.026 | 15 |
| Parkfield 2004 | no_lookback | 1.2 | 2,666 | +0.034 | −0.002 | 15 |
| Parkfield 2004 | no_truncation | 1.2 | 2,666 | +0.034 | −0.002 | 15 |
| Parkfield 2004 | primary_light | 1.2 | 2,666 | +0.034 | −0.002 | 15 |
| Parkfield 2004 | r0.5 | 1.5 | 573 | +0.009 | +0.002 | 15 |
| Parkfield 2004 | r1.5 | 1.2 | 4,485 | +0.046 | +0.011 | 15 |
| Parkfield 2004 | r2.0 | 1.2 | 4,805 | +0.044 | +0.001 | 15 |
| Parkfield 2004 | t0=1 | 1.2 | 2,566 | +0.041 | +0.009 | 15 |
| Parkfield 2004 | t0=2 | 1.2 | 2,483 | +0.048 | +0.028 | 12 |
| Parkfield 2004 | t_end=180 | 1.2 | 1,885 | +0.021 | −0.007 | 12 |
| Parkfield 2004 | wc_radius | 1.2 | 590 | +0.053 | +0.031 | 15 |