Official warnings and forecasts come from the National Hurricane Center. This page independently scores SHIPS-RII's published probabilities against what happened; it supplements NHC guidance and never replaces it.
RII Verification
Did SHIPS-RII's rapid-intensification probabilities actually predict what happened? Scored against best-track outcomes, 2018–2025 backfill plus the live 2026 season — a live number, not a fixed retrospective study, so it grows as pairs accrue. As of Nov 2, 2026, 11:15 AM UTC.
These numbers refresh automatically — every scheduled pipeline cycle re-scores the pool against that cycle's newly-elapsed SHIPS windows, not just once at launch. A refresh only lands if it passes an internal sanity check first; if a cycle's result looks implausible, this page keeps showing the last good numbers rather than a broken update, so "As of" above can lag by more than one cycle on occasion.
Headline pair: 30 kt / 24 h, SHIPS-RII row, pre-registered before backfill ran. Given SHIPS issued a forecast, how skillful was it — not skill across all storm-hours: cycles where SHIPS structurally couldn't run (sub-threshold intensity, center inland) create no pairs and are excluded from this number, not scored as zero. n = 6190 pairs from 351 storms.
SHIPS-RII ranks RI cases well while being systematically over-confident — these answer different questions, not one. Discrimination (AUC 0.874) asks whether RI cases got higher probabilities than non-RI cases; on that question SHIPS-RII does strongly. Calibration (the reliability table below, and the MHW-stratum calibration further down) asks whether a 30% forecast verifies about 30% of the time; on that question SHIPS-RII runs systematically high — real skill in ranking is not the same claim as well-calibrated probabilities, and this page reports both because a forecast can pass one and fail the other.
Reliability
| Forecast bin | n | Observed freq. | Wilson 95% | Block-bootstrap 95% |
|---|---|---|---|---|
| 0.0%–10.0% | 2955 | 0.6% | [0.4%, 1.0%] | [0.3%, 0.9%] |
| 10.0%–20.0% | 2046 | 5.1% | [4.2%, 6.1%] | [3.7%, 6.4%] |
| 20.0%–30.0% | 634 | 18.0% | [15.2%, 21.2%] | [14.3%, 22.2%] |
| 30.0%–40.0% | 275 | 25.5% | [20.7%, 30.9%] | [19.2%, 31.8%] |
| 40.0%–50.0% | 141 | 39.7% | [32.0%, 48.0%] | [28.6%, 50.0%] |
| 50.0%–60.0% | 48 | 52.1% | [38.3%, 65.5%] | [37.0%, 67.6%] |
| 60.0%–70.0% | 41 | 46.3% | [32.1%, 61.3%] | [26.9%, 64.6%] |
| 70.0%–80.0% | 18 | 38.9% | [20.3%, 61.4%] | [14.3%, 64.7%] |
| 80.0%–90.0% | 15 | 73.3% | [48.0%, 89.1%] | [45.5%, 94.7%] |
| 90.0%–100.0% | 17 | 82.4% | [59.0%, 93.8%] | [60.0%, 100.0%] |
Bins with n < 5 are shown hollow, not dropped — a bin's absence should never be silently invisible.
SHIPS prints each RI probability rounded to a whole percent, and scores here use that printed value. A forecast printed as exactly 10%, 20%, and so on counts in the upper reliability bin, even when the unrounded value was slightly lower.
Murphy decomposition
The Brier score splits into three terms (Murphy, 1973), each asking a different question of the same forecasts:
reliability − resolution + uncertainty = 0.056911 (binned Brier) · raw Brier score = 0.056877. The decomposition groups forecasts into the same 10 bins the reliability table above uses and treats each as its bin mean, which discards within-bin variation, so the binned score sits slightly below the raw one — a 0.06% gap here.
Calibration slope (b1) = 1.366, 95% CI [1.158, 1.613]
SHIPS-RII's probabilities are too compressed within the graded regime — it ranks well but does not spread. A slope above 1 means the spread is too narrow, which is distinct from the over-forecasting described above.
Frozen research fit from a manually re-run script (pipeline/snapshot_calibration.py), not a live, auto-refreshing metric like the numbers above it — see docs/prereg/mhw-residual-2026-07-29.md for the pre-registration and its amendment log.
Marine heatwave stratified analysis
Storm-level split across the full dataset: 239 storms reached MHW category ≥1 somewhere along their track, 105 never did. 8 storms (7 Central Pacific, structurally outside the tracked OISST domain; 1 whose track never resolved to a position on it) are excluded from every MHW comparison below, not silently folded into either side.
Within-season comparison (required, not a sensitivity row)
MHW-active and inactive storms are compared separately within each season, then pooled (Mantel-Haenszel) — a naive pool of all seasons together can't be trusted here, because 2023–2024 were both far more MHW-active and far more RI-active than every other season in this dataset, entangling an MHW effect with a season effect. The within-season method is what makes it possible to ask whether an effect survives once that confound is controlled for.
Suggestive, not established. The odds-ratio confidence interval's lower bound sits at 1.072 — barely above 1 — and the positive direction holds in 7 of 9 seasons shown below, not all of them. Two data points is not a trend and eight seasons is not a settled result; this is evidence worth taking seriously, not a confirmed effect. The per-season table below is shown in full every time this result is, on purpose — a pooled number without the breakdown would hide exactly the season-to-season variation that matters for judging how strong this evidence actually is.
| Season | n active | Rate active | n inactive | Rate inactive | Δ |
|---|---|---|---|---|---|
| 2018 | 21 | 47.6% | 14 | 28.6% | +19.0 pts |
| 2019 | 29 | 27.6% | 9 | 0.0% | +27.6 pts |
| 2020 | 33 | 30.3% | 18 | 11.1% | +19.2 pts |
| 2021 | 20 | 35.0% | 18 | 22.2% | +12.8 pts |
| 2022 | 19 | 26.3% | 16 | 37.5% | -11.2 pts |
| 2023 | 32 | 43.8% | 8 | 37.5% | +6.3 pts |
| 2024 | 28 | 39.3% | 4 | 0.0% | +39.3 pts |
| 2025 | 19 | 31.6% | 12 | 33.3% | -1.8 pts |
| 2026 | 36 | 33.3% | 4 | 0.0% | +33.3 pts |
8 storm(s) excluded for unmeasurable MHW, 4 for having no resolved 30kt/24h pair —340 storms enter the comparison above.
3 storm(s) awaiting final OISST: resolved, MHW not yet classified, not counted as unmeasurable.
No pair or storm in this dataset (2018-2025 backfill plus 2026 live) has ever had a tracked position coincide with MHW category 3 ("Severe") or 4 ("Extreme") water, even though both levels occur elsewhere in the tracked ocean domain. Every dose-response or MHW-stratum result on this page therefore covers categories 0-2 only and must not be read as extending to extreme marine-heatwave conditions (spec Decision 36).
Reliability by MHW stratum (descriptive)
These panels describe how SHIPS-RII verified in each MHW stratum. They are not the pre-registered test of whether marine heatwaves explain SHIPS-RII errors. That test uses 2018–2025 only, and its results are not published here.
MHW present (storm max category ≥ 1)
n = 4713 pairs / 237 storms
| Forecast bin | n | Observed freq. | Wilson 95% | Block-bootstrap 95% |
|---|---|---|---|---|
| 0.0%–10.0% | 2224 | 0.7% | [0.4%, 1.1%] | [0.3%, 1.1%] |
| 10.0%–20.0% | 1602 | 5.5% | [4.5%, 6.7%] | [3.9%, 7.1%] |
| 20.0%–30.0% | 465 | 19.6% | [16.2%, 23.4%] | [14.8%, 24.8%] |
| 30.0%–40.0% | 218 | 25.2% | [19.9%, 31.4%] | [18.4%, 33.0%] |
| 40.0%–50.0% | 104 | 39.4% | [30.6%, 49.0%] | [26.7%, 52.5%] |
| 50.0%–60.0% | 36 | 58.3% | [42.2%, 72.9%] | [38.7%, 76.7%] |
| 60.0%–70.0% | 31 | 41.9% | [26.4%, 59.2%] | [22.6%, 63.6%] |
| 70.0%–80.0% | 14 | 35.7% | [16.3%, 61.2%] | [7.7%, 69.6%] |
| 80.0%–90.0% | 9 | 77.8% | [45.3%, 93.7%] | [42.9%, 100.0%] |
| 90.0%–100.0% | 10 | 80.0% | [49.0%, 94.3%] | [25.0%, 100.0%] |
MHW absent (category 0)
n = 1304 pairs / 103 storms
| Forecast bin | n | Observed freq. | Wilson 95% | Block-bootstrap 95% |
|---|---|---|---|---|
| 0.0%–10.0% | 633 | 0.5% | [0.2%, 1.4%] | [0.0%, 1.3%] |
| 10.0%–20.0% | 399 | 3.5% | [2.1%, 5.8%] | [1.5%, 6.2%] |
| 20.0%–30.0% | 158 | 13.3% | [8.9%, 19.5%] | [7.6%, 19.8%] |
| 30.0%–40.0% | 56 | 26.8% | [17.0%, 39.6%] | [13.2%, 42.9%] |
| 40.0%–50.0% | 28 | 35.7% | [20.7%, 54.2%] | [10.0%, 57.6%] |
| 50.0%–60.0% | 11 | 27.3% | [9.7%, 56.6%] | [0.0%, 50.0%] |
| 60.0%–70.0% | 6 | 66.7% | [30.0%, 90.3%] | [0.0%, 100.0%] |
| 70.0%–80.0% | 4 | 50.0% | [15.0%, 85.0%] | [0.0%, 100.0%] |
| 80.0%–90.0% | 5 | 60.0% | [23.1%, 88.2%] | [0.0%, 100.0%] |
| 90.0%–100.0% | 4 | 75.0% | [30.1%, 95.4%] | [0.0%, 100.0%] |
Same storm-level strata as the calibration cards and the BSS/AUC sensitivity rows; eras pooled. Bins with n < 5 are shown hollow, not dropped.
No pair or storm in this dataset (2018-2025 backfill plus 2026 live) has ever had a tracked position coincide with MHW category 3 ("Severe") or 4 ("Extreme") water, even though both levels occur elsewhere in the tracked ocean domain. Every dose-response or MHW-stratum result on this page therefore covers categories 0-2 only and must not be read as extending to extreme marine-heatwave conditions (spec Decision 36).
Calibration by MHW stratum (secondary)
Not a BSS/AUC comparison here on purpose — BSS and AUC are base-rate- sensitive and the two strata differ in base rate by construction, so comparing them directly would partly just be re-measuring the split itself. This instead asks the more direct operational question: does SHIPS' own forecast probability track what actually happened, within each stratum?
SHIPS over-forecasts in both strata (negative gap, neither confidence interval crosses zero) — the same over-confidence the headline calibration finding above describes, and it shows up whether or not the storm ever crossed a marine heatwave.
No pair or storm in this dataset (2018-2025 backfill plus 2026 live) has ever had a tracked position coincide with MHW category 3 ("Severe") or 4 ("Extreme") water, even though both levels occur elsewhere in the tracked ocean domain. Every dose-response or MHW-stratum result on this page therefore covers categories 0-2 only and must not be read as extending to extreme marine-heatwave conditions (spec Decision 36).
Dose-response: forecast-track intersection with active MHW cells (secondary)
Category itself never exceeds 2 anywhere in this dataset (see the caveat below), so the fraction of a storm's forecast track intersecting an active MHW cell is the only real continuous exposure axis available here — a secondary, supporting look, not a second confirmation of the within-season result above.
| Track intersection | n | Observed RI freq. | Wilson 95% | Block-bootstrap 95% |
|---|---|---|---|---|
| 0.0 (none) | 3381 | 6.9% | [6.1%, 7.8%] | [5.4%, 8.4%] |
| partial | 1223 | 7.2% | [5.9%, 8.8%] | [4.9%, 9.5%] |
| 1.0 (full) | 807 | 9.5% | [7.7%, 11.8%] | [5.7%, 14.0%] |
The trend runs in one direction across all three buckets — more track intersecting active MHW cells lines up with a higher observed RI frequency. Reported as supporting the within-season finding above, not as independent confirmation of it: both are drawn from the same underlying storms and the same MHW classification.
722 pair(s) excluded for unmeasurable MHW position.
57 pair(s) awaiting final OISST: not yet classified, not counted as unmeasurable.
No pair or storm in this dataset (2018-2025 backfill plus 2026 live) has ever had a tracked position coincide with MHW category 3 ("Severe") or 4 ("Extreme") water, even though both levels occur elsewhere in the tracked ocean domain. Every dose-response or MHW-stratum result on this page therefore covers categories 0-2 only and must not be read as extending to extreme marine-heatwave conditions (spec Decision 36).
For 2026 live records, track_fraction_intersecting is a snapshot at capture time, not a stable measurement: NHC retroactively inserted/revised operational b-deck fixes for a completed storm (EP052026/Elida) after its verification record was archived, changing track_fraction_intersecting (0.833 -> 0.75 in one case) without changing category or peak_anomaly_c. 2018-2025 HURDAT2 backfill is unaffected -- it is a final, post-season record with no equivalent revision path. This dose-response axis therefore mixes a stable historical measurement with a mutable live one (spec Decision 41).
Sensitivity analysis
Every row below is a reported delta from the primary number above, not an alternate view — the headline is always the primary pool.
| Variant | n (pairs/storms) | BSS (sample) | AUC |
|---|---|---|---|
| dissipated excluded | 5443 / 323 | 0.144 | 0.864 |
| et included | 7299 / 355 | 0.137 | 0.891 |
| landfall excluded | 4725 / 306 | 0.183 | 0.885 |
| 2018–2025 only. Live-season records carry no landfall data, so they are left out of this row rather than counted either way. Computed over the 5243 pairs where landfall is known; 947 live-season pairs are excluded. Compare this row with the 2018–2025 row, not the primary pool. | |||
| era 2018 2025 | 5243 / 306 | 0.135 | 0.870 |
| era operational truth | 947 / 45 | 0.131 | 0.899 |
| basin al | 2993 / 166 | 0.076 | 0.847 |
| basin ep | 3000 / 176 | 0.155 | 0.882 |
| model version 2018 2019 | 1384 / 76 | 0.159 | 0.865 |
| model version 2020 2025 | 3859 / 230 | 0.127 | 0.872 |
The model-version rows (2018–2019 vs. 2020–2025) test a documented candidate SHIPS-RII predictor revision — sourced from secondary literature, not a primary read. A large delta there is suggestive of version-mixing, not proof.
MHW stratum, BSS/AUC (sensitivity only — see the calibration section above for the primary MHW comparison)
| Stratum | n (pairs/storms) | Base rate | BSS (sample) | AUC | AUC 95% |
|---|---|---|---|---|---|
| mhw active storms | 4713 / 237 | 7.3% | 0.130 | 0.868 | [0.843, 0.890] |
| mhw inactive storms | 1304 / 103 | 6.0% | 0.091 | 0.885 | [0.816, 0.932] |
Base rate shown deliberately next to BSS/AUC here — the two strata differ in base rate by construction, so a naive BSS/AUC comparison between them would partly just remeasure that difference. Read alongside the base rate, not instead of it.
SHIPS cycle coverage
Coverage is the share of a storm's 6-hourly best-track positions, while it was a tropical or subtropical cyclone, for which a SHIPS forecast cycle was captured. It is shown for context only; no score above filters on it.
2018 to 2025 (backfill), all seasons pooled: 91.1% (5619 of 6166 fixes, 317 storms). This is a lower bound: it was rebuilt from the scored records, and a cycle whose SHIPS file produced no scored pair leaves no record.
| Season | Covered / fixes | Coverage (lower bound) |
|---|---|---|
| 2018 | 862 / 1035 | 83.3% |
| 2019 | 627 / 676 | 92.8% |
| 2020 | 874 / 929 | 94.1% |
| 2021 | 701 / 731 | 95.9% |
| 2022 | 612 / 688 | 89.0% |
| 2023 | 787 / 868 | 90.7% |
| 2024 | 574 / 612 | 93.8% |
| 2025 | 582 / 627 | 92.8% |
2026 (live), named storms pooled: 97.4% (797 of 818 fixes, 35 storms). Live coverage runs lower because the site's pipeline, which runs a few times a day, doesn't capture every SHIPS cycle.
What these numbers can and can't support: this is a student research tool. Reliability and skill scores here reflect 351 distinct storms, not 6190 independent observations — SHIPS cycles every 6 hours, so pairs from one storm are correlated, and the block-bootstrap intervals above account for that. The MHW within-season result is suggestive, not established — treat it as evidence worth taking seriously, not a confirmed effect. Where a sensitivity row's n is small, treat its delta as suggestive, not conclusive.