SwimSignal

Model scorecard

Each model or candidate SwimSignal has built or is testing to forecast sewage spills, with how it was tested and how it did against a simple reference. A score is a past result on the rows it names, not a promise about tomorrow, and none of it is a water test.

Brier score: the mean squared gap between the chance given and what happened (1 for a spill, 0 for none). Lower is better; 0 is perfect. Calibration error (ECE): how far, on average, the chances given were from how often spills happened at those chances. Lower is better. A 95% interval is the range the difference would fall in 95 times in 100 if the rows were drawn again. When it does not include zero, the difference is unlikely to be chance alone.

The live spill forecast

Live: this is the forecast the site shows.

What it predicts
For each monitored storm overflow upstream of a spot, the chance that it spills on each of the next five days, from rain, how wet the ground is, the time of year and the overflow's own record. A spot's sewage spill risk adds those chances up along the river. On rivers from May to September, a second model estimates the chance a water sample would hold more than 900 E. coli per 100 ml.
Where and when it was tested
The spill model was fitted on United Utilities' overflows in 2023 and 2024 and tested on 2025: 826,725 overflow-days. The E. coli estimate was tested on 1,502 Environment Agency samples at 31 inland bathing waters, 2024 to 2026. Since 28 September 2026 every live forecast is scored afterwards against what the overflows did, at all nine companies.
Compared with
Each overflow's long-run spill rate; for the live record also "it is spilling now or spilled in the last two days" and "more than 10 mm or 5 mm of rain in two days".
How it did

It beats each overflow's long-run rate and ranks spill days well, but it is not yet shown to beat simple rules a swimmer could use. So for now it is worded as showing what the upstream monitors say, with the forecast still being tested.

ResultThis modelReferenceDifference [95% interval]
2025, United Utilities, same-day, reanalysis rainBrier, held out. Skill 0.33. Rain as it fell, so weather-forecast error is left out.0.04450.0669 long-run spill rate
E. coli over 900 on rivers and lakes, each year left outBrier, held out. Climatology by water type: 0.120. On lakes neither does reliably better than chance.0.0920.096 rain only
Live, all companies, same-day, 29 Sep to 23 Oct 2026Skill, live record. Skill is 1 − Brier ÷ the reference's Brier: above 0 is better than the reference.0.21long-run spill rate[0.08, 0.44]
Same rows, against "spilling now or in the last two days"Skill, live record. Not shown to beat persistence on the same day.−0.05persistence[−0.19, 0.17]
Live, all companies, one to four days ahead, against the best simple ruleSkill, live record.0.07rain over 10 mm in two days[−0.05, 0.12]
What it misses
The spill model learned from United Utilities only; other companies' overflows get the same model. It does not see runoff from fields and roads, treated sewage, wildlife, or wind and mixing on a lake, and on lakes E. coli spikes are mostly not from overflows. In Thames Water's area it was worse than persistence on the same day. Welsh overflows are not in it.
How often it was wrong
Its chances run too high: in the live check, 29 September to 23 October 2026, same-day spills happened 0.74 times as often as forecast (95% interval 0.65 to 0.87), and 388 of 963 high warnings came true (40%). In the 2025 test, days forecast at 85% had a spill 79% of the time.
Is a result shown?
Yes. It is the forecast, worded by what it has shown. Its live score is published every day on the Accuracy page, including where it is wrong.
Records
Accuracy page, Method, Interim usefulness read, Calibration check

Regional routing (regional-v1)

Running test: window 31 Oct 2026 to 27 Nov 2026. Read on or after 28 Nov 2026; no score from the window is shown before then.

What it predicts
The same daily spill chance per overflow. At eight companies it uses a model fitted on all nine companies' records with a correction for each overflow (the nine-company model's arm (b)); at United Utilities, and anywhere else, it keeps the live forecast.
Where and when it was tested
The nine-company model was trained on the Environment Agency's 2024 and 2025 records and tested on 1 January to 31 August 2026: 3,270,537 overflow-days at 13,459 overflows. The routing rule was then chosen from those same results.
Compared with
The live forecast, each overflow's past spill rate, and a flat rate per company.
How it did

The nine-company model was not adopted: 11% to 23% better at eight companies, 3.4% worse at United Utilities. The routing rule is a development idea tested live from 31 October.

ResultThis modelReferenceDifference [95% interval]
Nine-company model, arm (b), all nine companies, same-dayBrier, held out. Interval: 95%, resampling overflows and weeks.0.03650.042 live forecast−0.0055 [−0.00883, −0.00268]
Same, United Utilities onlyBrier, held out. 3.4% worse, interval wholly above zero: the reason United Utilities keeps the live forecast.0.0390.0377 live forecast[0.00007, 0.00261]
regional-v1 rule, all nine companies, same-dayBrier, seen: a direction only. The rows that chose the rule, with reanalysis rain: says the rule is worth a live look, not how it will do live.0.036330.04204 live forecast
What it misses
Chosen after seeing the results, so its seen-data gain will shrink live. United Utilities keeps the live forecast. The test is judged on SwimSignal's own polls of the companies' live feeds: a missing or stale status counts as unknown, never as a dry day.
How often it was wrong
On 2026 data its calibration error was 0.009 across the nine companies, against the live forecast's 0.024 (worst company: Thames, 0.0221 against 0.0401).
Is a result shown?
No. Not until the read on 28 November 2026. If it passes, a limited rollout at the eight companies is proposed in its own change.
Rows logged so far
124,740 rows from 30 Oct 2026 to 31 Oct 2026. A count only: no score is worked out from them before the read.
The rule at the read
Useful if its same-day Brier at the eight companies is at least 5% below the live forecast's, with the 95% interval below zero; a company more than 5% worse goes back to the live forecast. Settled before the window opened. Under 10,000 known overflow-days at the eight companies: one more 28-day window.
Records
Regional routing plan, Nine-company model test

Live spill status as an input

Running test: window 25 Oct 2026 to 21 Nov 2026. Read on or after 23 Nov 2026; no score from the window is shown before then.

What it predicts
The daily spill chance per overflow, today and four days ahead, with three more inputs: whether the overflow is spilling now, its spill hours in the last 24 hours and its spill days in the last week.
Where and when it was tested
Fitted on 2024, tested on 2025 at all nine companies: 5,077,150 overflow-days with reanalysis rain, and 1,928,619 a lead with archived forecast rain. Labels from the Environment Agency's start and stop records.
Compared with
The nine-company model without the status inputs (not the live forecast), and a flat rate per company.
How it did

Passed all five conditions at every lead on 2025. Not deployed: it must first work on SwimSignal's own live polls, with a fallback when the status is missing.

ResultThis modelReferenceDifference [95% interval]
2025, same-day, reanalysis rainBrier, held out. Interval: 95%, resampling calendar weeks.0.02650.0362 same model without status−0.0096 [−0.0125, −0.007]
2025, four days ahead, reanalysis rainBrier, held out.0.03320.0362 same model without status−0.003 [−0.0041, −0.0019]
2025, same-day, forecast rainBrier, held out.0.028680.03784 same model without status−0.0092 [−0.0118, −0.0067]
2025, four days ahead, forecast rainBrier, held out.0.04510.04919 same model without status−0.0041 [−0.0058, −0.0024]
What it misses
With every status input missing it is much worse than the model without them (0.0537 against 0.0362 same-day). About half the same-day gain comes from how the label is defined. Northumbrian gains least. Southern reports no usable live status, so the live test says nothing about it.
How often it was wrong
Calibration error same-day 0.003, against 0.008 for the model without status.
Is a result shown?
No. Logged beside the forecast but never shown: no page, alert or score reads it before the read.
Rows logged so far
1,255,470 rows from 20 Oct 2026 to 31 Oct 2026. A count only: no score is worked out from them before the read.
The rule at the read
Lead by lead, against the live forecast: the 95% interval wholly below zero over the nine companies, no company with 50 or more spill rows clearly worse, calibration no worse (or at most 0.02), and spills that start from dry not clearly worse. Too few spill rows with a known status: runs to 5 December, read on or after 7 December.
Records
Live-status test, Live-status shadow

Lead calibration, option B

Running test: window 18 Oct 2026 to 14 Nov 2026. Read on or after 16 Nov 2026; no score from the window is shown before then.

What it predicts
The live forecast's daily spill chances, re-mapped by a new calibration for each lead, fitted on 2025 at the eight companies other than United Utilities.
Where and when it was tested
A 2025 replay at 3,050 overflows at the eight companies, the map fitted on odd months and scored on even ones, then the reverse. Then a check on SwimSignal's own live forecasts, 29 September to 16 October 2026: 79,084 scored forecasts, 2,334 with a spill.
Compared with
The live forecast with the map in use today.
How it did

No verdict yet: the window decides. The live rows before it, and 2025's records, point in opposite directions.

ResultThis modelReferenceDifference [95% interval]
2025 replay, leads 0 to 4 togetherBrier, held out. No interval was worked out for the replay.0.038180.03941 today's map
Live forecasts, 29 Sep to 16 Oct 2026Brier, before its window: not deciding. Before the window: reported, not deciding.0.030990.02721 today's map
Same rowsECE, before its window: not deciding.0.0320.0117 today's map
What it misses
Fitted on 2025's records, where the forecast ran too low; on the live rows the forecast already runs too high at six of nine companies, and B raises it further.
How often it was wrong
On the live rows before the window its mean chance was 0.0615 against 0.0295 observed (today's map: 0.0393).
Is a result shown?
No. Nothing is deployed unless all three conditions hold at the read.
What is logged
Nothing extra is logged: at the read the map is applied to the live forecast's own published scored rows.
The rule at the read
All three: lower Brier than today's map; no company with 50 or more spills more than 2% worse; calibration error no worse. At least 200 spills.
Records
Lead calibration B test

Company calibration (A8)

Running test: window 26 Oct 2026 to 22 Nov 2026. Read on or after 25 Nov 2026; no score from the window is shown before then.

What it predicts
The live forecast's daily spill chances with a frozen correction for each overflow at the eight companies other than United Utilities, fitted on SwimSignal's own polls.
Where and when it was tested
Fitted on 20,783 same-day rows, target days 29 September to 23 October 2026, 1,010 overflows. No score was worked out on the fitting rows. The corrections are frozen in a file with a SHA-256 beginning 2662dcfa.
Compared with
The live forecast and a flat rate per company.
How it did

Not yet run. The prediction written before the window was that it will not be adopted.

No score yet.
What it misses
Overflows that never spilled get corrections only down to 0.73 of the forecast, not lower. United Utilities is not changed.
How often it was wrong
Background, before the window: the live forecast's Brier over the eight companies was about 9% above a flat rate's (Thames about 57%, Southern 41%, Wessex 23%).
Is a result shown?
No. Never deployed before the read, and only then if the conditions hold.
What is logged
Nothing extra is logged: at the read the frozen corrections are applied to the live forecast's own published scored rows.
The rule at the read
Shown better than both the live forecast and the flat rate over the eight companies (95% interval wholly below zero), and no changed company clearly worse. Too few spills at a changed company: 14 days more, to 6 December, read on or after 9 December.
Records
Company calibration test

Ensemble rain (D4)

Running test: window 23 Oct 2026 to 19 Nov 2026. Read on or after 21 Nov 2026; no score from the window is shown before then.

What it predicts
The spill forecast run on each of three members of the Met Office's MOGREPS-UK ensemble rain forecast, giving a range around the headline chance two to four days ahead.
Where and when it was tested
Not tested on past data: Open-Meteo does not keep the ensemble's past runs, so a backtest is not possible. The live window is the first test.
Compared with
The single rain forecast the headline uses now.
How it did
No score yet.
What it misses
Three members are a narrow sample of the weather. Before the window, the headline sat inside the members' range on about 70% of rows two days ahead and 65% four days ahead.
How often it was wrong
Not yet measured.
Is a result shown?
Yes. The range is shown, labelled as under test; the level and the headline chance do not use it.
Rows logged so far
1,387,920 rows from 18 Oct 2026 to 31 Oct 2026. A count only: no score is worked out from them before the read.
The rule at the read
Whether a wider range marks a larger error, and whether the members' mean is no worse than the single forecast, overall and at each lead and company. Too few spill rows: runs to 3 December, read on or after 5 December.
Records
Ensemble rain test

Upgraded overflows, option B

Running test: window 29 Sep 2026 to 30 Nov 2026. Read on or after 2 Dec 2026; no score from the window is shown before then.

What it predicts
The daily spill chance with each overflow's own correction fitted only on its record after a start date, so an overflow that was upgraded and went quiet is not judged on its old spills.
Where and when it was tested
Not yet scored. Two windows are fixed: the Environment Agency's records for 1 January to 31 August 2026, same-day; and SwimSignal's own live forecasts for target days 29 September to 30 November 2026, leads 0 to 4.
Compared with
The live forecast, and the same correction fitted on the whole record (B0).
How it did
No score yet.
What it misses
A broken monitor that reports no spills with fresh time stamps passes the dead-monitor check. The 16 Ilkley overflows are left out. The live window is autumn, when the forecast has run too high.
How often it was wrong
Not yet measured.
Is a result shown?
No. Not before the read. A pass leads to a separate running test early in 2027, not straight to the forecast.
What is logged
Nothing extra is logged: at the read the arms are rebuilt from the live forecast's logged raw chances.
The rule at the read
Seven conditions, all needed, including a gain on the 2026 records with the interval below zero, no loss over 1% in the live window, and no company clearly worse.
Records
Upgraded overflows test B, Recency note

Winter recalibration

Scheduled: its window runs 14 Dec 2026 to 7 Mar 2027, read on or after 9 Mar 2027.

What it predicts
The live forecast's chances with a fixed correction after the lead map, by lead and possibly by company, fitted on the live autumn rows to bring the chances down where they run too high.
Where and when it was tested
An earlier correction on 12 days of live rows (58,274 rows) was 4.3% better against a 5% bar. A later check on 117,324 live rows, 29 September to 23 October, found the chances run too high overall. The new correction is fitted on or after 10 December to target days 29 September to 6 December.
Compared with
The live forecast as shown.
How it did

The earlier correction missed its 5% bar. The winter test is its successor, with a different route written down before any of its rows.

ResultThis modelReferenceDifference [95% interval]
Earlier correction, company intercepts, 12 days of live rowsBrier, live record. Scored leaving one day out at a time; no interval.0.021820.0228 live forecast
What it misses
Fitted on autumn, judged on winter: whether the same bias holds in summer is not tested.
How often it was wrong
In the live check, of same-day forecasts in the highest band, 388 of 963 came true (40%).
Is a result shown?
No. Not before the read. It stays a labelled secondary for 2027 and never replaces the frozen 2027 evaluation's predictions.
What is logged
Logs a separate corrected chance from the fit, never shown before its rollout review.
The rule at the read
At least 1% better with the interval above zero, or no more than 1% worse with a shown calibration gain; no company clearly worse. If too few spills: 16 weeks, to 4 April 2027, read on or after 6 April.
Records
Winter recalibration plan, Calibration check, Earlier recalibration test

Spill duration

In the lab: not good enough yet to run beside the forecast. Next, 1 Dec 2026: the same check of the table on November's polls.

What it predicts
For an overflow spilling now, the chance it is still spilling 1, 3, 6, 12 and 24 hours later. When a spill ends is not when the water clears.
Where and when it was tested
Trained on 2024 and scored on 2025: 1,582,953 hours of spills at the nine companies. Then a replay of 2024-25 at the times SwimSignal actually polls, and a check of the 2024-25 table on SwimSignal's own polls, 28 September to 31 October 2026.
Compared with
B1: the share of a company's spills still running after the same elapsed time.
How it did

The model failed its reliability condition on 2025, and the calibration gate that followed also failed. The 2024-25 table is shown as a record, not a forecast.

ResultThis modelReferenceDifference [95% interval]
2025, all horizonsBrier, held out. 11.6% better, but its chances were unreliable.0.15070.1705 B1, company table−0.0198 [−0.0242, −0.0148]
2024-25 replay at the live poll times, calibratedBrier, seen: a direction only. Development on seen data: a direction, not a verdict.0.16520.1954 B1, company table−0.0302 [−0.0319, −0.0286]
What it misses
It under-forecasts at 24 hours. On the live polls the 2024-25 table's curve was too flat for autumn 2026: too low at 1 hour, too high at 12 to 24 hours.
How often it was wrong
On 2025 its reliability failed in 24 of the 45 company-and-horizon cells. In the replay, 19.4% of rows were spills that had already stopped.
Is a result shown?
No. No forecast of when a spill ends is shown. The 2024-25 durations are shown as a record of the past on the spill durations page.
Records
Duration test, Calibration test, Replay, Live check

The 2027 evaluation

Not started. Next, 1 May 2027: the model is frozen and tagged before the season.

What it predicts
Not a new model: the live forecast, frozen before the season, scored once over the whole 2027 season. Four results: spill forecasts same-day and one to four days ahead, and the E. coli estimate on rivers same-day and ahead.
Where and when it was tested
Target days 1 May to 30 September 2027, at the 1,089 overflows that reach the 105 listed spots, and about 340 river samples.
Compared with
Each overflow's spill rate adjusted for the month; each bathing water's own share of samples over 900.
How it did
No score yet.
What it misses
Each result shows skill only if the lower end of its 95% interval is above 0. Candidates are reported separately and never replace the frozen model's result.
How often it was wrong
Not started.
Is a result shown?
No. No result until the analysis runs after the season. A trial run on autumn 2026 rows checked the code; it is not a result.
Records
2027 preregistration

How a model reaches the forecast

  1. It is tested on past data, under rules written down before the scores are seen. A test that fails stays on record.
  2. It runs beside the live forecast for a fixed window, saving its own answers before anyone knows what the overflows will do. Nobody looks at its score until the agreed read date.
  3. On that date it is scored against the live model and a simple reference. A model that cannot beat them stays in the lab.
  4. Only then can a change to the forecast be proposed, in its own change, with a way back.

Sources