Predicting sepsis six hours before the clinician does

Generated by sepsis evaluate. Every number below comes from data the model in question never saw during fitting or tuning: hyperparameters and blend weights were selected on the validation split, the hospital A test split was scored once, and hospital B was untouched until this table.

Cohort

Headline results

Hospital A — held-out test

Hospital B — external validation, never seen in any form

Best model by clinical utility: ensemble — test AUROC 0.819 (95% CI 0.797–0.840, patient-level cluster bootstrap), normalised utility 0.410. On hospital B the same frozen model and threshold give AUROC 0.802 and utility 0.257.

Are the differences real?

DeLong test for correlated ROC curves — the models score identical rows, so an unpaired comparison would be needlessly conservative.

What the search actually bought

The Optuna search ran 60 trials (20 pruned) against 3-fold grouped cross-validation, reaching a best out-of-fold utility of 0.4483. The same configuration scores 0.3939 on the held-out test split.

That +0.0544 gap is selection optimism, and it is expected: taking the maximum over 60 trials of a noisy, piecewise-constant objective selects partly for luck. It is reported rather than hidden because the CV number is the one that would be quoted if nobody had held a test set back — which is exactly why one was held back.

What this looks like at the bedside

detection_rate counts a septic admission as caught only if the first alert lands before clinical onset; an alert raised after the team already suspected sepsis is not an early warning.

One admission at a time

A median lead time of 36 hours is a fact about 268 admissions, and it does not answer the question a clinician asks first, which is what this thing would have done in front of one patient. Below, four admissions replayed hour by hour at the frozen operating point: the risk the model produced, when it crossed the alert threshold, and when the care team actually started acting.

The cases come from the validation split, never from test. Validation was already spent on tuning, calibration and threshold selection, so nothing is lost by looking at it again. Choosing illustrative cases from the test split would quietly convert a held-out set into a presentation set. It also means the probabilities on screen are in-sample for the isotonic map, which was fitted on this split: the replay is about timing, not about calibration quality.

The headline case is the median, not the best. p014990 alerts 36 hours before onset because 36 hours is where the median of the 225 caught admissions sits. Picking the largest lead time in the cohort would have been easy and would have described nothing. Two of the four cases are failures for the same reason: p006176 alerts 3 hours after the team was already acting, which is not an early warning, and p018494 is a 25-year-old who never became septic and sat above the threshold for 56 of 58 hours. p002551 is the third kind of case worth seeing: caught, with 1 hour of warning, which is a catch by the metric and close to useless at the bedside.

Median catchp014990Only justp002551Missedp006176False alarmp018494

Play

Validation split, xgboost at the frozen threshold of 0.0308. Ticks under the axis mark hours when a sparse lab was drawn.

Lab draws are marked on the same timeline as the risk. The feature-block ablation found that ordering behaviour alone reaches 97% of the full matrix's AUROC, so what the team chose to measure belongs next to the risk it produced.

Experiments

Each experiment changes exactly one thing, refits, and reports what that change was worth. Every one of them asserts its own invariants before returning, so a variant that failed to commit the mistake it was studying raises rather than publishing a plausible wrong number.

What leakage is worth, measured

Splitting ICU hours at random instead of splitting admissions inflates AUROC by +0.0275 (0.821 to 0.848). It is not an exotic bug: it is what train_test_split does to a dataframe of rows, and it puts 17,236 of 17,285 admissions on both sides of the split at once. Filling gaps from the future with a single bfill inflates AUROC by +0.0056. Everything else is held fixed across the three rows: same hyperparameters, same boosting rounds, same seed, and for variant B the same admissions in train and test as the baseline.

The more useful number is the one underneath. Clinical utility inflates by +0.0796 and +0.0478 for the two mistakes, roughly 3x and 9x the corresponding AUROC movement. Leakage does not merely make the ranking look better; it moves the whole risk distribution, so the threshold sweep finds an operating point that does not exist on honest data. A project reporting AUROC alone would see a fraction of the damage it is actually doing.

What each feature block buys

With all 345 features the model scores AUROC 0.8206. No single block costs more than 0.0080 AUROC to remove — the most expensive is recency (hours since each channel was last measured). Yet every block scores between 0.720 and 0.782 on its own. That combination has one explanation: the matrix is enormously redundant, and the same physiology is reachable through several different encodings of it.

The sharpest number here is the ordering-only row. Using 109 features that contain no measured value whatsoever — only which channel was sampled, how recently, and how often — the model reaches AUROC 0.7943, or 97% of what the full matrix achieves. Nothing about the patient's physiology is in that subset. It is a record of what the care team chose to look at, and it is nearly as predictive as the measurements themselves.

Two readings follow. The optimistic one: missingness is signal, and the recency and intensity blocks earn their place rather than padding the matrix. The uncomfortable one: a model this dependent on ordering behaviour is partly learning clinical suspicion rather than physiology, so it would degrade wherever ordering habits differ — which is exactly what the drop from hospital A to hospital B looks like.

Case mix or degradation: decomposing the hospital B drop

At the operating point frozen on hospital A's validation split (threshold 0.519, applied unchanged everywhere), hospital A scores 0.4243 normalised utility and hospital B scores 0.2946: a gap of 0.1297 (95% CI 0.0655 to 0.1817). Hospital B's admissions are reweighted to hospital A's baseline case mix -- demographics, unit, early vitals and first-six-hour ordering volume -- by a propensity density ratio trimmed at the 1st and 99th percentiles. That weighting cuts mean covariate imbalance from 0.220 to 0.075 |SMD| and retains an effective 53% of 20,000 admissions.

Reweighted, hospital B scores 0.3120. So of the 0.1297 gap, +0.0174 is case mix (95% CI -0.0041 to +0.0419) and +0.1123 is what remains after adjustment (95% CI +0.0450 to +0.1724) — most of the gap survives the adjustment, and the case-mix component is not separable from zero. Both intervals come from a 200-replicate patient-level cluster bootstrap that refits the propensity model on every replicate, so the uncertainty of the reweighting is inside the interval rather than assumed away.

That component is not robust to how the adjustment is specified, which is itself the result. Matching to hospital A's test split rather than its training split gives +0.0206; using only covariates fixed at admission — dropping the six hours of early vitals and ordering volume, which are care process and plausibly mediators of the site difference being adjusted for — gives +0.0056. The pre-registered specification is the one reported above; these are reported beside it because a number that moves when a defensible choice moves belongs in the open as a range.

Two cautions on reading the second number as degradation. The adjustment can only correct for the 17 covariates it was given, so anything that differs between two health systems and is not in that list — treatment practice, labelling behaviour, unmeasured acuity — stays in the residual. And reweighting on baseline covariates does not force the septic rate to match: it moves from 5.7% to 6.7% against hospital A's 8.8%. The residual is an upper bound on degradation, not a measurement of it.

One candidate explanation can be ruled out without any modelling. The threshold is frozen at hospital A's, by design, because that is what deploying a model means, and a frozen threshold is exactly the kind of thing that stops working at a new site. Here it does not: re-picking the threshold on hospital B alone moves its utility from 0.2946 only to 0.3006, 5% of the gap. Hospital A's threshold is already very nearly the right threshold for hospital B, so the loss is in the scores themselves, not in where the line is drawn. The MICU-to-SICU experiment in the next section runs the same check across a different boundary and gets the opposite answer, which is the useful pairing: a model can cross a boundary intact while its operating point does not, and it can carry its operating point across intact and still lose most of its value.

MICU to SICU: a shift this cohort can actually support

Trained on 2,727 medical ICU admissions and evaluated at a threshold frozen on held-out MICU patients (0.445), the model scores AUROC 0.8046 (95% CI 0.7794 to 0.8348) on MICU admissions it has not seen and 0.7810 (0.7477 to 0.8183) on surgical ICU admissions. The difference is +0.0236 with a 95% interval of -0.0133 to +0.0635, from an independent two-sample cluster bootstrap — DeLong does not apply here, because the two cohorts are different patients rather than two scores on the same rows. So the interval on that difference spans zero, so this experiment does not establish a ranking loss across the unit boundary. That is not the same as showing there is none: with 4,650 SICU admissions it is a statement about what this test could detect.

Clinical utility is a different story, and it is measured on a held-out half of each bucket so that no threshold is graded on the admissions it was chosen from. At the MICU threshold the model scores 0.4687 at home against 0.0884 in the SICU (95% CI -0.0592 to 0.1883). Choosing a threshold on a separate half of the SICU and scoring it here recovers +0.3193 to 0.4077 (95% CI on the gain: +0.2433 to +0.4439, which excludes zero).

Two cautions on reading that as a free fix. Choosing a local threshold requires this unit's own labelled outcomes — utility is optimised against them — so it costs local outcome data, not merely a sweep. And the normalised utility score has a cohort-specific, outcome-informed denominator, so part of the movement between units is the metric responding to a septic rate of 10.8% against 4.0%, not the model behaving differently. What the score distribution actually does across the boundary is directly visible: mean predicted risk 0.3415 in the MICU against 0.3352 in the SICU, alerting on 76% against 75% of admissions at the same threshold.

The third bucket is the one it would be convenient to omit: 8,090 admissions where neither unit indicator was recorded — more than either named unit — scoring AUROC 0.7849 at 10.4% prevalence. Dropping them would have made the transfer claim cleaner and the cohort unrepresentative of the hospital it came from.

Is ordering behaviour what fails to transfer?

The feature-block ablation showed that 109 features containing no measured value reach most of the full matrix's AUROC. The obvious reading is that the model is partly learning clinical suspicion, which would explain why it transfers poorly to a hospital with different charting habits. That reading is a story that fits the numbers, so it is tested here rather than repeated.

Two models, identical but for the 109 withheld features. With everything, AUROC falls 0.8229 to 0.7868 across the hospital boundary, a gap of 0.0361. Without ordering behaviour, it falls 0.8174 to 0.7921, a gap of 0.0254.

The mechanism story survives its own test. Withholding ordering behaviour shrinks the transfer gap by 0.0107 AUROC (95% CI +0.0004 to +0.0205), so a meaningful part of what fails to cross the boundary is the model's dependence on what the care team chose to measure. The interval is tight against zero, so this is a real effect rather than a large one: it establishes the direction, not the size.

Either way the price of removing them is visible in the first column: hospital A performance drops from 0.8229 to 0.8174. Ordering behaviour is not noise to be regularised away — it is real signal about a real thing, which is precisely why its portability is worth knowing.

Statistical analysis

245 of 331 features separate septic from non-septic admissions at FDR < 0.05 (Benjamini-Hochberg over Welch tests). Tests are run on one summary value per admission, not per ICU hour: hours within a stay are strongly autocorrelated, and treating them as independent inflates significance by orders of magnitude.

Missingness is not noise

Whether a lab was ordered — ignoring its value entirely — separates the groups on its own. That is why the feature matrix carries recency and ordering-intensity channels rather than imputing them away.

Logistic regression, interpreted

Refit without penalty on the collinearity-pruned matrix, because Wald standard errors do not apply to penalised coefficients. Odds ratios are per standard deviation of the feature.

An elastic-net fit (L1-dominant) reduces the matrix to 45 non-zero coefficients — a candidate short list for a bedside score.

Model behaviour

Calibration

Class weighting buys ranking quality at the cost of the probability scale. Isotonic maps were fitted on validation and are scored here on the held-out test split — reporting them on validation would be circular, since in-sample isotonic ECE is 0 by construction. Discrimination is essentially unchanged while Brier and ECE move substantially.

What each model contributes

Blend weights were optimised against validation utility and then frozen. marginal_contribution is what the blend loses if that member is removed — a large weight with a small ablation cost means the member is redundant.

What drives the booster

Cross-site shift

78 of 345 features shift materially (PSI > 0.25) between the two hospitals, and admission-level sepsis prevalence falls from 8.8% to 5.7%. The external column in the headline table is the cost of that shift, measured rather than assumed.

Figures