Case FileTX-CRIS · 2017–2025 · ACCEPTED, HICSS-60

Toward Auditable & Calibrated AI for Dementia-Related Crash Severity

A selective deferral framework that reframes crash severity modeling as decision-aware triage — controlling outcome leakage, calibrating confidence, and routing uncertain cases to human review instead of maximizing accuracy alone.

Gaurab Chhetri · Dept. of Computer Science · Anika Baitullah · Ingram School of Engineering · Subasish Das, Ph.D. · Ingram School of Engineering

TEXAS STATE UNIVERSITY, USA

Read the Paper Cite this Work Try the Gauge

SEVERITY CODE MIXN=4,781

45%

10%

0Crash records

0Top macro-F1

0Calibration error

0Cost @ 70% coverage

§01

Overview

Triage, not accuracy maximization

Crash severity models rarely control what data was available at decision time, rarely report how badly they under-triage severe cases, and rarely say when they should defer to a person instead of guessing.

This study reframes dementia-related crash severity modeling as a decision-aware triage problem: classify each crash into no-injury/property-damage-only (O), minor or moderate injury (BC), or fatal/severe injury (KA) — while controlling outcome leakage, reporting severe under-triage, calibrating confidence, and preserving every raw prediction for audit.

RQ1

Do police narratives improve KA / BC / O classification beyond structured variables alone?

RQ2

How much performance survives after removing structured severity leakage and masking outcome-revealing narrative terms?

RQ3

Does calibrated deferral reduce severe under-triage — especially KA crashes classified as BC or O?

Leakage control

22 seed terms + up to 75 corpus-derived phrases masked from narratives before evaluation.

Calibrated uncertainty

Temperature scaling drops expected calibration error from 0.089 to 0.033 without changing labels.

Selective deferral

Withholding low-confidence cases raises retained-case macro-F1 from 0.522 to 0.573 at 70% coverage.

Full audit trail

Every split, prompt, response, probability, and log preserved for traceable, reproducible review.

§02

System Design

A six-layer decision architecture

Click a stage to see what happens at that layer. The same architecture underlies both the "full" and "leakage-controlled" evaluation settings.

Data Ingestion

Leakage Control

Representation Learning

Probability Prediction

Severity-Aware Rules

Audit Logging

01 · Data Ingestion — Texas CRIS crash records (2017–2025, ~5M records) are searched with predefined dementia-related terms, then manually reviewed for sufficient narrative evidence. The final analytical dataset retains 4,781 crashes with 25 structured predictors plus narratives.

Cost-sensitive learning

Separates probability estimates from decision rules under unequal error costs (Elkan, 2001) — KA→O errors cost 5×, KA→BC cost 3×.

Selective classification

Allows abstention when confidence is insufficient (Geifman & El-Yaniv, 2017), routing uncertain crashes toward human review.

Public-sector accountability

Deployment introduces accountability obligations (Busuioc, 2021) — model size affects who can run, audit, and update the system.

§03

Data & Experimental Design

4,781 Texas dementia-related crash records

Dementia status is crash-record reported, not a clinically verified diagnosis. A fixed stratified 70/15/15 split is used throughout.

Stratified split

3,346 TRAIN · 717 VALIDATION · 718 TEST (INCL. 73 KA)

Train3,346 · 70%

Validation717 · 15%

Test718 · 15%

25 structured predictors span roadway/environment, crash event, time/area, vehicle, and person factors. Crash ID, year, raw severity, and the target were excluded from predictors.

Why accuracy fails here

MAJORITY-CLASS BASELINE, FULL TEST SET

Accuracy0.532

KA share of test set73 / 718 · 10.2%

A model that always predicts the majority class reaches respectable accuracy without ever separately identifying a severe (KA) crash — the central reason this paper reports macro-F1, KA recall, and severity cost together rather than accuracy alone.

Note — bins were fixed without data-dependent estimation (e.g., speed ≤30/35–50/55–80 mph; age 15–30/30–50/51–65/≥65). The retained audit manifest includes run configuration, data snapshots, split indices, trained models, logits, LLM prompts/responses, predictions, calibration outputs, deferral tables, and run logs.

§04

Main Results

Structured, narrative, fusion, and LLM baselines

Evaluated on the fixed stratified test set (718 records, 73 KA). Click column headers to sort. "Full" retains all modeling inputs; "Leakage" additionally masks outcome-revealing narrative language.

Best macro-F1: Gemma (leakage-controlled) — 0.545

Best KA recall: Qwen — 0.671

Best calibration: Calib. late fusion — ECE 0.033

95% CI (Gemma F1): [0.507, 0.583]

Macro-F1, KA recall, and severity cost across models

LEAKAGE-CONTROLLED SETTING · LOWER COST IS BETTER; HIGHER F1/RECALL IS BETTER

Table 3 · Main stratified test results
Setting	Model	Acc.	Macro-F1	KA Prec.	KA Rec.	KA Under	Cost	Brier	ECE
Full	Late fusion	0.599	0.539	0.295	0.425	0.575	0.652	0.534	0.112
Full	Calibrated late fusion	0.599	0.539	0.295	0.425	0.575	0.652	0.515	0.039
Full	Early fusion logistic regression	0.579	0.533	0.300	0.493	0.507	0.663	0.524	0.017
Full	Narrative logistic regression	0.588	0.516	0.282	0.329	0.671	0.673	0.535	0.084
Leakage	Gemma	0.578	0.545	0.315	0.630	0.370	0.659	0.607	0.192
Leakage	Early fusion logistic regression	0.577	0.522	0.278	0.438	0.562	0.678	0.526	0.029
Leakage	Calibrated late fusion	0.584	0.522	0.280	0.384	0.616	0.675	0.520	0.033
Leakage	Narrative logistic regression	0.585	0.502	0.235	0.274	0.726	0.692	0.538	0.079
Leakage	Structured CatBoost	0.489	0.452	0.220	0.562	0.438	0.852	0.601	0.040
Leakage	Qwen	0.460	0.390	0.216	0.671	0.329	0.864	0.771	0.300
Leakage	DistilBERT	0.444	0.380	0.122	0.233	0.767	0.937	0.619	0.024

Note — relative to early fusion, Gemma's paired bootstrap differences include zero for macro-F1 (0.022 [-0.026, 0.070]) and cost (-0.019 [-0.102, 0.065]) — but the KA-recall difference (0.192 [0.041, 0.329]) does not include zero. Gemma obtains the highest observed macro-F1 in this split, but the results do not establish general model superiority.

§05

Leakage Audit

What happens when outcome-revealing language is masked?

Narratives are lowercased and masked at token boundaries with [MASK_SEVERITY] — 22 seed expressions plus up to 75 corpus-derived phrases with strong one-vs-rest log-odds for O, BC, or KA.

Macro-F1 drop after masking

TEXT-DEPENDENT MODELS LOSE THE MOST SIGNAL — BUT PERFORMANCE DOES NOT COLLAPSE

Table 4 · Leakage audit for models evaluated in both settings
Model	Full F1	Leak F1	F1 drop	Full KA Rec.	Leak KA Rec.
Late fusion	0.539	0.522	0.018	0.425	0.384
Narrative logistic regression	0.516	0.502	0.014	0.329	0.274
Early fusion logistic regression	0.533	0.522	0.010	0.493	0.438
Structured CatBoost	0.452	0.452	0.000	0.562	0.562

Note — Structured CatBoost is unchanged because current structured inputs never included direct severity fields — the intervention only affects narrative-dependent models. High-support masked phrases include multiword patterns like "were transported," "university hospital," and "for possible injuries," which seed-only masking would miss.

§06

Selective Deferral · Interactive

Turn the coverage dial to see the tradeoff

Calibrated late fusion ranks records by confidence. Lowering coverage withholds more low-confidence cases from automatic classification — retained-case performance rises, but so does the human-review burden.

100%

Coverage

708090100

0.522

Macro-F1

0.384

KA Recall

0.675

Severity Cost

Deferred (KA)

Fully automated classification; no review burden, but all uncertainty remains in the automatic labels.

Table 5 · All selective deferral operating points for calibrated late fusion
Coverage	Classified	Deferred	Deferred KA	Macro-F1	KA Rec.	KA Under	Cost
100%	718	0	0	0.522	0.384	0.616	0.675
90%	646	72	7	0.544	0.379	0.621	0.635
80%	574	144	16	0.560	0.368	0.632	0.601
70%	503	215	25	0.573	0.396	0.604	0.577

Note — metrics below 100% coverage are calculated only on retained automatic predictions. Deferred cases are not further evaluated in this experiment — they represent candidates for a proposed human-review process; reviewer accuracy and workload are not measured here.

§07

Behavioral Audit

Where the strongest model actually fails

Aggregate metrics hide boundary-case behavior. The leakage-controlled confusion matrix for Gemma (highest macro-F1) shows its main failure mode.

PRED. O

PRED. BC

PRED. KA

TRUE O

156

59.3%

34.6%

6.1%

TRUE BC

22.3%

213

55.8%

22.0%

TRUE KA

11.0%

26.0%

63.0%

Gemma under-triages 27 of 73 true KA records (8 as O, 19 as BC) while also over-predicting KA for many non-KA cases — a more severe-case-sensitive operating point rather than uniform improvement, better suited to screening than fully automated classification.

Early–late fusion agreement

83.3% label agreement, but mean absolute probability distance of 0.084 — late fusion mainly changes boundary-case uncertainty.

Confidence gap

Correct calibrated-fusion predictions average confidence 0.622, vs. 0.562 for errors — supporting the deferral rule.

Cost-matrix sensitivity

Comparing lower-asymmetry, baseline, and higher-severe-miss cost scenarios changes cost by −11.2% to +9.5% for direct prediction, but the 80%-coverage deferral benefit stays directionally stable (−9.2% to −12.3%) across all three.

§08

Limitations & Future Work

Reading the results with appropriate caution

Dementia status is record-reported, not clinical+

Candidate cases were identified via high-recall term search plus manual contextual review of narratives — this is crash-record-reported information, not a clinically verified diagnosis.

Leakage terms are dataset-specific+

The masking procedure targets identified outcome-revealing terms and corpus-derived phrases, but resulting inputs are "leakage-controlled" rather than provably free of all outcome proxies.

Single split, no repeated-split uncertainty+

Model rankings are based on one fixed stratified 70/15/15 split and are presented without confidence intervals for repeated-split variation — they are conditional on this evaluation split.

Confidence-only deferral rule+

The deferral rule uses calibrated confidence alone. Future work should combine confidence, expected severity cost, class-specific uncertainty, and analyst capacity in deferral policy.

No reviewer-in-the-loop evaluation+

Deferred cases are treated as candidates for human review but are not further evaluated — reviewer accuracy, workload, disagreement, response time, and post-review outcomes are not measured in this study.

Small, specialized, imbalanced dataset+

Transformer baselines used a relatively small imbalanced dataset; local LLM results depend on deterministic prompting and model availability. Generalizability to broader older-driver or medical-condition crash populations may be limited.

§09

Citation

BibTeX

Accepted at HICSS-60. Preprint live on arXiv — this entry will be updated with the DOI once the proceedings version is published.

@misc{chhetri2026auditablecalibratedaidementiarelated,
      title={Toward Auditable and Calibrated AI for Dementia-Related Crash Severity Prediction: A Selective Deferral Framework to Support Human Review},
      author={Gaurab Chhetri and Anika Baitullah and Subasish Das},
      year={2026},
      eprint={2609.22694},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.22694},
}