Target validity matters more than headline predictive accuracy.
AI value · Black share identified for extra care
Verified
46.5% if ranked by actual health need
Counterfactual repair in one system; vendor disputed use characterization.
Before
Has a limited number of intensive nursing and coordination slots to allocate. → Use clinical knowledge to identify patients with complex needs.
After
Ranks patients by predicted future cost as a proxy for health need. → Use the score among selection inputs and allocate scarce extra care.
Human boundary
Programs own enrollment; clinicians retain judgment.
Why it matters
A predictable proxy can still allocate scarce care unjustly.
How the work changed
Before
How the work ran before the change.
Step 1 of 2
Care-management program
Has a limited number of intensive nursing and coordination slots to allocate.
ControlThe Science study does not provide a dated pre-algorithm workflow; this is a bounded resource-allocation baseline.
Step 2 of 2
Clinicians and program staff
Use clinical knowledge to identify patients with complex needs.
ControlHuman judgment, constrained by program capacity.
What changed
A predictable proxy can still allocate scarce care unjustly.
Decision rightHuman authority remains at the consequential boundary
After
How the same work runs now.
Step 1 of 2
Commercial risk algorithm
Ranks patients by predicted future cost as a proxy for health need.
ControlTop 97% risk threshold at the studied site; proxy choice embeds unequal access and spending.
Step 2 of 2
Program staff and clinicians
Use the score among selection inputs and allocate scarce extra care.
ControlClinicians can supplement the score, but scaled allocation is score-shaped; health-need counterfactual exposed bias.
Process model built from the published workflow evidence for Large US health system / Optum algorithm. Every step, actor, and control appears in full below.Every step, actor, and control
Exception path
Clinical judgment can add patients, but the queue is score-shaped.
Work removed
Some manual population prioritization
Decision authority
Programs own enrollment; clinicians retain judgment.
Before
01
Care-management program
Has a limited number of intensive nursing and coordination slots to allocate.
Control: The Science study does not provide a dated pre-algorithm workflow; this is a bounded resource-allocation baseline.
02
Clinicians and program staff
Use clinical knowledge to identify patients with complex needs.
Control: Human judgment, constrained by program capacity.
After
01
Commercial risk algorithm
Ranks patients by predicted future cost as a proxy for health need.
Control: Top 97% risk threshold at the studied site; proxy choice embeds unequal access and spending.
02
Program staff and clinicians
Use the score among selection inputs and allocate scarce extra care.
Control: Clinicians can supplement the score, but scaled allocation is score-shaped; health-need counterfactual exposed bias.
Work that left the path
Some manual population prioritization
Human role before
Clinicians and care-program staff used clinical knowledge and limited program capacity to select patients for intensive care management; the study does not provide a dated pre-algorithm queue.
Human role after
Program teams set enrollment thresholds and clinicians may supplement the score.
AI roleRanks patients by predicted future cost as a proxy for need.
Outcomes
Black share identified for extra care
Verified
17.7% under cost proxy→46.5% if ranked by actual health need
Published 2019 · 43,539 White and 6,079 Black patients
Counterfactual repair in one system; vendor disputed use characterization.
What leaders can reuse
Anti-pattern
Do not call cost prediction inaccurate; the target was wrong.
Questions
01Where is the operating threshold set and who can override it?
02What measured result would trigger rollback or retraining?
03Which residual decisions must remain human-owned?
Portability conditions
Need-based targets
Subgroup audits
Override capacity
Reputation risk
high
Evidence and authority
What the public record supports.
Current · updated
1 independent, 1 peer reviewed; publication outcomes are verified.
Bundle 1.0.0 · reviewed 2026-08-23 · stable ID 81c7b1bc482d6d8f
