The evidence ladder: from synthetic data to the bedside.
The five rungs every Healytix clinical model must climb, and why a strong benchmark on synthetic patients is the first rung, not the last.
Benchmarks are not outcomes
A model can score well on held-out synthetic patients and still fail on real ones: different equipment, different coding habits, different populations. Clinical AI has a long record of models that looked strong in development and degraded at a new site. Validation has to be staged.
Rungs one and two
Development starts on synthetic cohorts and published evidence, which proves the pipeline and the feature engineering. The second rung tests on de-identified retrospective records from partner hospitals, reporting performance by age, sex and site rather than as one headline average, so weak subgroups are visible.
Rungs three and four
Third, the model runs silently on live data: predictions are logged and compared with outcomes, but no clinician sees them. Fourth, outputs reach clinicians as suggestions with cited evidence, and every accept, edit and override is recorded and fed back into evaluation.
Rung five
Models that make clinical claims go through the applicable regulatory pathway, such as CDSCO software-as-a-medical-device rules in India or FDA clinical decision support guidance in the US. Calibration and drift monitoring continue after approval, because populations and practice change.
Key takeaways
- 1Report results by subgroup and site, not only as an average.
- 2Shadow mode is cheap and catches problems before clinicians see them.
- 3Know which rung each model is on, and say so.