What is the impact of predictive AI in the health care setting?
Models built on machine learning in health care can be victims of their own success, according to researchers at the Icahn School of Medicine and the University of Michigan. Their study assessed the impact of implementing predictive models on the subsequent performance of those and other models. Their findings — that using the models to adjust how care is delivered can alter the baseline assumptions that the models were “trained” on, often for worse — were detailed in the October 9 online issue of Annals of Internal Medicine.
“We wanted to explore what happens when a machine learning model is deployed in a hospital and allowed to influence physician decisions for the overall benefit of patients,” says first and corresponding author Akhil Vaid, M.D., Clinical Instructor of Data-Driven and Digital Medicine (D3M), part of the Department of Medicine at Icahn Mount Sinai. “For example, we sought to understand the broader consequences when a patient is spared from adverse outcomes like kidney damage or mortality. AI models possess the capacity to learn and establish correlations between incoming patient data and corresponding outcomes, but use of these models, by definition, can alter these relationships. Problems arise when these altered relationships are captured back into medical records.”
The study simulated critical care scenarios at two major health care institutions, the Mount Sinai Health System in New York and Beth Israel Deaconess Medical Center in Boston, analyzing 130,000 critical care admissions. The researchers investigated three key scenarios:
1. Model retraining after initial use
Current practice suggests retraining models to address performance degradation over time. Retraining can improve performance initially by adapting to changing conditions, but the Mount Sinai study shows it can paradoxically lead to further degradation by disrupting the learned relationships between presentation and outcome.
2. Creating a new model after one has already been in use
Following a model’s predictions can save patients from adverse outcomes such as sepsis. However, death may follow sepsis, and the model effectively works to prevent both. Any new models developed in the future for prediction of death will now also be subject to upset relationships as before. Since we do not know the exact relationships between all possible outcomes, any data from patients with machine-learning influenced care may be inappropriate to use in training further models. More