February 1, 2019

Medicine as a data science (3)

High-performance medicine: the convergence of human and artificial intelligence

If you want to know the current state of artificial intelligence in medicine, then Eric Topol review in Nature is the article you have to read. A highlighted statement:
There are differences between the prediction metric for a cohort and an individual prediction metric. If a model’s AUC is 0.95, which most would qualify as very accurate,
this reflects how good the model is for predicting an outcome, such as death, for the overall cohort. But most models are essentially classifiers and are not capable of precise prediction at the individual level, so there is still an important dimension of uncertainty.
And this is good summary:
Despite all the promises of AI technology, there are formidable obstacles and pitfalls. The state of AI hype has far exceeded the state of AI science, especially when it pertains to validation and readiness for implementation in patient care. A recent example is IBM Watson Health’s cancer AI algorithm (known as Watson for Oncology). Used by hundreds of hospitals around the world for recommending treatments for patients with cancer, the algorithm was based on a small number of synthetic, nonreal cases with very limited input (real data) of oncologists. Many of the actual output recommendations for treatment were shown to be erroneous, such as suggesting the use of bevacizumab in a patient with severe bleeding, which represents an explicit contraindication and ‘black box’ warning for the drug. This example also highlights the potential for major harm to patients, and thus for medical malpractice, by a flawed algorithm. Instead of a single doctor’s mistake hurting a patient, the potential for a machine algorithm inducing iatrogenic risk is vast. This is all the more reason that systematic debugging, audit, extensive simulation, and validation, along with prospective scrutiny, are required when an AI algorithm is unleashed in clinical practice. It also underscores the need to require more evidence and robust validation to exceed the recent downgrading of FDA regulatory requirements for medical algorithm approval

Therefore, take care when you look at tables like this one:

PredictionnAUCPublication (Reference number)
In-hospital mortality, unplanned readmission, prolonged LOS, final discharge diagnosis216,2210.93* 0.75+0.85#Rajkomar et al.96
All-cause 3–12 month mortality221,2840.93^Avati et al.91
Readmission1,0680.78Shameer et al.106
Sepsis230,9360.67Horng et al.102
Septic shock16,2340.83Henry et al.103
Severe sepsis203,0000.85@Culliton et al.104
Clostridium difficile infection256,7320.82++Oh et al.93
Developing diseases704,587rangeMiotto et al.97
Diagnosis18,5900.96Yang et al.90
Dementia76,3670.91Cleret de Langavant et al.92
Alzheimer’s Disease ( + amyloid imaging)2730.91Mathotaarachchi et al.98
Mortality after cancer chemotherapy26,9460.94Elfiky et al.95
Disease onset for 133 conditions298,000rangeRazavian et al.105
Suicide5,5430.84Walsh et al.86
Delirium18,2230.68Wong et al.100