The urgent answer to the coming black box medicine

Black box medicine and transparency

 The series of reports Black Box Medicine and Transparency examines the human interpretability of machine learning in healthcare and research:

1. Machine learning landscape considers the broad question of where machine learning is being (and will be) used in healthcare and research for health 

2. Interpretable machine learning outlines how machine learning can be or may be rendered human interpretable

3. Ethics of transparency and explanation asks why machine learning should be made transparent or be explained, drawing upon the many lessons that the philosophical literature provides

4. Regulating transparency considers if (and to what extent) does the General Data Protection Regulation (GDPR) require machine learning in the context of healthcare and research to be transparent, human interpretable, or explainable

5. Interpretability by design framework distils the findings of the previous reports, providing a framework to think through human interpretability of machine learning in the context of healthcare and health research

6. Roundtables and interviews summarises the three roundtables and eleven interviews that provided the qualitative underpinning of preceding reports 

Each report interlocks, building on the conclusions of preceding reports.

Meanwhile you can start with the executive summary. Does anybody care about it? 

Against black box medicine

Explainable machine learning practices: opening another black box for reliable medical AI 

In regulating medical AI, we should address not only algorithmic opacity, but also on other black boxes plaguing these tools. In particular, there are many opaque choices that are made in the training process and in the way algorithmic systems are built, which can potentially impact SaMD-MLs performances, and hence their reliability. Second, we have said that opening this alternative black box means explaining the training process. This type of explanation is in part documenting the technical choices made from problem selection to model deployment, but it is also motivating those choices by being transparent about the values shaping the choices themselves—in particular, performance-centered values and ethical/social/political values. Overall, our framework can be considered as a starting point to investigate which aspects of the design of AI tools should be made explicit in medicine, in order to inform discussions on the characteristics of reliable AI tools, and how we should regulate them. We have also highlighted some limitations, and we have claimed that in the future it will be necessary to empirically investigate the practice of machine learning in light of our framework, and to identify more nuances in the values shaping ML training.

We want to end this article by repeating that the problem of explaining opaque technical choices is not an alternative to explain the opacity lying at the algorithmic level. Unlike London, we think that the worries about algorithmic opacity in medicine are more than justified. However, we leave any consideration on how the two opacities are connected to each other for future works.

Huge business interests are at  stake, who cares about citizens?

Didier Lourenço at Galeria Barnadas

Algorithms: the underpinning of black box medicine?

 Algorithms as medical devices

The rapid growth of digital devices, software and technologies means that the medical device sector is changing. Many small and independent manufacturers are encountering medical device regulation for the first time. At the same time, responsive and effective regulation of digital devices requires sound understanding of the underlying new technologies and concepts.

Algorithms as medical devices describes how digital health is covered by existing medical device regulation and outlines three critical areas: 

The challenges that the digital health sector may pose for regulators and developers

How digital devices can be regulated as medical devices under UK/EU and US law

The specific problems that machine learning could pose to medical device regulation

 Meanwhile the market for black box medicine is growing, unless any regulator leaves their continous vacation.

Against black box medicine (2)

 Time to reality check the promises of machine learningowered precision medicine

Both machine learning and precision medicine are genuine innovations and will undoubtedly lead to some great scientific successes. However, these benefits currently fall short of the hype and expectation that has grown around them. Such a disconnect is not benign and risks overlooking rigour for rhetoric and inflating a bubble of hope that could irretrievably damage public trust when it bursts. Such mistakes and harm are inevitable if machine learning is mistakenly thought to bypass the need for genuine scientific expertise and scrutiny. There is no question that the appearance of big data and machine learning offer an exciting chance for revolution, but revolutions demand greater scrutiny, not less. This scrutiny should involve a reality check on the promises of machine learning-powered precision medicine and an enhanced focus on the core principles of good data science—trained experts in study design, data system design, and causal inference asking clear and important questions using high-quality data.

The deep side of medicine and the gift of time

Deep Medicine

Nowadays the impact of Artificial Intelligence in Medicine is unknown. Every other day you may hear about robots and how they will replace humans. Nobody knows about it, distrust charlatans. The only thing that is real is what is already happening. Eric Topol has tried to do this in his new book Deep Medicine. But at the same time he considers that AI will let physicians humanise medicine, "the gift of time", and says:

"As machines get smarter, humans will need to evolve along a different path from machines and become more humane"

This may be Eric Topol's desire, nothing to add. My view is quite different. I'm not sure about the contribution of AI to a humanised medicine . This has to do with professionalism, not with AI. And the incentives for professionalism are plunging, while commercialism is on the rise. This is the key issue.
The remaining elements of the book are of interest to explain the current state of advances in apps and tools for clinical decision making. You'll find helpful information and a great summary of AI in medicine. However, my suggestion is that you can forget the subtitle of the book: "How artificial intelligence can make healthcare human again". It's naïve.

PS. Amb IA l'anys 2026 faig aquest resum

El llibre "Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again" (2019), escrit pel cardiòleg i investigador Eric Topol, explora com la intel·ligència artificial (IA) pot transformar la pràctica mèdica i resoldre la seva gran crisi actual: la pèrdua de la relació humana entre metge i pacient.

Tesi central i el model de la "Deep Medicine"

La medicina moderna pateix de superficialitat (shallow medicine): visites precipitades, diagnòstics erronis, sobrediagnòstic i un ús massiu d'historials electrònics que han convertit els metges en administratius de teclat, generant taxes de burnout i depressió superiors al 50%.

Per superar aquesta crisi, Topol proposa el model de la Deep Medicine, basat en tres pilars interconnectats:

1. Deep Phenotyping (Fenotipat profund): La caracterització digital i biològica completa de cada individu ("des del desig fins a la pols"), integrant genòmica, epigenòmica, microbioma, biosensors, historial clínic i entorn.
2. Deep Learning (Aprenentatge profund): L'ús de xarxes neuronals profundes per analitzar masses de dades complexes, reconèixer patrons i generar prediccions precises.
3. Deep Empathy (Empatia profunda): La recuperació de la connexió humana, el contacte físic, l'escolta activa i la confiança mútua entre metges i pacients, un factor que la IA fa possible en alliberar els professionals de les tasques burocràtiques i analítiques de rutina.

Capítol 1: Introduction to Deep Medicine

Topol comença amb la seva pròpia experiència traumàtica darrere d'una operació de genoll complicada. Malgrat l'èxit tècnic de la cirurgia, el seu metge va mostrar una falta absoluta d'empatia i d'anàlisi dels seus antecedents mèdics. L'autor exposa la necessitat d'integrar la IA en la Quarta Revolució Industrial per analitzar dades massives (Big Data) i, simultàniament, utilitzar el temps estalviat per humanitzar l'atenció mèdica.

Capítol 2: Shallow Medicine (Medicina superficial)

S'analitzen les patologies del sistema sanitari actual: dades insuficients, falta de temps i desconnexió emocional. L'autor denuncia l'ús ineficient dels historials electrònics de salut (EHR), el sobrediagnòstic (com el canvi de criteris de la hipertensió o les mamografies de cribratge que generen milers de falsos positius) i el fet que la majoria dels fàrmacs més venuts només funcionen en un percentatge molt reduït dels pacients que els prenen.

Capítol 3: Medical Diagnosis (El diagnòstic mèdic)

S'examinen els errors diagnòstics i els biaixos cognitius humans (heurístiques del Sistema 1 descrites per Kahneman i Tversky, com el biaix de confirmació o la representativitat). Es repassen eines d'ajuda diagnòstica com symptom checkers (Ada, Buoy, Isabel), aplicacions de crowdsourcing mèdic (Medscape Consult, Human Dx) i les lliçons de la fallida del projecte IBM Watson for Oncology, que va evidenciar la dificultat de processar dades mèdiques no estructurades.

Capítol 4: The Skinny on Deep Learning (El bàsic de l'aprenentatge profund)

A través de la història de l'aplicació AliveCor (capaç de detectar nivells de potassi a la sang i arítmies mitjançant un ECG a l'Apple Watch), Topol explica el funcionament de les xarxes neuronals profundes (DNN), l'aprenentatge supervisat, no supervisat i per reforç. Estableix un paral·lelisme clau entre els vehicles autònoms i la IA mèdica: la medicina mai superarà el Nivell 2 o 3 d'autonomia; la supervisió humana serà sempre imprescindible.

Capítol 5: Deep Liabilities (Les vulnerabilitats i riscs de la IA)

L'autor detalla els perills de la IA algorísmica: el problema de la "caixa negra" (black box) i la falta d'explicabilitat, els biaixos algorítmics (de gènere, raça o nivell socioeconòmic) basats en dades d'entrenament poc representatives, les amenaces a la privacitat (analitzant el cas de DeepMind i el NHS al Royal Free Hospital) i la necessitat d'assaigs clínics prospectius per validar els resultats obtinguts in silico.

Capítol 6: Doctors and Patterns (Els metges dels patrons)

S'analitzen les especialitats mèdiques basades en el reconeixement d'imatges i patrons visuals: Radiologia, Patologia i Dermatologia. Topol repassa estudis on els algorismes igualen o superen els metges en la detecció de càncer de pell (estudi de Stanford) o en l'anàlisi de biòpsies i radiografies. Rebutja la idea que aquests metges siguin substituïts; en canvi, la IA esdevindrà un assistent essencial i proposa la fusió de radiòlegs i patòlegs en la nova figura de l'Especialista de la Informació (Information Specialist).

Capítol 7: Clinicians Without Patterns (Clínics sense patrons)

Aborda les especialitats que no es basen en patrons visuals simples (atenció primària, oncologia, cirurgia, oftalmologia). La IA servirà per eliminar el tecleig mitjançant assistents de veu, processar dades de laboratori i genòmiques mitjançant el suport Mèdic Individualitzat Augmentat (AIMS), millorar el diagnòstic retinal (OCT), optimitzar tractaments oncològics complexos (projectes com Tempus Labs) i integrar la visió per computador en la cirurgia robòtica (Verb Surgical, "Surgery 4.0").

Capítol 8: Mental Health (Salut mental)

S'explora la paradoxa segons la qual moltes persones prefereixen confessar els seus problemes de salut mental a un agent virtual o chatbot (com Woebot o Wysa) abans que a un humà, per evitar sentir-se jutjades. Es detalla el concepte de fenotipat digital (Digital Phenotyping): l'ús de dades de tecleig, veu, patrons de son i xarxes socials per detectar de manera precoç episodis de depressió, ansietat o risc de suïcidi.

Capítol 9: AI and Health Systems (IA i els sistemes de salut)

Analitza com la IA pot predir la mortalitat a hospitals per millorar les cures pal·liatives (algoritmes de Stanford), preveure reingressos, detectar la sèpsia en temps real i gestionar el flux de treball hospitalari. Es descriu la transició cap a "hospitals virtuals" de monitoratge remot (com el Mercy Virtual Care Center) i es repassen les estratègies nacionals de IA en salut a països com el Canadà, la Xina i l'Índia.

Capítol 10: Deep Discovery (Descobriment profund en ciència)

Descriu la revolució de la IA en la investigació biomèdica bàsica: l'anàlisi del 98,5% del DNA no codificant (DeepSequence, DeepVariant, CRISPR/Elevation), el disseny accelerat de fàrmacs sense necessitat d'experimentació animal prèvia (Atomwise, BenevolentAI, Insilico Medicine) i la microscòpia d'alta resolució sense etiquetatat químic (in silico labeling).

Capítol 11: Deep Diet (La dieta profunda)

Topol demostra la fallida de les recomanacions nutricionals generals i de les dietes de moda. Basant-se en els estudis de Segal i Elinav (Weizmann Institute / DayTwo), s'explica com la resposta glicèmica als mateixos aliments varia dràsticament entre individus segons el seu microbioma intestinal, la qual cosa permet dissenyar una nutrició personalitzada guiada per IA.

Capítol 12: The Virtual Medical Assistant (L'assistent mèdic virtual)

Planteja el futur del "coach" de salut virtual: un assistent de veu integrat que rep, analitza i actualitza contínuament totes les dades de l'individu (genòmica, biosensors, estil de vida, historial mèdic) combinades amb tota la literatura científica. L'autor analitza els obstacles principals: la propietat de les dades per part del pacient, la gestió dels "falsos positius" (incidentalomes) i la necessitat de validació mitjançant assaigs clínics.

Capítol 13: Deep Empathy (Empatia profunda)

El capítol de tancament resumeix el missatge essencial del llibre: el gran regal de la IA a la medicina és el regal del temps. Si la tecnologia allibera els metges del treball administratiu, aquest temps s'ha de retornar als pacients per escoltar la seva història (evitant la interrupció mitjana que fan els metges als 18 segons), practicar l'observació, oferir confort i recuperar la presència i la confiança. Topol crida a una reforma profunda de l'educació mèdica per seleccionar i formar els futurs metges en funció de la seva intel·ligència emocional i empatia.

Conclusió

El model de Deep Medicine no pretén una medicina robotitzada, sinó utilitzar la tecnologia més avançada per restaurar el component més antic i valuós de la professió: el cura humà, la presència i l'empatia entre les persones.

Medicine as a data science (3)

High-performance medicine: the convergence of human and artificial intelligence

If you want to know the current state of artificial intelligence in medicine, then Eric Topol review in Nature is the article you have to read. A highlighted statement:

There are differences between the prediction metric for a cohort and an individual prediction metric. If a model’s AUC is 0.95, which most would qualify as very accurate,
this reflects how good the model is for predicting an outcome, such as death, for the overall cohort. But most models are essentially classifiers and are not capable of precise prediction at the individual level, so there is still an important dimension of uncertainty.

And this is good summary:

Despite all the promises of AI technology, there are formidable obstacles and pitfalls. The state of AI hype has far exceeded the state of AI science, especially when it pertains to validation and readiness for implementation in patient care. A recent example is IBM Watson Health’s cancer AI algorithm (known as Watson for Oncology). Used by hundreds of hospitals around the world for recommending treatments for patients with cancer, the algorithm was based on a small number of synthetic, nonreal cases with very limited input (real data) of oncologists. Many of the actual output recommendations for treatment were shown to be erroneous, such as suggesting the use of bevacizumab in a patient with severe bleeding, which represents an explicit contraindication and ‘black box’ warning for the drug. This example also highlights the potential for major harm to patients, and thus for medical malpractice, by a flawed algorithm. Instead of a single doctor’s mistake hurting a patient, the potential for a machine algorithm inducing iatrogenic risk is vast. This is all the more reason that systematic debugging, audit, extensive simulation, and validation, along with prospective scrutiny, are required when an AI algorithm is unleashed in clinical practice. It also underscores the need to require more evidence and robust validation to exceed the recent downgrading of FDA regulatory requirements for medical algorithm approval

Therefore, take care when you look at tables like this one:

Prediction	n	AUC	Publication (Reference number)
In-hospital mortality, unplanned readmission, prolonged LOS, final discharge diagnosis	216,221	0.93* 0.75+0.85#	Rajkomar et al.96
All-cause 3–12 month mortality	221,284	0.93^	Avati et al.91
Readmission	1,068	0.78	Shameer et al.106
Sepsis	230,936	0.67	Horng et al.102
Septic shock	16,234	0.83	Henry et al.103
Severe sepsis	203,000	0.85@	Culliton et al.104
Clostridium difficile infection	256,732	0.82++	Oh et al.93
Developing diseases	704,587	range	Miotto et al.97
Diagnosis	18,590	0.96	Yang et al.90
Dementia	76,367	0.91	Cleret de Langavant et al.92
Alzheimer’s Disease ( + amyloid imaging)	273	0.91	Mathotaarachchi et al.98
Mortality after cancer chemotherapy	26,946	0.94	Elfiky et al.95
Disease onset for 133 conditions	298,000	range	Razavian et al.105
Suicide	5,543	0.84	Walsh et al.86
Delirium	18,223	0.68	Wong et al.100

1. LOS, length of stay; n, number of patients (training+ validation datasets). For AUC values:
2. *, in-hospital mortality; + , unplanned readmission; #, prolonged LOS; ^, all patients; @, structured + unstructured data; + +, for University of Michigan site.

L'explicabilitat de la Intel·ligència Artificial a les ciències de la vida

Explainable AI in Healthcare. Unboxing Machine Learning for Biomedicine

Solving the explainable AI conundrum by bridging clinicians’ needs and developers’ goals

Tot això va molt ràpid. I potser algú s'hauria de posar a revisar-ne els fonaments i les seves conseqüències. Em refereixo en concret al que es coneix com explicabilitat de la intel·ligència artificial. En aquest blog m'hi he referit com Black Box Medicine.

Caldria que els algoritmes que s'utilitzen puguin ser permeables al coneixement dels humans, o dit altrament, que si no és explicable doncs aleshores no sabem com s'arriba al resultat i que si ho repliquem tampoc sabem si sortirà el mateix (fiabilitat). A les ciències de la vida és encara més rellevant, perquè l'impacte pot ser determinant per la salut i la vida.

Avui aporto un llibre i un article. El llibre no l'he llegit, però tinc la impressió que és el més complet que hi ha per ara en aquesta qüestió des d'un punt de vista pràctic. L'article de Nature reflecteix les perspectives dels informàtics versus els clínics, i en una gràfica es mostra tot: 

I ja podem veure que no hi ha acord i que cal reduir les diferències de perspectiva. Per ara ha sortit guanyant els primers.

Del llibre en reprodueixo l'índex:

1. Human–AI Relationship in Healthcare

2. Deep Learning in Medical Image Analysis: Recent Models and Explainability

3. An Overview of Functional Near-Infrared Spectroscopy and Explainable Artificial Intelligence in fNIRS

4. An Explainable Method for Image Registration with Applications in Medical Imaging

5. State-of-the-Art Deep Learning Method and Its Explainability for Computerized Tomography Image Segmentation

6. Interpretability of Segmentation and Overall Survival for Brain Tumors

7. Identification of MR Image Biomarkers in Brain Tumor Patients Using Machine Learning and Radiomics Features

8. Explainable Artificial Intelligence in Breast Cancer Identification

9. Interpretability of Self-Supervised Learning for Breast Cancer Image Analysis

10. Predictive Analytics in Hospital Readmission for Diabetes Risk Patients

11. Continuous Blood Glucose Monitoring Using Explainable AI Techniques

12. Decision Support System for Facial Emotion-Based Progression Detection of Parkinson’s Patients

13. Interpretable Machine Learning in Athletics for Injury Risk Prediction

14. Federated Learning and Explainable AI in Healthcare

Si un dels principis de la bioètica com diu Floridi, hauria de ser l'explicabilitat, ens queda un llarg camí per aconseguir que així sigui. Tinc la impressió que estem davant una estratègia de fets consumats: els algoritmes són aquí, i no sé explicar com s'arriba a aquest resultat amb aquest algoritme.

PS. Ha arribat la intel·ligència artificial open source, la regulació i explicabilitat esdevenen encara més rellevants.

The Theranos contretemps as a serious scandal

Last Thursday WSJ released a long article on Theranos clinical lab. In this blog you may check my February and July posts on this firm under the title: A closely guarded secret. As you may imagine, such a title was not coincidental. There were some clues that justified it, something unusual was happening. And WSJ has contributed to shed light on the issue. All the details in it. Basically, the summary is that analytic validity and clinical validity is under compromise. This is an exemple:



If you want to read a first person account, you'll find it here and here. Some additional articles: Wired, New Yorker, Clinical Chemistry and Laboratory Medicine (CCLM), Forbes, NYT, WP,...
This is not only a contretemps, it is a serious scandal and a huge problem to credibility for this start-up.
From Wired:

Theranos got a lot of traction by tapping into the frustration—both from consumers and the medical community—that diagnostic testing is too painful, too slow, and too expensive. “Their problem is they tried to do it with existing diagnostic instrumentation, instead of innovating new diagnostic instrumentation,”

Theranos is a black box that has touted results rather than process. “The ability of the lab medicine community to police and correct itself depends on that flow of information,” says Master. Instead, Theranos’ research was internal, and rather than submit their work to peer review the company cited their FDA approvals as evidence that the technology worked.

At least in the USA there is a regulator, the FDA, lab regulation in Europe was enacted in 1998, completely outdated under a third party scheme, not a direct public regulator. Therefore, there is a pressing motive to speed up new and different rules in Europe. Microfluidics and nanotechnologies are calling for and urgent overhaul.


 PS. An statement from WSJ:

In 2005, Ms. Holmes hired Ian Gibbons, a British biochemist who had researched systems to handle and process tiny quantities of fluids. His collaboration with other Theranos scientists produced 23 patents, according to records filed with the U.S. Patent and Trademark Office. Ms. Holmes is listed as a co-inventor on 19 of the patents.

The patents show how Ms. Holmes’s original idea morphed into the company’s business model. But progress was slow. Dr. Gibbons “told me nothing was working,” says his widow, Rochelle. In May 2013, Dr. Gibbons committed suicide. Theranos’s Ms. King says the scientist “was frequently absent from work in the last years of his life, due to health and other problems.” Theranos disputes the claim that its technology was failing.