In healthcare, artificial intelligence (AI) is like a two-sided coin. On one side, AI can help doctors make better decisions about our health.
But on the other side, it can sometimes worsen things, especially if the AI learns from flawed data.
Doctors and other health experts face life-or-death choices every day. That’s why they’re working hard to make sure AI tools in healthcare are reliable and safe. In the United States, the Food and Drug Administration (FDA) monitors these AI tools.
They’ve even set rules for the people who make these AI systems, asking them to make their AI’s decisions clear and understandable.
This is so doctors can understand why the AI suggests a certain treatment or diagnosis.
Despite these efforts, a new study in the Journal of the American Medical Association (JAMA) has found something worrying. Even when doctors get explanations from the AI, they can still be misled if the AI has learned from biased data.
Sarah Jabbour, a computer science Ph.D. student, explains that understanding these AI explanations can be tricky.
Her team at the University of Michigan looked into this issue using AI models that help diagnose patients with acute respiratory failure—a severe condition where patients struggle to breathe.
Michael Sjoding, a doctor and professor, points out that diagnosing respiratory failure can be challenging. Typically, doctors are about 73% accurate.
They examine the patient’s medical history, lab tests, and imaging results. It seems logical that an AI could help improve this accuracy.
Jabbour, Sjoding, Jenna Wiens (another professor), and their team tested this idea. They worked with 457 medical professionals like doctors, nurse practitioners, and physician assistants.
These professionals were asked to diagnose patients using either just the AI’s decision or the AI’s decision along with an explanation. The explanation was a kind of map showing what part of a chest X-ray the AI was focusing on.
The study found something interesting. When the medical professionals used the AI’s decision without an explanation, their accuracy increased by about 3%. But when they had the AI’s explanation too, their accuracy improved even more—by about 4.4%.
However, the study had a twist. The team also tested what happens when the AI is intentionally biased. For example, the AI might wrongly suggest that older patients, say 80 years and above, are likely to have pneumonia.
This kind of bias happens when the AI learns from data that isn’t quite right. Jenna Wiens explains that if, for instance, women are often misdiagnosed with heart failure in the data the AI learns from, it might wrongly think women are at a lower risk of heart failure.
The team wanted to see if explanations from the AI could help doctors spot these biases. Unfortunately, when the AI was biased, the doctors’ accuracy dropped significantly—by about 11.3%.
Even when the AI’s explanation clearly showed it was focusing on irrelevant things, like a patient’s bone density, it didn’t help the doctors much.
This drop in performance is in line with other studies. It shows that even with explanations, AI models can sometimes mislead doctors.
Sarah Jabbour believes there’s much more work to be done. The team needs to create better ways to explain AI decisions to doctors in a way they can easily understand. This big task will require experts from different fields to work together.
The hope is that this study will encourage more research into using AI safely in healthcare for everyone. It also highlights the need for medical education about AI and its potential biases.
This way, we can all benefit from the advancements in AI, while making sure it’s used in the safest and most effective way possible.
If you care about heart health, please read studies about diabetes drug that could revolutionize heart failure treatment, and this drug can be a low-cost heart failure treatment
For more information about heart health, please see recent studies that exercise in middle age reversed worrisome heart failure, and results showing this drug combo can cut risk of stroke and heart attack by half.
The research findings can be found in JAMA.
Copyright © 2023 Knowridge Science Report. All rights reserved.
