It is an oncologist’s responsibility to prepare cancer patients for difficult decisions. At times, they may forget to perform this crucial task. At the University of Pennsylvania Health System, doctors are reminded to discuss a patient’s treatment and end-of-life preferences by an artificially intelligent algorithm that predicts the likelihood of death.
However, this tool is not something that can be left unattended. A routine check-up on the technology revealed that the algorithm’s performance deteriorated during the COVID-19 pandemic. According to a study conducted in 2022, the algorithm became 7 percentage points less accurate in predicting patient outcomes.
This decline in accuracy likely had real-world consequences. Lead author of the study, Dr. Ravi Parikh from Emory University, mentioned that the tool failed to prompt doctors hundreds of times to initiate vital discussions with patients, potentially preventing unnecessary chemotherapy in some cases.
Dr. Parikh also highlighted that many institutions are not consistently monitoring the performance of their AI products. He believes that several algorithms designed to enhance medical care may have experienced similar weaknesses during the pandemic.
These algorithmic glitches point to a larger issue that computer scientists and healthcare providers have been aware of for some time – artificial intelligence requires consistent monitoring to ensure optimal performance. Hospital executives and researchers are beginning to realize that these systems need proper oversight and maintenance.
Dr. Nigam Shah, Chief Data Scientist at Stanford Health Care, raised concerns about the financial implications of AI technologies in healthcare. While AI has the potential to improve patient care and access, it must be cost-effective and sustainable in the long term.
As AI becomes more prevalent in healthcare settings, the need for evaluating and monitoring these technologies becomes increasingly critical. Currently, there is no standardized approach for assessing the performance of AI algorithms once they are deployed in clinical settings.
Investing resources into monitoring AI systems is essential for ensuring their reliability and effectiveness. Experts suggest the idea of using AI to monitor AI, with human oversight, to prevent failures and ensure consistent performance. However, such an approach would require additional funding and resources, posing challenges for hospitals grappling with budget constraints and a shortage of AI specialists.
KFF Health News, formerly known as Kaiser Health News (KHN), is dedicated to providing comprehensive journalism on health issues and operates as one of the key programs at KFF, an independent source for health policy research, polling, and journalism.
