Tell me why: The imperative of explainability in AI in healthcare

The following guest article is by Neeraj Mainkar, VP of Software Engineering and Advanced Technology at Proprio

Artificial intelligence is revolutionizing healthcare by improving diagnostic accuracy, personalizing treatment plans, and potentially improving patient outcomes. However, the rapid demand for integrating AI into healthcare systems raises significant concerns about the transparency and explainability of advanced technologies. In a field where decisions can mean the difference between life and death, the ability to understand and trust AI decisions is both a technical requirement and an ethical imperative.

Understanding explainability in AI

Explainability refers to the ability to understand and articulate how an AI model arrives at a particular decision. In simple AI models such as decision trees, this process is relatively straightforward. However, in complex deep learning models with numerous layers and complicated neural networks, it is nearly impossible to understand the decision-making process. Reverse engineering, or investigating specific problems in the code, is extremely difficult. When a prediction does not turn out as expected, the complexity of these models can make it difficult to determine why. Even the developers cannot always explain their behavior or results.

This lack of transparency, or the “black box” nature of AI, is a significant problem in healthcare, where understanding the reasoning behind an AI-assisted treatment or diagnosis is incredibly high stakes because human lives are at stake.

The importance of explainability in healthcare

The push for AI in healthcare is driven by its potential to improve diagnostic accuracy and treatment planning. Understanding AI’s decision-making process and ensuring its explainability is a top priority before it can be implemented in healthcare. This need for explainability is multi-faceted:

• Patient safety and trust: Patients and healthcare providers must trust AI-driven decisions; without explainability, trust dwindles and the acceptance of AI in the clinical setting becomes a challenge
• Error identification: In healthcare, errors can have serious consequences. Explainability allows errors to be identified and corrected, ensuring the reliability of AI systems.
• Compliance with legal regulations: Healthcare is a highly regulated industry. For AI systems to be approved and deployed, they must meet strict regulatory standards that often require a clear understanding of decision-making.
• Ethical standards: Transparency in AI decision-making is consistent with ethical standards in healthcare and ensures that decisions are fair, unbiased and defensible.

There are also significant financial implications associated with explainability. Research shows that companies that derive at least 20% of their profits from AI are more likely to adhere to explainability best practices. In addition, companies that build digital trust through transparent AI practices are more likely to benefit from annual revenue and profit growth rates of 10% or more.

Challenges of explainability

There are several challenges to explaining AI in healthcare. The biggest hurdle is the inherent complexity of AI models. The more accurate and dense a model is, the less explainable it becomes. This paradox means that while complex models can produce highly accurate results, their decision-making process remains opaque.

Another challenge is balancing performance and explainability. Simplifying models to improve interpretability often reduces accuracy. In a complex healthcare setting where every detail is critical for disease prediction or diagnosis, models should not be simplified because preserving their complexity is crucial.

Towards solutions: research and collaboration

Explainability is something that all AI companies must grapple with. Significant research efforts are being made to decipher how large language models work and understand the reasons behind the answers generated. Recently, researchers at Anthropology made progress in making AI models more understandable. They extracted millions of features from one of their production models, showing that interpretable features exist and are important for safety, controlling model behavior, and classification.

While this progress is encouraging, there is still much to be discovered, especially with regards to how AI works in healthcare. For this reason, organizations should prioritize transparency and continue to be open about their research efforts. For example, MIT’s IBM Watson research lab, Google, and many others are making progress in this area. In addition, there are several approaches that can be explored to improve explainability:

• Interpretable AI models: Developing models that are inherently more interpretable using techniques such as attention mechanisms and feature importance
• Stakeholder involvement: Involving health experts, ethicists, regulators and AI researchers in the development process to ensure that different perspectives and needs are taken into account
• Education and training: Improving AI literacy among healthcare professionals and the general public to create a better understanding of AI decision-making processes
• Regulatory framework: Establish robust regulatory frameworks and ethical guidelines to ensure the transparency and accountability of AI systems

The way to the future

As research efforts continue to develop fully explainable AI in healthcare, this is a necessary path to ensure that these technologies can be safely and effectively integrated into clinical practice. Responsible AI means operating within ethical boundaries. Demanding complex explainability in AI means building trust and reliability while ensuring that AI-driven decisions are transparent, defensible, and ultimately benefit patient care. As AI will undeniably revolutionize healthcare, the demand for explainability will only grow, making it a compelling area of ​​joint focus for researchers, developers, and healthcare providers. They must work to maintain extreme complexity and explainability in AI models to ensure robust support in diagnosis, treatment planning, and patient care across the care continuum.