The “AI stack” has arrived in healthcare – here is how to manage the transition. Photo: Aspen Medical.
A patient arrives at a modern clinic armed with a self-diagnosis generated by a chatbot. Their GP uses an AI scribe to capture the conversation, which then feeds into an automated system for billing and a digital referral for a specialist. Each tool has been tested and validated in isolation, but never together.
Therein lies the potential powder keg dubbed by Aspen Medical chief medical officer Dr Katrina Sanders the “AI stack”.
If a single error has entered the flow at the start, it propagates, travelling through the untested stack like a digital virus, whispered from one machine to the next until the error becomes the “ground truth” for the entire medical team.
There are multiple risk points in an AI stack. Patients with a chatbot-generated diagnosis may be committed to a conclusion, tainting the clinical conversation from the start. Moreover, evidence suggests that when patients present a potential diagnosis, clinicians themselves can anchor to it, influencing the diagnosis.
Then there’s shadow AI — when a clinician or a member of a team in a health system is using AI without authorisation.
“We know shadow AI is occurring in clinical settings,” Dr Sanders says. “The opacity of some AI tools poses a challenge when you’re trying to figure out how a clinician reached a conclusion.”
These problems pose an existential threat to trust in the medical system, yet swathes of the industry are already in deep.
AI tools now assist with diagnosis, documentation, triage and clinical decision-making, often stacked, and adoption across the health sector is accelerating.
With healthcare systems facing workforce shortages and rising demand, Dr Sanders says AI is “both exciting and necessary”. Used well, it can support clinicians and improve patient care in meaningful ways.
But that’s not the problem.
“The usefulness, safety, accuracy and efficacy of individual tools are important questions, but should not obscure a larger issue: what happens when multiple tools operate together inside a healthcare system?” Dr Sanders says.
It highlights an emerging double standard in the medical industry, when we look at the assurances and accountability required of clinical care delivered by teams in a health setting.
Each individual is registered, credentialled and working to a scope of practice. They undertake continued professional development and performance evaluation. The tools used are calibrated or validated to an Australian standard endorsed by an organisation, body or standard and regularly checked. Together, both team and tools are tested, regularly, to ensure optimal performance.
Dr Sanders says AI stacks used in the medical industry should be held to the same standards, including when things go wrong.
“When something goes wrong, a team can dissect and deep dive to identify what went wrong to prevent it from happening again,” she says.
“When multiple AI-enabled systems interact, cascading effects can be difficult to detect, reproduce and audit, especially when outputs are copied, summarised or transformed from one tool to the next.”
But before we hit the panic button, there’s one place Dr Sanders says we can look.
As AI adoption accelerates in healthcare, Aspen Medical’s Dr Katrina Sanders outlines the critical need for transparency, evidence and human contingency plans. Photo: Aspen Medical.
Regulation in this space is evolving and evidence is still emerging, but healthcare cannot afford to wait for a comprehensive framework on governing its AI stacks.
Dr Sanders says there are three practical steps all organisations should take now.
The first is to mandate the declaration of AI use, ensuring all organisations, clinicians and patients know when AI tools are being used in care delivery.
“Transparency builds trust and supports accountability,” Dr Sanders says.
The second is proof or disclosure. AI application must be backed by relevant clinical evidence. If that evidence isn’t there yet, don’t hide the gap — label the uncertainty clearly so clinicians can treat it as a known risk rather than a proven fact.
The third is contingency.
“Every AI-enabled workflow should have a clear fallback plan if the tool fails or is unavailable and staff should know what that contingency is so they can step in safely and confidently,” Dr Sanders says.
“These steps are not perfect, but they are practical and they are better than doing nothing whilst governance frameworks catch up with the technology.”
As organisations tread the tension between moving fast enough to realise AI’s benefits and safeguarding patients, clinicians and organisations, Dr Sanders suggests that navigating the AI stack requires a cultural shift that must begin upstream.
“When you think about governance, you’re thinking about the training, policies, deployment and adoption. But it is very much about that concept of culture and leadership,” she says.
“At Aspen Medical, we’re quite advanced in our approach and in our adoption of AI, but we’re doing it cautiously and based on evidence. We’ve got a very strong tech team and a strong clinical team and it’s the combination of the two.
“We’ve got a saying here at Aspen Medical: we’re clinically led, digitally enabled. And so we approach everything we do through that lens, including adoption of AI tools.”
For more information visit Aspen Medical.
Original Article published by Dione David on Region Canberra.
Region Media Partner Content
