A program was listening.
An ambient scribe, meant to help, generated a note saying a patient had been prescribed medication for post-traumatic stress disorder.
There was no PTSD diagnosis.
There was no medication discussed.
It was a complete hallucination, born from silence and noise alike.
Fortunately, the note died in the clinic. Dr. Jennifer Shannon, a child psychiatrist and CMO of Glacis, caught it. She read the transcript herself. Nothing there justified the claim.
“Defaulting to human-in-the-loop… is not enough.”
Imagine twenty patients a day. Pages of auto-generated notes for each.
Who reads it all?
Not the tired clinician on a Tuesday afternoon with thirty messages in their inbox and a kid waiting for a ride. Humans fail. They are busy. They are overtired. Expecting perfect vigilance is a fantasy.
The burden of documentation is real.
Nobody argues clinicians should spend nights clicking boxes. But the shift in how we work changes the work itself. The clinician is no longer distilling their understanding into notes; they are auditing machine-generated structure.
The responsibility hasn’t vanished. It just moved. And with it, the skill required to hold that responsibility.
Expertise Is Part Of The Safety Architecture
Dr. Richard Rieck flies airplanes and reads brain scans. He knows high-stakes environments.
Pilots live in simulators.
They stack failures. Engine quits. Weather turns. Instruments lie.
The goal isn’t realism for realism’s sake. It’s muscle memory.
“You train the failures so that when something unexpected happen… you don’t have to invent a response.”
Real flying is actually boring by comparison. The systems work.
That’s the point.
In aviation, pilots stay sharp not because the planes are unreliable. The planes are excellent. They stay sharp because their own competence is part of the safety net. Expertise is the architecture.
Radiology works the same way.
Dr. Rieck sees parallels. Radiologists need to know where they might slip, and where the AI will.
Neither is infallible.
Safety comes from the overlap, from the shared blind spots.
Automation Changes The Work
“HUman-in-the-loop.”
We love this phrase.
It sounds like insurance. The idea: machine drafts, human checks. The human is a backup, a catcher of broken things.
That’s a lie.
Or at least, it’s a lazy one.
The person reviewing the note isn’t just there to fix glitches. Their judgment is the system.
As machines get better at the easy stuff, human instinct gets more valuable. Not less.
If we treat humans like cheap QA testers, we degrade their role. And we degrade the safety margin.
Patients Know The System Too
We forget who else is in the room.
Patients have been auditing their care for decades. Before ambient AI existed, they were matching up meds. Spotting referral errors. Fixing demographic typos. Reconciling conflicting advice from five different specialists.
They aren’t just receiving care.
They’re identifying failures in a system they can barely see.
Hugo Campos, an advocate alongside Liz Salmi, frames this clearly. Distinguish between institutional AI and patient-directed AI.
“We’ve got to stop the system… and allow for people to help themselves.”
AI should boost patient literacy. Strengthen their expertise.
When clinicians are buried under inboxes and automation outputs, the patient becomes a critical resilience layer.
Not because the clinic shirked responsibility. But because the patient sees things the doctor, the nurse, and the algorithm miss.
A Playbook For Humans
We ask if the machines will break.
We should ask what happens to the humans who trust them.
Years of smooth sailing erode skill.
Pilots rehearse codes. Sports teams run drills for injuries that hopefully never happen. The goal? To avoid asking professionals to think clearly under extreme pressure.
It’s unsettling.
The systems work too well.
“Nobody wants to discover skill decay… in the middle of an emergency.”
If the AI is right 99% of the time, does the radiologist remember what that 1% looks like?
Does the doctor remember how to take a note from scratch when the network goes down?
This isn’t about fear-mongering.
It’s about maintenance.
We need a playbook for skills that must remain sharp, regardless of how good the tool gets. Because expertise is fragile. You won’t miss it when you don’t need it.
You only realize it’s gone when you desperately need it, and find nothing but silence where your instinct used to be.
