The trials of human-in-the-loop clinical AI monitoring
It seems that human-in-the-loop can’t save us after all.
Not sure what I’m talking about?
- Human-in-the-loop refers to the common AI accuracy safeguard, wherein a human is “in the loop,” double-checking AI’s work before it can make real-world impacts.
- In a new npj Digital Medicine study, researchers demonstrated that human checkers are not as effective a safeguard against AI-caused errors in medicine as we may have hoped.
In a best-case framing, this approach to ensuring clinical AI accuracy is a way of growing trust between clinical AI developers, hospitals, and providers—and, hopefully, decreasing clinician burnout along the way.
However, more cynical commentators have referred to human-in-the-loop as a band-aid on the bigger problem of AI’s inaccuracy issues, or a way to “[pawn] off liability for AI-generated errors onto doctors.”
In my view, the truth lies somewhere in the middle.
- As we innovate in this clinical AI frontier, clinician guidance and knowledge is critical. We’re not ready to hand the reins over to AI.
- Yet, we also cannot afford not to use innovations available to us, especially as clinician burnout continues to threaten more and more providers.
A bit of this truth is evident in the study itself. There’s a key disconnect evident in the results:
- On one hand, participating clinicians found many mistakes generated by the clinical AI.
- At the same time, they reported high levels of trust in the tool.
The problem isn’t necessarily that AI makes mistakes. It’s that clinicians are burnt out.
One of the study authors said it himself: the disconnect “really paints a picture of just an overburdened workforce generally—and that you’re giving them some helpful tool and there’s a certain eagerness to go, ‘Yes, I trust it, because I want to trust it, because I’m overworked.’”
So, what do we medical innovators need to do?
We can’t stop innovating—and we must keep including clinicians in the innovation process. However, we must also raise the alarm further about clinician burnout.
We need to create tools that our overworked providers can trust, ones that don’t create more work and liability for them. Anything less will not do.
