AI in Healthcare: Guardrails? What Guardrails?
As we drop #AIToyToTools video podcast episodes, we are reminded that those who ensure the safety, security and accuracy of domain-specif AI deployment are not the big guns in the AI world.
(Image: iStock)
We cope with an extraordinary time, as announcements, promises and claims for making mega investments in AI infrastructure — datacenters and the semiconductors to go inside them — sweep the market and shoot the stock prices of the companies involved toward the stratosphere.
Flooded by news about exciting technology and the boomtown spirit in the market, we forget to step back and ponder the real-world consequences of the half-cocked AI implementations that got us into this tizzy.
I wonder who will be left holding the bag when something goes terribly wrong with AI deployed in autonomous vehicles or healthcare systems. Mine are concerns not triggered by some hypochondriac “what if” scenarios.
There is, for example, the lawsuit filed by the family of 16-year-old Adam Raine, who committed suicide earlier this year.
According to the suit, filed in August against OpenAI and chief executive Sam Altman, Adam Raine started using ChatGPT for help for his homework and increasingly got engaged in conversation with the chatbot. The teenager began expressing his loneliness. ChatGPT asked whether he wanted to explore his feelings more, instead of urging him to seek help from a mental health professional.
The boy then escalated to several suicide attempts. Each time he reported back to his ChatGPT chatbot. At one point, according to the plaintiffs in the suit, the chatbot offered to help Adam write a suicide note, while discouraging him from talking to his mother about his feelings.
This week, the family amended the lawsuit, alleging that OpenAI relaxed its ChatGPT guardrails just before Adam Raine killed himself. His parents claim that this suicide was the “predictable result of deliberate design choices” by OpenAI.
The Financial Times reported this week:
“OpenAI’s intentional removal of the guardrails included instructing the artificial intelligence model in May last year not to ‘change or quit the conversation’ when users discussed self-harm, according to the amended lawsuit, marking a departure from previous directions to refuse to engage in the conversation.”
Such terms as “safeguards” or “guardrails” for AI are often used by companies like OpenAI when discussing their aspirations to win broader acceptance by society.
Indeed, right after the initial lawsuit, OpenAI gave the usual lip services noting that the company is continuing to roll out stronger guardrails for users under 18, so that that the chatbot “recognizes teens’ unique developmental needs”, reported the Guardian.
Nonetheless, just last week, OpenAI rolled out an updated version of its assistant that would allow users to customize the chatbot for more human-like experiences, including permitting erotic content for verified adults. In an X post announcing the changes, OpenAI CEO Sam Altman lamented that the strict guardrails intended to make the chatbot less conversational made it “less useful/enjoyable to many users who had no mental health problems,” according to the Guardian.
But seriously, what guardrails?
Who builds the guardrails? How are they implemented? Can we trust them?
As we search for domain-specific AI in our #AIToyToTools video podcast series, we are often reminded that the people who ensure the safety, security and accuracy of AI deployments in specific domains are not necessarily the big guns in the AI world—like OpenAI, Google or Microsoft.
Sure, those AI companies claim that they are installing fundamental guardrails in their AI models. But when the AI models go into safety-critical domain-specific systems such as automotive and healthcare, it’s the system companies, system integrators, software and hardware suppliers who end up bearing the burden of safety.
Patient care devices at the edge
NXP Semiconductors, for example, is helping the designers of patient care devices to deploy AI at the edge.
Our discussion with Ali Ors, Global Director of AI Strategies and Technologies at NXP, reveals the company’s measured approaches, concerns over safety, and efforts to make AI more useful, efficient and accountable in a much more granular manner in devices ranging from hearing aids to insulin pumps and EKG devices.
In the episode dropped today, Ors explains how AI helps adapt and customize patient care devices. He also makes clear the importance of explainability of AI. While it isn’t easy to get AI to explain itself, the healthcare industry must figure out how to do it, if it hopes to gain the trust of the medical community and its patients.
In this episode, we discuss:
Medical AI deployed at edge
Roles of AI in patient care devices
How AI makes diagnostic decisions
Making AI decision flow viewable and queryable
MedGemma
Why LLMs must be domain-specific.
The AIToyToTools video podcast series is lining up Ben Goldberg, Global Industry Lead, Health and Life Sciences at CGI, next week to discussAI in healthcare from the global consultant’s point of view.
Ben Goldberg’s summary video is here:
Please sign up for Junko’s Talk To Us YouTube channel so that you don’t have to miss our newest episodes.
Nice interview Junko! I think the challenges of explainable AI are underestimated. There is a risk (I would say an almost-inevitability) that explainability approaches will leave a gap between what is really going on and an invented, plausible-seeming explanation for AI behavior.
See: How the Leopard Got Its Spots. https://en.wikipedia.org/wiki/Just_So_Stories
AI is a robot but it violates at least the first two of Isaac Asimov's Three Rules of Robotics, the first of which is, “(1) a robot may not injure a human being or, through inaction, allow a human being to come to harm." The tech bros of Silicon Valley, all of whom certainly read their Asimov when they were kids, need to bone up on "the Bible."