This article was originally on KFF Health News.
What good is health care for someone who makes things up, can’t keep a secret, doesn’t really know anything, and when he talks just fills in the next word based on what comes before? A lot, if that person is the latest form of artificial intelligence, according to some of the biggest companies out there.
Companies pushing the latest AI technology – known as “generative AI” – are piling up: Google And Microsoft to bring types of so-called grand language models to healthcare. Large firms familiar to those in white coats — but perhaps less so to the average Joe and Jane — are equally enthused: Electronic health record giants Epic and Oracle Cerner are not far behind. The space is also full of startups.
The companies want their AI to take notes for doctors and give them second opinions, assuming they can keep the intelligence from ‘hallucinating’or, for that matter, releasing patients’ private information.
“There’s something going on that’s pretty exciting,” says Eric Topol, director of the Scripps Research Translational Institute in San Diego. “His abilities will ultimately have a big impact.” Topol, like many other observers, wonders how many problems it could cause, such as leaking patient data, and how often. “We’re going to find out.”
The specter of such problems inspired more than 1,000 technology leaders sign an open letter in March urged companies to pause development of advanced AI systems until “we are confident that their effects will be positive and their risks will be manageable.” Still, some of them are putting more money into AI ventures.
The underlying technology is based on synthesizing huge chunks of text or other data, for example some medical models rely on 2 million notes in intensive care from Beth Israel Deaconess Medical Center in Boston – to predict text that would follow a given query. The idea has been around for years, but the gold rush and the marketing and media frenzy surrounding it are more recent.
The frenzy was kicked off in December 2022 by OpenAI supported by Microsoft and its flagship product, ChatGPT, which answers questions with authority and style. For example, it can explain genetics in a sonnet.
Started as a research venture set up by Silicon Valley elites like Sam Altman, Elon Musk, and Reid Hoffman, OpenAI has brought the excitement to investors’ pockets. The company has a complex, hybrid, for-profit and non-profit structure. But a new $10 billion funding round from Microsoft pushed the value of OpenAI to $29 billion, This is reported by the Wall Street Journal. Currently, the company licenses its technology to companies like Microsoft and sells subscriptions to consumers. Other startups are considering selling AI transcription or other products to hospital systems or directly to patients.
Hyperbolic quotes are everywhere. Former Secretary of the Treasury Larry Summers tweeted recently“It will replace what doctors do – hear symptoms and make diagnoses – before it changes what nurses do – help patients get up and treat themselves in the hospital.”
But just weeks after OpenAI made another huge cash injection, even Altman, the CEO, is wary of the fanfare. “The hype about these systems – even if everything we hope for is true in the long run – has completely spiraled out of control in the short term,” he said for a March article in The New York Times.
Few in healthcare believe this latest form of AI is about to take their jobs (although some companies are – controversially – experimenting with chatbots that act as therapists or care guides). Still, those optimistic about the technology think it will make some parts of their jobs much easier.
Eric Arzubi, a psychiatrist in Billings, Montana, led fellow psychiatrists for a hospital system. Time and time again he was presented with a list of health care providers who had not yet completed their notes – their summaries of a patient’s condition and treatment plan.
Writing these notes is one of the biggest stressors in the health system: it’s generally an administrative burden. But it is necessary to develop a record for future providers and of course insurers.
“If people are way behind on documentation, that creates problems,” Arzubi said. “What happens if the patient comes into the hospital and there’s a note that hasn’t been completed and we don’t know what’s going on?”
The new technology can help ease those burdens. Arzubi is testing a service called Nabla Copilot, which for its part attends virtual patient visits and then automatically summarizes them, organizing the complaint, case history and treatment plan in a standard notes format.
The results are solid after about 50 patients, he said: “It’s 90% of the way there.” Copilot produces actionable summaries that Arzubi usually edits. The summaries don’t necessarily pick up on nonverbal cues or thoughts that Arzubi may not want to voice. Still, he said, the benefits are significant: He doesn’t have to worry about taking notes and can instead focus on talking to patients. And it saves time.
“If I have a full patient day, where I see maybe 15 patients, I’d say it saves me a good hour at the end of the day,” he said. (If the technology is widely adopted, he hopes hospitals won’t take advantage of the time saved simply by scheduling more patients. “That’s not fair,” he said.)
Nabla Copilot isn’t the only service; Microsoft is trying out the same concept. At the Healthcare Information and Management Systems Society conference in April — an industry confab where health techs exchange ideas, make announcements and sell their wares — Evercore investment analysts highlighted reducing administrative burdens as a top opportunity for the new technologies.
But in general? They heard mixed reviews. And that opinion is common: many technologists and doctors are ambivalent.
For example, if you’re stumped about a diagnosis, entering patient data into one of these programs “can undoubtedly get you a second opinion,” Topol said. “I’m sure clinicians are doing it.” However, this comes up against the current limitations of the technology.
Joshua Tamayo-Sarver, a clinician and executive at the startup Inflect Health, ran fictionalized patient scenarios based on his own practice in an emergency department into one system to see how it would perform. It lacked life-threatening conditions, he said. “That seems problematic.”
The technology also has a tendency to “hallucinate,” that is, make up information that sounds convincing. Formal studies have found a wide range of performance. One preliminary research document examining ChatGPT and Google products using an open board study questions from neurosurgery found a hallucination rate of 2%. A study of Stanford researchers, which examined the quality of AI responses to 64 clinical scenarios, found fabricated or hallucinatory quotes 6% of the time, study co-author Nigam Shah told KFF Health News. Another preliminary paper found that ChatGPT agreed with expert opinion half the time in complex cardiology cases.
Privacy is another concern. It’s unclear whether the information fed into this type of AI-based system will stay inside. For example, enterprising ChatGPT users have managed to get their hands on the technology to tell them the recipe for napalmwhich can be used to make chemical bombs.
In theory, the system has guardrails that prevent private information from escaping. For example, when KFF Health News asked ChatGPT for his email address, the system refused to release that private information. But when told to role-play as a character and asked for the email address of the author of this article, he gladly provided the information. (It was indeed the author’s correct email address in 2021, when ChatGPT’s archive ends.)
“I wouldn’t put patient data in there,” says Shah, chief data scientist at Stanford Health Care. “We don’t understand what happens to this data once it hits OpenAI servers.”
Tina Sui, a spokesperson for OpenAI, told KFF Health News that one should “never use our models to provide diagnostic or treatment services for serious medical conditions.” They are “not geared to providing medical information,” she said.
With the explosion of new research, Topol said, “I don’t think the medical community really has a good idea of what’s going to happen.”
KFF Health News is a national newsroom that produces in-depth journalism on health issues and is one of the core programs of KFF – an independent source of health policy research, polling and journalism. Learn more about KFF.
