Google has removed its AI-generated “AI Overviews” from several medical search queries following an investigation that found the summaries could provide misleading or even harmful health information.
The move underscores ongoing concerns about the reliability of generative AI when handling sensitive or life-impacting topics, such as healthcare.
Here's everything you need to know.
TL; DR
- Google has disabled AI Overviews for certain health-related searches after reports of inaccurate information.
- The Guardian’s investigation revealed misleading AI summaries in medical contexts.
- Google said it is refining its models and working with health experts to improve accuracy.
- Critics say this exposes deeper flaws in the use of AI tools to deliver medical guidance.
What Happened
Google’s AI Overviews, launched in 2024 to provide users with quick, AI-generated summaries of search results, recently came under scrutiny after The Guardian discovered that some medical-related results displayed misleading advice.
For instance, searches like “what is the normal range for liver blood tests” triggered overly simplified explanations that omitted critical medical context such as age, gender, or ethnicity—factors that directly affect the interpretation of test results.
Following the report, Google removed AI Overviews for certain sensitive queries. A Google spokesperson confirmed the change, telling TechCrunch that the company “continually refines its systems and removes certain overviews in cases where our models do not yet meet our quality and reliability standards.”
Expert Findings And Reported Errors
According to The Guardian, the issue extended beyond minor inaccuracies.
In one example, the AI overview suggested that individuals with pancreatic cancer should avoid high-fat foods, advice that health experts described as “completely incorrect and doing so could be really dangerous and jeopardize a person’s chances of being well enough to have treatment.”
Other tests found that AI Overviews were presenting normal medical ranges without context, which could lead users to assume that their own blood results were normal when they were not.
A TechCrunch report added that the company has not ruled out reinstating the summaries once the quality concerns are resolved. “We’re working to improve how our systems handle sensitive queries and will reintroduce these features when we’re confident in their reliability,” the spokesperson said.
What Experts Are Saying
Medical and AI researchers widely supported Google’s temporary removal of the summaries, though many said it reveals the broader problem of relying on AI to condense complex medical data.
Dr. Jenny Radesky, a pediatrician and researcher quoted by The Guardian, said, “AI tools are not designed to understand clinical nuance. Even if the source is accurate, how the model interprets and presents that data matters, especially in medicine.”
Meanwhile, Professor David Leslie of The Alan Turing Institute commented that “the real issue lies in the system’s lack of epistemic awareness—it does not know what it does not know.”
These sentiments were echoed by digital policy analyst Dr. Anna Jobin, who noted, “Companies are realizing that accuracy in AI-generated knowledge is not only a technical issue but also an ethical one.”
Broader Implications
The controversy over AI Overviews comes amid growing debate over the role of generative AI in healthcare and medical search. AI systems that summarize complex information have proven useful for general education, but their inability to fully grasp context and nuance can lead to dangerous misinterpretations.
Google’s AI Overviews feature is powered by its Gemini model, which aggregates data from web pages, health databases, and knowledge panels to provide concise answers. While Google claims that most AI Overviews are accurate, experts say the stakes in medical contexts are far higher, requiring expert-reviewed, human-vetted content.
For now, Google appears to be treading carefully. Its decision reflects a larger industry trend toward limiting AI systems in sensitive areas such as health, law, and finance, where misinformation could have real-world consequences.
Topics For More Insights
What’s Next
Google says it is working with clinical specialists and health information partners to improve the model’s medical accuracy and contextual reasoning. The company also plans to expand its internal review process to better handle queries related to health, safety, and legal guidance.
As AI becomes more deeply integrated into search, the incident serves as a reminder that automated intelligence is only as trustworthy as the data and human oversight behind it.
“What we’re seeing is a necessary recalibration,” said Dr. Jobin. “The balance between innovation and responsibility is what will define the next phase of AI search.”
Join The Discussion