In what they call the “largest study of its kind,” the European Broadcasting Union and BBC have confirmed what so many of us have seen when chatting with AI about the news: AI isn’t even getting it half right.
In fact, the “News Integrity in AI Assistants” report found that 81% of AI responses had some sort of issue, with 45% of all responses having at least one issue that was significant.
With nearly 15% of those under 25 already using AI as their news source, the ramifications can’t be overstated.
“This research conclusively shows that these failings are not isolated incidents,” says EBU Media Director and Deputy Director General Jean Philip De Tender. “They are systemic, cross-border, and multilingual, and we believe this endangers public trust. When people don’t know what to trust, they end up trusting nothing at all, and that can deter democratic participation.”
Sourcing — be it unverifiable, incorrect, or nonexistent — was the significant issue for 31% of responses, followed by accuracy (20%) and context (14%).
(Source: European Broadcasting Union)
“Across the entire dataset of 3,113 core and custom questions asked, only 17 were met with refusal (0.5% — down from 3% in the first BBC round),” the study says. “This suggests assistants are more willing to answer questions, whether or not they are capable of providing a high-quality answer.”
The researchers created a toolkit for AI developers looking to “evaluate, monitor and improve the performance of AI assistants in a news context.” For publishers, the researchers suggest better control on how their content is used.
“Unauthorized use of content in AI assistants is still a widespread problem that is compounded by errors,” the study says. “Where content is authorized for use, we need clearer attribution and citations in an agreed format with prominent links to original content.”
SEE FOR YOURSELF
The Magazine Manager is a web-based CRM solution designed to help digital and print publishers manage sales, production, and marketing in a centralized platform.
