To Block, Or Not To Block: How Blocking AI Bots May Be Costing Publishers More Than They Think

It’s been easy for publishers to anecdotally blame AI for traffic declines.

Now, research is backing up what so many of us have been feeling, finding both a “consistent and moderate decline in traffic to news publishers” over the last 16 months as well as a significant drop amongst those publishers who’ve proactively blocked bots.

“Using a difference-in-differences approach,” write the researchers behind “The Impact of LLMs on Online News Consumption and Production” paper, “we find that blocking genAI bots can have adverse effects on large publishers by reducing total website traffic by 23% and real consumer traffic by 14% compared to not blocking.”

Fraction of sites that disallow genAI bots (Source: “The Impact of LLMs on Online News Consumption and Production”)

Hacks/Hackers helps break down much of the research, explaining how these effects can vary based on the size of the publisher.

“Some mid-sized publishers [experience] traffic increases rather than declines, though these findings are more exploratory and the mechanisms aren’t fully understood,” says the Hacks/Hackers article. “For major publishers, you lose significant traffic — not just from bots, but from humans who discover content through AI-powered search. You risk becoming invisible in the emerging AI-mediated information ecosystem.”

For those more worried about the human factor and jobs with LLMs’ rise, the research found “no near-term contraction in newsroom hiring relative to other roles.”

“These facts suggest that generative AI is not yet functioning as a complete substitute for traditional news production and monetization,” the paper concludes.”Rather, publishers appear to be adjusting along multiple domains: access control, hiring composition, and content format and quantity production.”