Key Points
- IAB Tech Lab released new guidance to help publishers manage bots, crawlers and AI-driven web traffic.
- The framework complements the CoMP API by outlining business strategies for controlling automated access to content.
- Public comments are open through June 26 as the group works toward finalizing standards for broader adoption.
IAB Tech Lab has released a new set of guidelines aimed at helping publishers and content owners develop strategies for managing bots, crawlers and other forms of non-human web traffic, as the digital advertising industry grapples with the growing influence of artificial intelligence systems.
The guidance, published this week for public comment, is intended to complement the organization’s recently-launched CoMP API V1 by providing practical frameworks for how content owners can evaluate and control access to their content by automated systems. Public comments on the proposal will be accepted through June 26.
The move comes as publishers face increasing pressure to determine how AI companies, search engines and other automated services interact with their content. While the CoMP API establishes technical standards for communicating permissions and preferences, IAB Tech Lab said many organizations have yet to develop broader business policies governing access by bots and crawlers.
“Giving content owners clear, practical guidance is key if we want adoption to move forward in a meaningful way,” IAB Tech Lab CEO Anthony Katsur said in a statement. “This work helps simplify a complex area so companies can make decisions that fit their business while supporting a more sustainable marketplace.”
The guidance was developed by IAB Tech Lab’s CoMP Working Group and is designed primarily for business leaders rather than technical teams. The document outlines different approaches organizations can take when evaluating automated traffic, along with the benefits, costs and trade-offs associated with each strategy.
According to IAB Tech Lab, the goal is to help companies better understand how AI systems and other automated agents access content, while reducing operational complexity and supporting new commercial relationships between publishers and AI platforms.
“Content owners are being asked to make important decisions quickly, often without clear frameworks,” said Shailley Singh, the Executive Vice President and Chief Operating Officer of IAB Tech Lab. “This guidance helps break down the options so they can choose an approach that aligns with their goals.”
The organization said the guidance is intended to help publishers and content owners understand the operational and financial implications of different bot management approaches. By developing formal policies, companies can reduce risk, lower costs and more efficiently engage with AI systems while preserving opportunities for future partnerships.
Industry participants involved in the project said the need for such guidance has become more urgent as AI companies increasingly rely on web content to train and power their products.
“The AI revolution has caught many publishers with their front doors wide open, essentially leaving the lights on for bots and crawlers to pilfer intellectual property,” said Scott Messer, Principal and Founder of Messer Media. He described the guidance as a practical framework for helping publishers distinguish between automated systems that create value and those that consume resources without delivering benefits in return.
Dave Bellous, the Vice President of Strategy at Metal Toad, said non-human traffic now accounts for a majority of activity across much of the open web, with AI agents and crawlers accelerating that trend. As a result, he said, blanket bot-blocking strategies are no longer sufficient and publishers need more-granular tools and standards-based controls to manage access.
Tom Koch, the Head of Sales, Ads & Marketing Ecosystem at TwelveLabs, said the framework arrives as AI agents increasingly act as intermediaries between consumers and digital content.
“As AI agents increasingly serve as proxies for consumers, publishers and platforms need clearer frameworks for transparency and control,” Koch said. “The crawler management framework is an important step toward giving content owners greater visibility into how their content is accessed and used.”
Koch added that the initiative is particularly important for video content, which he said has often been overlooked in discussions around how AI systems interact with the web.
IAB Tech Lab said it will continue working with members and industry stakeholders following the public comment period, with the goal of finalizing the guidance and encouraging broader adoption alongside the CoMP API. The organization views both efforts as key components of a more-sustainable framework for managing AI access to online content and supporting a fair exchange of value between publishers and technology platforms.
To submit a public comment on the matter, click or tap here.
