Sara Guaglione · Nov 9, 2026
WTF is the AI training opt-out policy token?
Markdown · JSON · Original source
’s Sara Guaglione explains a proposed “policy token” that would let publishers express AI training preferences in robots.txt through one shared instruction, instead of setting instructions for each crawler. , Google, Microsoft and have said they support it, but the proposal remains under deliberation at the Internet Engineering Task Force (IETF). The article’s central tension is that a common signal could give publishers another way to state their preferences, but robots.txt cannot make companies comply. Data shared with Digiday suggests crawlers rarely check those files. Alex Springer argues that the companies’ support for an unreviewed draft risks bypassing public standards-setting and treating training as permitted unless publishers opt out. Paul Bannister calls the token “window dressing,” but also compares it to another lock on a door: a measure that could help without preventing every intrusion. There is another tradeoff for publishers. Blocking crawlers for AI grounding may reduce visibility in AI experiences, according to Microsoft, though the effect for most providers is unclear. The token does not cover AI inference, and it cannot prevent providers from getting content through other sources. Its practical reach, and the consequences of opting out, remain unsettled.
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 9, 2026 (2026-11-09T05:01:00+00:00).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-09T05:08:28.111Z. First observed: 2026-11-09T05:03:32.562Z. Retrieved: 2026-11-09T05:07:27.450Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The policy token is a proposal, not an official standard; its practical adoption and compliance remain uncertain. The crawler data cited was shared with by ScalePost and tracked activity across more than 50 publisher properties; the article does not establish that it represents all crawlers or publishers. The article says the effect of opting out on visibility in AI tools is unclear for most providers; Microsoft is the only one of the four supporters it identifies as explicitly addressing that implication. The token concerns AI training, not AI inference, and does not resolve whether providers may obtain the same content from other sources.
Open question
How much would publishers’ visibility in AI tools change if they disallow crawlers for AI grounding, given that Microsoft warns of reduced visibility but the effects for most providers are unclear?
Source-stated
The proposed policy token lets publishers state AI training preferences in robots.txt using one shared instruction rather than separate instructions for each crawler.
The new AI training policy token would allow publishers to do so in one step. Instead of managing separate instructions for every crawler, they can set broad preferences to participating AI and tech companies.
Source-stated
The proposal is still a draft under deliberation, despite support from Anthropic, Google, Microsoft and OpenAI.
While the largest search and AI providers have agreed to support this policy token, it still needs to be deliberated on by the IETF.
Source-stated
The token cannot ensure that companies follow publishers’ preferences, and the article reports that crawlers rarely check robots.txt files.
Robots.txt preferences are infamously more of a suggestion than an enforceable doctrine. In fact, data suggests that web crawlers rarely even check publishers’ robots.txt files when scraping their sites.
Source-stated
The article presents a tension between restricting AI use and maintaining visibility: Microsoft says blocking crawlers for AI grounding will reduce visibility in AI experiences, while the broader effect remains unclear.
If a publisher does disallow crawlers for AI grounding, “you will see reduced brand visibility in AI experiences,” reads Microsoft’s blog post.
Source-stated
The policy token does not cover AI inference or prevent providers from obtaining content through other sources.
The AI training policy token doesn’t have anything to do with AI inference (or AI usage), which is when an AI model generates responses or performs tasks by retrieving or processing content from publishers’ websites in real time.
Source and contributor context
Sara Guaglione · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
Media declared by the original publisher; availability checked separately from the editorial reading.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Digiday
Digiday is a trade publication covering digital media, marketing and advertising businesses.
Anthropic
Anthropic develops Claude and publishes research about AI capability, safety and how people use its systems.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
Editorial provenance
Version 1. Publication ID: b064bf22-04a2-431d-8be7-e90ebaa2a6af.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T05:08:28.111Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: f8fe93dd6538b4db9bcbe1a40cbc1d11c3e362c10315f3d53982f6acddeb15b9. Source text hash: fd53ad2a2736534042cddd490254ca5be034811ffc271f4c9052d4aac63c9d89.
Revision and source-check history
Version 1 · 2026-11-09T05:08:28.111Z
WTF is the AI training opt-out policy token?
’s Sara Guaglione explains a proposed “policy token” that would let publishers express AI training preferences in robots.txt through one shared instruction, instead of setting instructions for each crawler. , Google, Microsoft and have said they support it, but the proposal remains under deliberation at the Internet Engineering Task Force (IETF). The article’s central tension is that a common signal could give publishers another way to state their preferences, but robots.txt cannot make companies comply. Data shared with Digiday suggests crawlers rarely check those files. Alex Springer argues that the companies’ support for an unreviewed draft risks bypassing public standards-setting and treating training as permitted unless publishers opt out. Paul Bannister calls the token “window dressing,” but also compares it to another lock on a door: a measure that could help without preventing every intrusion. There is another tradeoff for publishers. Blocking crawlers for AI grounding may reduce visibility in AI experiences, according to Microsoft, though the effect for most providers is unclear. The token does not cover AI inference, and it cannot prevent providers from getting content through other sources. Its practical reach, and the consequences of opting out, remain unsettled.
Original source
· Article
Source details
Original source date: Nov 9, 2026 (2026-11-09T05:01:00+00:00).
Date basis: Date stated by the publisher.
Added to Escape Meridian: 2026-11-09T05:08:28.111Z. First observed: 2026-11-09T05:03:32.562Z. Retrieved: 2026-11-09T05:07:27.450Z.
Limits
The policy token is a proposal, not an official standard; its practical adoption and compliance remain uncertain. The crawler data cited was shared with by ScalePost and tracked activity across more than 50 publisher properties; the article does not establish that it represents all crawlers or publishers. The article says the effect of opting out on visibility in AI tools is unclear for most providers; Microsoft is the only one of the four supporters it identifies as explicitly addressing that implication. The token concerns AI training, not AI inference, and does not resolve whether providers may obtain the same content from other sources.
Open question
How much would publishers’ visibility in AI tools change if they disallow crawlers for AI grounding, given that Microsoft warns of reduced visibility but the effects for most providers are unclear?
Source-stated
The proposed policy token lets publishers state AI training preferences in robots.txt using one shared instruction rather than separate instructions for each crawler.
The new AI training policy token would allow publishers to do so in one step. Instead of managing separate instructions for every crawler, they can set broad preferences to participating AI and tech companies.
Source-stated
The proposal is still a draft under deliberation, despite support from Anthropic, Google, Microsoft and OpenAI.
While the largest search and AI providers have agreed to support this policy token, it still needs to be deliberated on by the IETF.
Source-stated
The token cannot ensure that companies follow publishers’ preferences, and the article reports that crawlers rarely check robots.txt files.
Robots.txt preferences are infamously more of a suggestion than an enforceable doctrine. In fact, data suggests that web crawlers rarely even check publishers’ robots.txt files when scraping their sites.
Source-stated
The article presents a tension between restricting AI use and maintaining visibility: Microsoft says blocking crawlers for AI grounding will reduce visibility in AI experiences, while the broader effect remains unclear.
If a publisher does disallow crawlers for AI grounding, “you will see reduced brand visibility in AI experiences,” reads Microsoft’s blog post.
Source-stated
The policy token does not cover AI inference or prevent providers from obtaining content through other sources.
The AI training policy token doesn’t have anything to do with AI inference (or AI usage), which is when an AI model generates responses or performs tasks by retrieving or processing content from publishers’ websites in real time.
Source and contributor context
Sara Guaglione · Article author
These are credits stated by the source; they do not establish a profile match or authorship of every passage.
Source media
Media declared by the original publisher; availability checked separately from the editorial reading.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Digiday
Digiday is a trade publication covering digital media, marketing and advertising businesses.
Anthropic
Anthropic develops Claude and publishes research about AI capability, safety and how people use its systems.
OpenAI
OpenAI develops AI models and products and publishes research, release announcements and evaluations.
Editorial provenance
Version 1. Publication ID: b064bf22-04a2-431d-8be7-e90ebaa2a6af.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T05:08:28.111Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: useful_perspective.
Observation: f8fe93dd6538b4db9bcbe1a40cbc1d11c3e362c10315f3d53982f6acddeb15b9. Source text hash: fd53ad2a2736534042cddd490254ca5be034811ffc271f4c9052d4aac63c9d89.
uncertain · 2026-11-09T05:08:28.111Z · anthropic/claude-opus-5.5
Publication: b064bf22-04a2-431d-8be7-e90ebaa2a6af
attribution — uncertain: Digiday, Sara Guaglione, Springer, Bannister and Microsoft are correctly named. However, the summary says in its own voice that the token 'cannot prevent providers from getting content through other sources.' The article attributes that point to Bannister. Smallest repair: add 'Bannister notes' to that clause.
it doesn’t mean the AI provider won’t be able to get that data elsewhere, such as from other crawlers, Bannister said.
summary — pass: The reading identifies the actor, the mechanism, IETF status, the compliance gap, the critics' objections and the visibility tradeoff, which makes it understandable on its own. Robots.txt, grounding and inference are kept as exact technical terms, suitable for tooltips. One minor fidelity point: the summary softens Microsoft's 'will see reduced' to 'may reduce,' while claim 4 keeps 'will.' Aligning the two would preserve the source's force. The question follows from the evidence.
For the most part, that’s unclear.
date — pass: The displayed original publication date of 2026-11-09 matches the article byline. The IETF submission on Oct. 30 is a separate event and is not misdated.
By Sara Guaglione • November 9, 2026
support — uncertain: Most claims are directly supported. Claim 5's excerpt supports only the inference point. The 'other sources' point rests on Bannister's attributed statement and is presented as fact without a supporting excerpt. The crawler-data limitation is appropriately scoped to ScalePost's sample of more than 50 properties.
“None of them have said, ‘We won’t buy data from third parties that scrape the content that don’t obey this,’ or ‘We won’t take open source crawls like Common Crawl,’ Bannister said.
media — pass: The selected media is the verified direct link to the original Digiday article, which is the most direct access to the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Digiday is a trade publication covering digital media, marketing and advertising businesses.
Explore DigidayAnthropic develops Claude and publishes research about AI capability, safety and how people use its systems.
Explore AnthropicOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIDigiday is a trade publication covering digital media, marketing and advertising businesses.
Explore DigidayDigiday is a trade publication covering digital media, marketing and advertising businesses.
Explore DigidayDigiday is a trade publication covering digital media, marketing and advertising businesses.
Explore DigidayAnthropic develops Claude and publishes research about AI capability, safety and how people use its systems.
Explore AnthropicOpenAI develops AI models and products and publishes research, release announcements and evaluations.
Explore OpenAIDigiday is a trade publication covering digital media, marketing and advertising businesses.
Explore DigidayDigiday is a trade publication covering digital media, marketing and advertising businesses.
Explore Digiday