AI training preferences in robots.txt: implementing the emerging shared signal

Dated development

Three October 30 developments align—but remain at different maturity levels

An individual Internet-Draft proposed the policy token __usage_ai-training. It reuses ordinary robots.txt groups and path matching to express preferences about using content to modify the learned parameters of generative AI models.

OpenAI said it plans to honor the shared signal in the coming months while retaining its existing crawler controls. Google said it plans support in 2027 and will keep Google-Extended available for provider-specific choices.

Mechanism

The proposal reuses familiar robots.txt path rules

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Under the draft, this expresses a preference against generative-AI training for the site except content under /press-kit/. A matching Disallow means a preference against the named use; Allow means a preference in favor. If no group names the policy token, no shared preference is expressed.

The wildcard group User-Agent: * does not stand in for the policy token. Existing crawler-specific groups continue to govern crawling independently. This separation lets a publisher allow discovery while declining training, or make different decisions for different paths.

Control boundary

Preference, access and legal authority are separate questions

A robots.txt file is public and advisory. Do not place secrets in its paths or rely on it to protect confidential content. Private material needs identity-aware access, least privilege, logging and data-governance controls regardless of AI preference signals.

Reference architecture

Generate one policy artifact from a reviewed source of truth

Content inventory and owner
    ↓
Path-level usage decision
    ↓
Versioned policy source
    ↓
robots.txt generator + tests
    ↓
CDN/origin deployment
    ↓
Synthetic fetch, monitoring and evidence

Keep the policy decision in source control rather than hand-editing production robots.txt. Generate the shared token group and any provider-specific groups from the same reviewed configuration, then test precedence and representative URLs before deployment.

Record who approved each path rule, the business rationale, the draft version used and the production hash. This provides evidence when a provider changes behavior, a site reorganizes paths or legal requirements evolve.

Migration

Add the shared signal without deleting working controls

1. Inventory crawler-specific rules, contractual permissions and protected content paths.
2. Classify paths by intended training preference; avoid a site-wide default that accidentally covers licensed or partner content.
3. Add the proposed token in a staging robots.txt and test exact, prefix and longest-match behavior.
4. Preserve OpenAI, Google-Extended and other provider-specific controls until each provider documents precedence and production support.
5. Deploy through the normal release pipeline and verify the file from every CDN region or hostname that serves content.
6. Monitor the Internet-Draft and provider documentation; review the configuration when either changes.

Operations

Make policy drift observable

• Continuously fetch robots.txt from canonical, regional and alternate hostnames.
• Alert when generated output differs from the reviewed policy hash.
• Run path-level tests after routing, CMS, CDN or localization changes.
• Maintain a provider matrix covering current support, precedence and known exceptions.
• Retain dated snapshots so the preference in force when content was retrieved can be reconstructed.
• Keep an escalation path for suspected non-compliance; a crawler log alone may not reveal downstream use.

Readiness checklist

Before production rollout

1. Is every content path assigned an accountable owner and usage decision?
2. Have you separated crawl access from AI-training preference?
3. Are current provider-specific rules preserved?
4. Do tests cover wildcard, exact-path and longest-match behavior?
5. Can you prove which robots.txt version was deployed on a given date?
6. Are private resources protected by authorization rather than advisory files?
7. Is someone responsible for tracking draft and provider rollout changes?

A common signal can reduce configuration sprawl, but trustworthy operation still comes from explicit ownership, tested rules and verifiable deployment.