]> Independent

contactkaylacard@gmail.com

Applications and Real-Time Independent Submission AI policy well-known URI training policy content licensing machine learning This document requests registration of two Well-Known URIs under the "/.well-known/" path: "ai.txt" and "ai.json". These URIs define a structured, machine-readable file in which a site operator can declare AI usage preferences (training, scraping, indexing, caching), licensing terms, required attribution, and per-agent rules. "ai.txt" is positioned as a structured attachment surface for AI usage preferences in addition to robots.txt and HTTP-header carriage proposed by the IETF AIPREF working group. As the AIPREF vocabulary stabilizes, "ai.txt" can carry those preferences in a typed, single-file form alongside the broader licensing, attribution, and policy declarations defined in this document. This format is complementary to "robots.txt" . Where "robots.txt" can block crawling entirely, "ai.txt" expresses nuanced policies such as "you may crawl but not train on this content" -- a distinction that "robots.txt" alone cannot express.

Introduction AI systems increasingly interact with website content in ways that go beyond traditional crawling: training language models on web content, indexing content for retrieval-augmented generation, caching content for future reference, and scraping data for analysis. Website operators currently have no standard, machine-readable mechanism to communicate their policies regarding these AI-specific uses. "robots.txt" can block crawling entirely, but it cannot express nuanced policies. A newspaper may wish to allow crawling (for search indexing) while prohibiting training (for model development). A blog may wish to allow training under a specific license. A corporation may wish to allow some AI agents while blocking others. "ai.txt" addresses this gap. It is a policy declaration file, served at a well-known location, that communicates to AI systems:

• Whether content may be used for AI model training
• Whether content may be scraped, indexed, or cached
• Under what license terms AI training is permitted
• Which AI agents are permitted and under what conditions
• What attribution and disclosure requirements apply
• What compliance and audit expectations exist

Relationship to Existing Standards "ai.txt" is complementary to, and does not replace, existing standards:

robots.txt :

Declares crawling restrictions. "ai.txt" adds training, licensing, and per-agent policy declarations that "robots.txt" cannot express. Both files may coexist.

agents.txt:

Declares AI agent capabilities (endpoints, protocols, auth). "ai.txt" declares policy. A site may use both: "agents.txt" to declare what agents can DO, and "ai.txt" to declare what is ALLOWED.

security.txt :

Declares security vulnerability disclosure contacts. Similar well-known file pattern; different domain.

Relationship to AIPREF The IETF AIPREF working group is developing a vocabulary for expressing AI usage preferences and an attachment specification for carrying those preferences via robots.txt directives and HTTP response headers. "ai.txt" complements that work; it does not replace it. AIPREF defines the vocabulary (the set of preference terms and their semantics) and two carriage mechanisms (robots.txt and HTTP headers). "ai.txt" is a third carriage mechanism -- a single, structured, typed file -- that provides three properties not addressed by robots.txt attachment or per-response headers:

• Carriage of preferences for an entire site, independent of any individual response or robots.txt path block.
• A single audit surface -- one file at one URL -- that can be fetched once and cached for site-wide preference resolution.
• A place to declare preferences alongside related declarations (licensing, attribution, per-agent rate limits) that fall outside AIPREF's scope.

When the AIPREF vocabulary stabilizes, "ai.txt" implementations SHOULD use AIPREF preference names where they apply. Implementations SHOULD treat the preferences carried in "ai.txt" as equivalent in authority to the same preferences carried via the AIPREF robots.txt or HTTP-header mechanisms. Where multiple carriers disagree for the same site and resource, conflict resolution is out of scope for this document and may be addressed by future AIPREF output.

Related Work The following efforts overlap with or are adjacent to this document.

Spawning ai.txt (2023) :

An earlier file at "/ai.txt" published by Spawning Inc. for text-and-data-mining opt-out, scoped narrowly to TDM permission per file pattern. The format defined in this document is a strict superset, covering training, scraping, indexing, caching, per-agent rules, licensing, and attribution. The present document acknowledges Spawning's prior use of the name and positions itself as a successor declaration surface rather than a competing one.

W3C TDM Reservation Protocol :

Defines a "/.well-known/tdmrep.json" file for declaring text and data mining reservations under EU Directive 2019/790. Adjacent in domain (machine-readable opt-outs) but narrower in scope (TDM reservation only). "ai.txt" can reference or coexist with "tdmrep.json"; sites with TDM-only requirements MAY use "tdmrep.json" alone.

Cloudflare Content Signals Policy :

A robots.txt extension deployed at scale (millions of domains) that adds AI-specific signals (search, ai-input, ai-train) to robots.txt User-agent / Allow / Disallow records. Like AIPREF attach, it carries preferences inside robots.txt. "ai.txt" carries the same class of preferences -- plus licensing, attribution, and per-agent metadata -- in a separate file. Sites MAY publish both; their semantics SHOULD agree.

agents.txt:

A companion well-known file that declares what AI agents CAN do on a site (sanctioned endpoints, protocols, authentication). Where "ai.txt" expresses usage preferences and policy, "agents.txt" expresses positive capability. They are designed to coexist.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

The "ai.txt" Well-Known URI

Location The "ai.txt" file MUST be served at: The file MUST be served over HTTPS in production deployments. HTTP is permitted only in development or testing environments. The file MUST be served with Content-Type "text/plain; charset=utf-8".

Format The "ai.txt" file uses a block-based key-value format inspired by "robots.txt". Each line contains a key, a colon, and a value. Lines beginning with "#" are comments. Indented lines (two or more spaces, or one or more tabs) belong to the preceding block. A minimal "ai.txt" file:

Site Fields

Site-Name (REQUIRED):

Human-readable name of the site or service.

Site-URL (REQUIRED):

Canonical HTTPS URL of the site.

Spec-Version (OPTIONAL):

Version of the "ai.txt" specification the file conforms to (e.g., "1.0"). This is a regular field, not a comment.

Generated-At (OPTIONAL):

ISO 8601 timestamp of when the file was generated. This is a regular field, not a comment.

Description (OPTIONAL):

Brief description of the site.

Contact (OPTIONAL):

Contact email for AI policy inquiries.

Policy-URL (OPTIONAL):

URL to a human-readable AI policy page.

Content Policy Fields These fields declare site-wide defaults. Each accepts "allow" or "deny". The value "conditional" is valid only for the Training field, where it activates the per-path rules defined in the Training Path Fields section; implementations encountering "conditional" on any other field SHOULD treat it as "deny".

Training (OPTIONAL, default "deny"):

Whether AI systems may use content for model training.

Scraping (OPTIONAL, default "allow"):

Whether AI agents may scrape or read content.

Indexing (OPTIONAL, default "allow"):

Whether AI systems may index content for retrieval.

Caching (OPTIONAL, default "allow"):

Whether AI systems may cache content.

Training Path Fields When Training is "conditional", these fields specify per-path rules:

Training-Allow (OPTIONAL):

Glob pattern for paths where training is permitted.

Training-Deny (OPTIONAL):

Glob pattern for paths where training is denied.

Multiple Training-Allow and Training-Deny lines MAY appear. More specific patterns take precedence.

Licensing Fields

Training-License (OPTIONAL):

SPDX license identifier for AI training use (e.g., "CC-BY-4.0").

Training-Fee (OPTIONAL):

URL to commercial licensing or pricing page.

Agent Blocks Agent blocks declare per-agent policy overrides. The wildcard "*" sets the default for all agents. Agent identifiers SHOULD match the first token of the agent's User-Agent header (case-insensitive). Fields within an Agent block:

• Training, Scraping, Indexing, Caching: Override site-wide policy
• Rate-Limit: Advisory rate limit in "N/window" format (second, minute, hour, day)

Content Requirement Fields

Attribution (OPTIONAL):

Whether AI outputs must attribute the source. One of: "required", "recommended", "none".

AI-Disclosure (OPTIONAL):

Whether AI-generated content derived from this site must be disclosed as AI-generated. One of: "required", "recommended", "none".

Compliance Fields

Audit (OPTIONAL):

Whether AI agents must provide audit receipts. One of: "required", "optional", "none".

Audit-Format (OPTIONAL):

Expected audit format identifier (e.g., "rer-artifact/0.1").

The "ai.json" Well-Known URI

Location The JSON companion file MUST be served at: The file MUST be served with Content-Type "application/json; charset=utf-8".

Format The JSON format contains equivalent information to "ai.txt" in a typed JSON structure suitable for direct consumption by programmatic clients. The "ai.txt" file MAY reference the JSON file via: A minimal "ai.json" document: Field semantics are identical to those defined in Section 2 for the text format, with one structural difference: in the JSON form, the "specVersion" member, the "policies" member (with all four of its "training", "scraping", "indexing", and "caching" members), and the "agents" member are REQUIRED. Defaults that the text format applies implicitly MUST be stated explicitly in JSON documents.

Agent Behavior

Discovery AI agents and crawlers SHOULD fetch "/.well-known/ai.txt" and/or "/.well-known/ai.json" before interacting with an unfamiliar site. Agents SHOULD prefer the JSON format when both are available. Agents SHOULD cache the policy for the duration declared by the HTTP Cache-Control header, with a minimum TTL of 60 seconds.

Compliance "ai.txt" is advisory. It declares the site owner's policy. Compliance is expected in good faith but is not enforced by the file itself. Agents SHOULD respect Training declarations by not using content for model training when Training is "deny". Agents SHOULD respect rate limit declarations. Servers MUST enforce rate limits and access control independently of the declarations in "ai.txt".

Security Considerations Policy declarations MUST NOT include actual credentials, tokens, or secrets of any kind. "ai.txt" is advisory; servers MUST enforce policies independently. Agents MUST validate that referenced URLs use HTTPS before following them. Site owners SHOULD review their "ai.txt" periodically to ensure it accurately reflects current policy.

IANA Considerations

Well-Known URI Registration: "ai.txt" This document requests registration of the following Well-Known URI in the "Well-Known URIs" registry established by :

URI suffix:

ai.txt

Change controller:

Kayla Cardillo

Specification document(s):

This document.

Related information:

Text-format AI policy declaration file. Allows website operators to declare their AI content policy - training permissions, licensing terms, per-agent rules, and compliance requirements.

Well-Known URI Registration: "ai.json" This document requests registration of the following Well-Known URI in the "Well-Known URIs" registry established by :

URI suffix:

ai.json

Change controller:

Kayla Cardillo

Specification document(s):

This document.

Related information:

JSON-format AI policy declaration file. Companion format to ai.txt.

References Normative References In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. This memo defines a path prefix for "well-known locations", "/.well-known/", in selected Uniform Resource Identifier (URI) schemes. In doing so, it obsoletes RFC 5785 and updates the URI schemes defined in RFC 7230 to reserve that space. It also updates RFC 7595 to track URI schemes that support well-known URIs in their registry. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. Informative References

Example: News Site

Acknowledgments The "ai.txt" format draws on the design of "robots.txt" and "security.txt" for structural inspiration. The SPDX license identifiers referenced in Training-License are maintained by the Linux Foundation .