]> Cloudflare

maxime.guerreiro@gmail.com

Amazon

ulaskira@amazon.com

Cloudflare

ot-ietf@thibault.uk

Web and Internet Transport Web Bot Auth not-yet This document describes a JSON based format for clients using to advertise information about themselves. This document describes a JSON-based "Signature Agent Card" format for signature agent using to advertise metadata about themselve. This includes identity, purpose, rate expectations, and cryptographic keys. It also establishes an IANA registry for Signature Agent Card parameters, enabling extensible and interoperable discovery of agent information. About This Document The latest revision of this draft can be found at . Status information for this document may be found at . Discussion of this document takes place on the Web Bot Auth Working Group mailing list (), which is archived at . Subscribe at . Source for this draft and an issue tracker can be found at .

Introduction Signature Agents are entities that originate or forward signed HTTP requests on behalf of users or services. They include bots developers, platforms providers, and other intermediaries using . These agents often need to identify themselves, and establish trust with origin servers. Today, the mechanisms for doing so are inconsistent: some rely on User-Agent strings (e.g. MyCompanyBot/1.0), others on IP address lists hosted on file servers (e.g. badbots.com), and still others on out-of-band definitions (e.g. documentation on docs.example.com/mybot). This diversity makes it difficult for operators and origin servers to reliably discover and share a Signature Agent’s purpose, contact information, or rate expectations. Existing discovery mechanisms, such as , do not have the necessary granularity, and pursue different goals. This document introduces a JSON-based "Signature Agent Card" format for Signature Agents, to be published in registries and discovered by servers. It also creates a new IANA registry of "Signature Agent Card Parameters" to ensure extensibility and consistency of future attributes.

Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Signature Agent Card Signature-Agent header is defined in . This section describes Signature Agent Card, a JSON object containing parameters describing the Signature Agent. Unless otherwise specified, all parameters in this document are OPTIONAL. There MUST be at least one parameter set. Parameters for which the value is unknown MUST be ignored. All string values are UTF-8.

Client Name The client_name parameter provides a friendly identifier for the Signature Agent. Example

• ExampleBot
• My remote browser company

Client URI The client_uri parameter provides inline content or a web page describing the bot: e.g. what does it do, how it handles data it fetches. Only http, https or data:text/plain are allowed. Example

• https://example.com/bot/about.html
• data:text/plain,The Example bot is about providing an example.

Contacts The contacts parameter provides reliable communication channels in URI forms. Typically, this is an email address. Example

• ["mailto:bot-support@example.com"]
• ["https://example.com/contact"]

Logo URI The logo_uri parameter provides an image reference for visual identification. TODO: Recommendation for size and format, if there is a clear consensus or reference we can point to. Example

• data:image/svg+xml;base64,deadbeef
• https://example.com/logo.png

Expected user agent The expected-user-agent parameter specifies one or more User-Agent strings as defined in or prefix matches. Prefixes MAY use * as a wildcard. Example

• Mozilla/5.0 ExampleBot

robots.txt product token The rfc9309-product-token parameter specifies the product token used for robots.txt directives per . Example

• ExampleBot

robots.txt compliance The rfc9309-compliance parameter lists directives from robots.txt that the agent implements. Example

• ["User-Agent", "Disallow"]
• ["User-Agent", "Disallow", "CrawlDelay"]

Trigger The trigger parameter indicates the operational mode of the agent. Valid values:

1. fetcher - request initiated by the user
2. crawler - autonomous scanning

Purpose The purpose parameter describes the intended use of collected data. Values SHOULD be drawn from a controlled vocabulary, such as . Example

• search
• tdm

Targeted content The targeted-content parameter specifies the type of data the agent seeks. Its format is arbitrary UTF-8 encoded string. Example

• SEO analysis
• Vulnerability scanning
• Ads verification

Rate control The rate-control parameter indicates how origins can influence the agent’s request rate. TODO: specify a format Example

• CrawlDelay in robots.txt (non-standard)
• Custom tool
• 429 +

Rate expectation The rate-expectation parameter specifies anticipated request volume or burstiness. TODO: consider a format such as avg=10rps;max=100rps Example

• 500 rps
• Spikes during reindexing

Known URLs The known-urls parameter lists predictable endpoints accessed by the agent. These URLs may be absolute URLs like https://example.com/index.html. They could be relative path like /ads.txt. Or they can use * as wildcard such as *.png. Example

• ["/"]
• ["/ads.txt"]
• ["/favicon.ico"]
• ["/index.html"]

Keys The keys parameter contains a JWKS as defined in . If keys is present, it is RECOMMENDED that the card is signed using . Content-Digest header MUST be included in the covered components. TODO: describe signature, CWS keys. Example

• https://example.com/.well-known/http-message-signatures-directory
• JWKS-directory

Discovery A registry is a list of URLs, each refering to a signature agent card. The URI scheme MUST be one of:

• https (RECOMMENDED): Points to an HTTPS resource serving a signature agent card
• http: Points to an HTTP resource serving a signature agent card
• data: Contains an inline signature agent card

Example

Formal Syntax Below is an Augmented Backus-Naur Form (ABNF) description, as described in . The below definition imports http-URI and https-URI from , and dataurl from .

Out-of-band communication between client and origin A signature agent MAY submit their signature agent card to an origin, or the origin MAY manually add them to their local registry.

Public list A registry MAY be provided via a GitHub repository, a public file server, or a dedicated endpoint. The registry SHOULD be served over HTTPS. A client application SHOULD validate the directory format and reject malformed entries.

Signature-Agent header Signature Agent Card format defined in extends the format of Signature-Agent header as defined in . When used for HTTP Message Signatures, and hosted on a well-known URL, Signature Agent Card MAY be discovered via a Signature-Agent header.

Security Considerations Malicious actors may put properties which are not theirs in the registry. If signatures are present, clients MUST verify them. Clients SHOULD reject cards with invalid signatures.

Privacy Considerations TODO

IANA Considerations

Registration template New registrations need to list the following attributes:

Parameter Name:

The name requested (e.g. "useragent"). This name is case sensitive. Names may not match other registered names in a case-insensitive manner unless the Designated Experts state that there is a compelling reason to allow an exception

Parameter Description:

Brief description of the Header Parameter

Change Controller:

For Standards Track RFCs, list the "IESG". For others, give the name of the responsible party. Other details (e.g., postal address, email address, home page URI) may also be included.

Reference:

Where this parameter is defined

Notes:

Any notes associated with the entry

New entries in this registry are subject to the Specification Required registration policy (). Designated experts need to ensure that the token type is defined to be used for both token issuance and redemption. Additionally, the experts can reject registrations on the basis that they do not meet the security and privacy requirements defined in TODO.

Initial Registry content This section registers the Signature Agent Card Parameter names defined in in this registry.

Client Name Parameter

Parameter Name:

client_name

Parameter Description:

A friendly name for your signature agent.

Change Controller:

IETF

Reference:

Notes:

N/A

Client URI Parameter

Parameter Name:

client_uri

Parameter Description:

Describes what the bot does inline with data URI or with an HTTP link to an external resource

Change Controller:

IETF

Reference:

Notes:

N/A

Logo URI Parameter

Parameter Name:

logo_uri

Parameter Description:

Image for a quick visual identification

Change Controller:

IETF

Reference:

Notes:

N/A

Contacts Parameter

Parameter Name:

contacts

Parameter Description:

An array of URI with a reliable communication channel; typically email addresses

Change Controller:

IETF

Reference:

Notes:

N/A

Expected User Agent Parameter

Parameter Name:

expected-user-agent

Parameter Description:

String or fragment patterns

Change Controller:

IETF

Reference:

Notes:

N/A

RFC9309 Product Token Parameter

Parameter Name:

rfc9309-product-token

Parameter Description:

Robots.txt product token your signature-agent satisfies.

Change Controller:

IETF

Reference:

Notes:

N/A

RFC9309 Compliance Parameter

Parameter Name:

rfc9309-compliance

Parameter Description:

Does your signature-agent respect robots.txt.

Change Controller:

IETF

Reference:

Notes:

N/A

Trigger Parameter

Parameter Name:

trigger

Parameter Description:

Fetcher/Crawler

Change Controller:

IETF

Reference:

Notes:

N/A

Purpose Parameter

Parameter Name:

purpose

Parameter Description:

Intended use for the collected data

Change Controller:

IETF

Reference:

Notes:

N/A

Targeted Content Parameter

Parameter Name:

targeted-content

Parameter Description:

Type of data your agent seeks

Change Controller:

IETF

Reference:

Notes:

N/A

Rate control Parameter

Parameter Name:

rate-control

Parameter Description:

How can an origin control your crawl rate

Change Controller:

IETF

Reference:

Notes:

N/A

Rate expectation Parameter

Parameter Name:

rate-expectation

Parameter Description:

Expected traffic and intensity

Change Controller:

IETF

Reference:

Notes:

N/A

Known URLs Parameter

Parameter Name:

known-urls

Parameter Description:

Predictable endpoint accessed

Change Controller:

IETF

Reference:

Notes:

N/A

Keys Parameter

Parameter Name:

keys

Parameter Description:

JWKS Endpoint

Change Controller:

IETF

Reference:

Notes:

N/A

References Normative References Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] Open Future Mozilla This document defines a vocabulary for expressing preferences regarding how digital assets are used by automated processing systems. This vocabulary allows for the declaration of restrictions or permissions for use of digital assets by such systems. Cloudflare This document describes a method for clients using [HTTP-MESSAGE-SIGNATURES] to advertise their signing keys. It defines a key directory format based on JWKS as defined in Section 5 of [JWK], as well as new HTTP Method Context to allow for in-band key discovery. This document describes a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message. This mechanism supports use cases where the full HTTP message may not be known to the signer and where the message may be transformed (e.g., by intermediaries) before reaching the verifier. This document also describes a means for requesting that a signature be applied to a subsequent HTTP message in an ongoing HTTP exchange. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document defines HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages. This document obsoletes RFC 7234. This document specifies additional HyperText Transfer Protocol (HTTP) status codes for a variety of common situations. [STANDARDS-TRACK] A JSON Web Key (JWK) is a JavaScript Object Notation (JSON) data structure that represents a cryptographic key. This specification also defines a JWK Set JSON data structure that represents a set of JWKs. Cryptographic algorithms and identifiers for use with this specification are described in the separate JSON Web Algorithms (JWA) specification and IANA registries established by that specification. This document defines how to use the Diffie-Hellman algorithms "X25519" and "X448" as well as the signature algorithms "Ed25519" and "Ed448" from the IRTF CFRG elliptic curves work in JSON Object Signing and Encryption (JOSE). This specification defines a method for computing a hash value over a JSON Web Key (JWK). It defines which fields in a JWK are used in the hash computation, the method of creating a canonical form for those fields, and how to convert the resulting Unicode string into a byte sequence to be hashed. The resulting hash value can be used for identifying or selecting the key represented by the JWK that is the subject of the thumbprint. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. Informative References A new URL scheme, "data", is defined. It allows inclusion of small data items as "immediate" data, as if it had been included externally. [STANDARDS-TRACK] This specification describes how to use bearer tokens in HTTP requests to access OAuth 2.0 protected resources. Any party in possession of a bearer token (a "bearer") can use it to get access to the associated resources (without demonstrating possession of a cryptographic key). To prevent misuse, bearer tokens need to be protected from disclosure in storage and in transport. [STANDARDS-TRACK] n.d. Team Digitale, Italian Government Red Hat Microsoft This document defines the RateLimit-Policy and RateLimit HTTP header fields for servers to advertise their quota policies and the current service limits, thereby allowing clients to avoid being throttled. This document specifies version 1.3 of the Transport Layer Security (TLS) protocol. TLS allows client/server applications to communicate over the Internet in a way that is designed to prevent eavesdropping, tampering, and message forgery. This document updates RFCs 5705 and 6066, and obsoletes RFCs 5077, 5246, and 6961. This document also specifies new requirements for TLS 1.2 implementations. This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279.

Test Vectors TODO

Implementations TODO

Acknowledgments TODO The editor would also like to thank the following individuals (listed in alphabetical order) for feedback, insight, and implementation of this document -

Changelog v01

• Add contributors
• Aligning registry draft with oauth dynamic client registration iana registry
• Add an about-url field
• Add ABNF for discovery of signature-agent card (registry)
• Add precisions about known URLs
• Add placeholder for image size

v00

• Initial draft