UZPIF Outbound Indexing for Search Engines and AI

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶

This Internet-Draft will expire on 17 September 2026.¶

Copyright Notice

Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document.¶

Table of Contents

1. Scope and Status

This Internet-Draft is part of an experimental, research-oriented suite prepared for the Independent Stream. It is published to enable structured technical review, interoperability discussion, and disciplined specification development around outbound, consent-first indexing mechanisms for UZPIF-style transports.¶

Within that suite, this document defines the current normative baseline for trust objects, validation rules, and security semantics for outbound, consent-first indexing over UZPIF-style transports, especially Discovery Grants, policy objects, and freshness signalling. Hard interoperability is expected for shared object semantics and validation rules.¶

The material is a research artefact. It does not claim technical completeness, production readiness, or endorsement by the IETF or any other standards body, and it is not presented as a standards-track specification.¶

Full wire-level, clustering, and proof-family interoperability is not claimed everywhere yet. Message encodings, transport bindings, proof families, and deployment profiles remain intentionally profile-defined or deferred. This draft therefore should not be read as claiming a fully closed wire-level system, universal discoverability control, or solved availability properties.¶

It is designed for experimentation, operator feedback, and profile-driven deployments. It does not require changes to the HTTP protocol, but it can carry or reference HTTP-origin content.¶

During conversion from internal research documents into IETF XML, care has been taken to:¶

• preserve a clear distinction between normative and informative content;¶

• use requirement language (e.g., "MUST", "SHOULD", "MAY") only where behaviour is intentionally specified;¶

• avoid any implication of registry finalisation, mandatory implementation, or standards-track status; and¶

• maintain intellectual-property neutrality, with no implied patent grants or licensing commitments beyond the IETF Trust copyright licence applicable to Internet-Draft text.¶

Ongoing research, implementation, performance validation, and real-world pilot work remain outside the scope of this Internet-Draft text and may be pursued separately.¶

2. Executive Summary

This document defines an outbound indexing model in which content publishers (servers) initiate outbound, authenticated sessions to trusted indexers to advertise content availability, request refresh, and communicate explicit usage constraints. The model is intended as a complement to inbound crawling and robots.txt-based opt-out signalling.¶

The core components are:¶

• Outbound Discovery Session: A publisher-initiated secure session to an indexer service, established over UZPIF and secured using identity-bound handshakes (e.g., TLS-DPA).¶

• Discovery Grant: An identity-bound, purpose-scoped authorisation object that grants an indexer permission to index, cache, summarise, or otherwise process specific content for specific purposes.¶

• Policy Communication: Machine-readable statements about permitted uses (e.g., search indexing, snippet generation, retrieval augmentation, AI training), retention, attribution, and derivative generation.¶

• Freshness Signalling: A method for publishers to explicitly request refresh or indicate change without exposing unauthenticated inbound endpoints.¶

The proposal is compatible with legacy web publishing and can be adopted incrementally. It is especially suited to "zero-port" deployments where inbound crawling is undesirable or impossible.¶

3. Terminology

Requirements Language: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119] and [RFC8174] when, and only when, they appear in all capitals.¶

This Internet-Draft is primarily exploratory; requirement language is used sparingly and only where behaviour is intentionally specified.¶

Publisher

A server or origin that wishes to make content available for indexing.¶

Indexer

A service (e.g., search engine, AI retrieval system, dataset builder) that consumes content for indexing, ranking, retrieval, summarisation, training, or related processing.¶

Indexing Node

An independently operated Indexer instance participating in outbound indexing.¶

Trusted Indexer

An Indexer whose identity is known to, and explicitly authorised by, a Publisher via a Discovery Grant.¶

Outbound Discovery Session

A Publisher-initiated secure session to an Indexer service endpoint; within this session, discovery, policy, and content transfer messages may be exchanged.¶

Discovery Grant

A cryptographically bound, application-specific Grant profile that conveys consent and scope (what may be indexed, by whom, and for what purposes) while preserving the suite-level Grant semantics defined by UZPIF ([UZPIF]); see Section 10.¶

Content Scope

A set of resources to which a Discovery Grant or policy applies (e.g., URL set, content-hash set, semantic collection, feed).¶

Usage Purpose

A declared intent for automated processing, such as traditional search indexing, snippet generation, retrieval augmentation, or AI training.¶

Freshness Signal

A notification indicating that indexed content may have changed and that refresh is desired.¶

Inclusion Log

A transparency record stream or append-only structure that records content accepted for indexing under a declared scope and purpose; the exact append-only proof family is profile-defined.¶

UZPIF Session

A secure, identity-bound connectivity substrate defined by [UZPIF], typically established via outbound connections to one or more Rendezvous Nodes.¶

4. Introduction

Traditional web indexing relies on inbound crawling, where automated clients (crawlers) initiate connections to servers and respect opt-out signals such as robots.txt (as standardised in the Robots Exclusion Protocol; [RFC9309]). While effective for many years, this model exposes servers to unsolicited traffic, abuse from malicious crawlers, and challenges in enforcing preferences - particularly as AI systems increasingly use crawled content for training or real-time retrieval.¶

Recent developments, including crawler best practices ([draft-illyes-aipref-cbcp]) and discussions on AI-specific controls, highlight limitations of opt-out regimes. Inbound crawling assumes servers are reachable and willing to respond, which conflicts with emerging zero-port and zero-trust architectures.¶

This document describes an outbound opt-in alternative: servers explicitly initiate authenticated connections to authorised indexers (traditional search engines or AI agents) when they wish to be discovered or refreshed. Discovery requests are bound to cryptographic identities, Discovery Grants, and policy artefacts, enabling fine-grained control over who may index content and for what purpose (e.g., traditional search, AI training, or summarisation).¶

The approach builds on identity-first transports such as the Universal Zero-Port Interconnect Framework (UZPIF; [UZPIF]) and Universal Zero-Port Transport Protocol (UZP; [UZP]), where endpoints establish outbound-only sessions. It provides a proactive, consent-first model suited to both human-readable web content and AI-driven search consumption.¶

This is a research proposal intended for experimentation and discussion, particularly in contexts where reducing inbound exposure and strengthening consent are priorities.¶

This draft should therefore be read as part of an experimental, research-oriented Independent Stream suite and as the current normative baseline for trust objects, validation rules, and security semantics within its scope. Hard interoperability is expected for shared object semantics and validation rules. Full wire-level, clustering, and proof-family interoperability is not claimed everywhere yet; the remaining details are intentionally profile-defined or deferred. Outbound initiation can reduce unsolicited crawling exposure, but it does not by itself provide traffic-pattern privacy, universal discoverability control, or solved rendezvous or indexer availability.¶

5. Problem Statement

The inbound crawling model is built on an assumption of open reachability: crawlers discover a server, initiate inbound connections, and learn policies after connecting. For modern deployments - especially those seeking to minimise exposed attack surface - this is an inversion of the desired trust model.¶

Specific limitations include:¶

• Unsolicited load and abuse: REP is advisory, and many automated clients do not comply. Even compliant crawlers can produce significant load when scaled across multiple indexers and AI agents.¶

• Weak identity and accountability: A User-Agent string is not a strong identity; it is easy to spoof and difficult to bind to policy obligations.¶

• Purpose ambiguity: The same content acquisition can be used for search indexing, summarisation, retrieval augmentation, or AI training. Without explicit purpose signalling and enforcement, site operators cannot make informed consent decisions.¶

• Incompatibility with zero-port architectures: If a Publisher exposes no public inbound listening ports, inbound crawling becomes impossible by design.¶

• Policy enforcement gaps: Opt-out mechanisms require a crawler to first connect and then choose to comply, rather than enforcing access authorisation at session establishment.¶

The outbound indexing model aims to preserve the benefits of web discoverability while shifting control to the Publisher, making consent explicit, identity-bound, and enforceable within authenticated channels.¶

6. Design Goals

The mechanism defined in this document has the following goals:¶

• Opt-in discoverability: indexing and refresh occur only when the Publisher chooses to contact an Indexer.¶

• Identity binding: all sessions are authenticated and bound to cryptographic identities, enabling durable accountability.¶

• Purpose limitation: Publishers can grant indexing permission for specific purposes (e.g., search indexing) while denying others (e.g., AI training).¶

• Policy expressiveness: Publishers can express usage constraints, retention expectations, attribution requirements, and derivative permissions.¶

• Freshness signalling: Publishers can efficiently request refresh without being continuously crawled.¶

• Transport independence: the mechanism should operate over UZPIF sessions and may be profiled for other outbound-friendly transports.¶

• Incremental adoption: the mechanism complements existing protocols such as REP and does not require immediate ecosystem-wide migration.¶

7. Architectural Overview

At a high level, outbound indexing replaces "Indexer-driven fetch" with a Publisher-initiated session that can carry announcements, policy, and (optionally) content. Under UZPIF, both Publisher and Indexer may maintain outbound connectivity to one or more Rendezvous Nodes (RNs), which stitch permitted sessions.¶

Publisher (Site)          RN(s)             Trusted Indexer
  |-- outbound setup ----->|<-- outbound presence --|
  |<== identity-bound secure session (via RN stitch) ==>|
  |-- ANNOUNCE + GRANT + POLICY ------------------------->|
  |<-- (optional) REQUEST(resource set) ------------------|
  |-- CONTENT(resource set/deltas) ---------------------->|
  |<-- RECEIPT / STATUS ----------------------------------|

The session is initiated by the Publisher, but once established it is a bidirectional secure channel in which the Indexer may request specific resources and the Publisher may provide them. The key property is that the Publisher does not expose an unauthenticated public inbound service for discovery.¶

8. Trust and Identity Model

Outbound indexing relies on cryptographic identities for both Publishers and Indexers. In UZPIF deployments, these identities are typically represented by certificates or equivalent credentials issued within an identity plane (e.g., Pantheon as described in [UZPIF]), and sessions are established over secure channels such as TLS-DPA ([TLS-DPA]).¶

A Publisher MUST authenticate the Indexer identity before sending any content beyond minimal discovery metadata. An Indexer MUST authenticate the Publisher identity before accepting Discovery Grants, policy, or content.¶

Identity binding serves two purposes:¶

• Consent enforcement: Discovery Grants are bound to specific identities and cannot be meaningfully replayed by unauthorised parties.¶

• Operational accountability: Publishers can select and audit trusted Indexers, and Indexers can maintain verifiable provenance of content acquisition.¶

9. Decentralisation and Index Governance

Outbound indexing MUST NOT require a central registry for Indexer discovery, listing eligibility, or participation.¶

Indexing Nodes MAY operate independently. Federation between Indexers is voluntary and MAY be bilateral or multilateral according to local policy.¶

No entity SHALL possess mandatory inclusion authority. No index SHALL be required for network participation.¶

Participation in outbound indexing SHALL NOT be required for transport-layer operability.¶

These constraints are intended to prevent gatekeeper capture of discovery and listing decisions.¶

10. Discovery Grants

A Discovery Grant conveys explicit permission from a Publisher to an Indexer, scoped by content and purpose. It is an authorisation artefact, not merely a preference hint.¶

A Discovery Grant is an application-specific Grant profile. It MUST preserve the baseline Grant semantics defined by UZPIF ([UZPIF]) while adding the indexing-specific body fields defined in this section.¶

Discovery Grants MUST use the common signed artefact envelope defined by [UZPIF] with "object_type" set to "discovery-grant". This document defines only the additional indexing-specific fields and semantics carried in the object-specific body. Discovery Grants inherit the UZPIF common envelope unchanged, including canonical serialisation, exact signature coverage, object_id derivation, unknown extension handling, signature ordering, algorithm identifier matching, epoch-versus-sequence precedence, and the rule that detached signatures are not part of baseline interoperability.¶

11. Policy Communication

Outbound indexing provides a channel for Publishers to communicate content usage policy to Indexers in a form that is:¶

• bound to authenticated identities;¶

• associated with explicit Content Scope; and¶

• auditable and revocable.¶

The policy data model is intentionally generic. Deployments MAY use the vocabulary defined by the IETF AIPREF working group (e.g., [draft-ietf-aipref-vocab]), or they MAY define private purpose tokens under bilateral agreement.¶

{
  "scope": "https://example.com/docs/*",
  "allowed_purposes": [
    "search.index",
    "search.snippet",
    "rag.retrieve"
  ],
  "denied_purposes": ["ai.train", "ai.finetune"],
  "derivatives": {
    "summary": "allowed",
    "embeddings": "allowed",
    "snippets": "allowed"
  },
  "retention": {
    "cached_copy_days": 14,
    "embedding_days": 30
  },
  "attribution": {
    "required": true,
    "link_back": true
  }
}

12. Protocol Operation

This section describes a baseline operational sequence. Session message framing remains intentionally abstract in this version of the document; the focus is on semantics and security properties. This abstraction does not apply to Index Transparency Entries, Signed Checkpoints (Section 9.1.3), Indexer Receipts (Section 12.5.1), or Revocation Acknowledgement artefacts (Section 12.6.1), which use the interoperable object formats defined in this document.¶

13. Relationship to Existing Mechanisms

Outbound indexing is designed to be complementary to existing web controls and does not attempt to obsolete them.¶

14. Security Considerations

Outbound indexing reduces exposure to unsolicited inbound traffic by eliminating the need for publicly reachable discovery endpoints. However, it introduces new considerations around trust, grant handling, and policy enforcement.¶

Implementations SHOULD consider:¶

• Indexer impersonation: Publishers must authenticate Indexer identities; otherwise, an attacker could harvest content by posing as an Indexer.¶

• Grant replay: Discovery Grants should be bound to identities and sessions; replay in a different context should be rejected.¶

• Scope escalation: Indexers must enforce the declared Content Scope and Purposes; ambiguous scopes should be avoided.¶

• Confused deputy: Publishers should avoid issuing broadly scoped grants that a downstream Indexer could use to justify unexpected processing.¶

• Compromised Indexers: A trusted Indexer compromise can lead to large-scale misuse; publishers should prefer short-lived grants and revocation readiness.¶

• Metadata leakage: Even announcing content existence can reveal sensitive information; publishers should consider minimal announcements and staged disclosure.¶

When used with UZPIF ([UZPIF]) and TLS-DPA ([TLS-DPA]), outbound indexing benefits from identity-bound handshake properties and reduced scanning surface. This document does not define cryptographic primitives; it relies on the referenced transports for channel security.¶

15. Privacy Considerations

Outbound indexing provides publishers with positive control over who may access content for automated processing. This can reduce privacy harms associated with indiscriminate crawling and reduce exposed discovery surface for automated access.¶

In this document, reduced publisher-side discoverability, encrypted content protection, and traffic-pattern privacy are distinct properties. Outbound indexing primarily changes discoverability and policy control. Confidentiality of transferred content depends on the authenticated encrypted session in use, and privacy against relationship or timing analysis depends on separate metadata-minimising measures. Indexers, rendezvous infrastructure, and network observers may still learn useful metadata about who contacted whom, when refreshes occurred, and how often updates were requested.¶

Publishers SHOULD consider:¶

• minimising announcement metadata to what is necessary for indexing;¶

• scoping grants narrowly to avoid unintended disclosure;¶

• using short-lived grants and explicit retention policy; and¶

• auditing relationships with trusted indexers.¶

Indexers SHOULD consider:¶

• providing transparency about processing purposes and retention;¶

• supporting publisher revocation and deletion requests; and¶

• limiting onward disclosure of content to third parties unless explicitly permitted.¶

16. IANA Considerations

This document has no IANA actions.¶

Author's Address