Inspect the model before you trust the gate.

robots.txt communicates crawl preferences to user agents that choose to follow it. It does not authenticate a crawler, return a denial response, or connect access to commercial status.

DataLicenser converts publisher policy and live license state into a decision at the CDN or local server layer.

User-agent: GPTBot
Disallow: /

# Declared policy above
# Enforced decision below
license.active && identity.verified → ALLOW
otherwise                           → DENY

The server layer reads a versioned JSON configuration from local disk. The control plane updates it in the background; the request path does not wait for that network sync.

{
  "publisher_id": "pub_…",
  "version": 42,
  "expires_at": 1786536000,
  "crawler_policy": { "GPTBot": "deny" },
  "active_licenses": ["lic_…"],
  "signature": "hmac-sha256:…"
}

The design target is a policy refresh approximately every 15 minutes. If the signature is invalid or the cache expires beyond its configured grace policy, known crawler routes fail closed. Human requests continue through the publisher’s normal stack.

1. Create a token scoped to Zone Read and the dedicated rules resource used by DataLicenser.
2. Select the zone. DataLicenser verifies read access before proposing any rule.
3. Review the generated crawler expression. Existing rules are shown but not edited.
4. Publish the rule, then run the verification request from Setup.

1. Create an API client with read access to the target property and scoped write access to its security configuration.
2. Enter the EdgeGrid credentials through the encrypted connection flow.
3. Select a staging configuration and review the proposed behavior.
4. Verify on Akamai staging before promotion through your existing activation process.

1. Create and verify the domain from the publisher setup screen.
2. Download datalicenser.php and store the issued site key and API secret outside the web root.
3. Load the engine with PHP's auto_prepend_file setting.
4. Schedule php /path/datalicenser.php --flush every five minutes.

The local verifier reads cached policy before protected crawler content is returned. It checks policy expiry and signature, crawler identity, and any presented license token.

location / {
  auth_request /_datalicenser/verify;
  try_files $uri $uri/ /index.php?$args;
}

location = /_datalicenser/verify {
  internal;
  proxy_pass http://127.0.0.1:8787/verify;
}

Reference packages are not yet published. The setup flow will provide versioned snippets and checksums when available.

Policy files and licensed-access tokens use separate scoped secrets. HMAC is symmetric: any system with a given secret can compute a signature, so storage, rotation, and scope form part of the security boundary.

Access tokens are short-lived and bound to the publisher, license, request class, timestamp, and nonce. The verifier reconstructs the canonical message and compares signatures in constant time.

canonical = method + "\n" + path_class + "\n" +
            publisher_id + "\n" + license_id + "\n" +
            timestamp + "\n" + nonce

expected = HMAC-SHA256(site_secret, canonical)
allow = timing_safe_equal(expected, presented)

Expired, malformed, replayed, or invalid tokens fail closed for licensed crawler routes.

When the license, publisher scope, intended use, and token are valid, a supported integration can return Markdown or JSON derived from the canonical CMS record. Advertising, navigation, and unrelated page chrome are excluded.

GET /licensed/v1/content/{publisher}/{content_id}
Authorization: DL-HMAC credential=lab_…, signature=…
Accept: text/markdown | application/json

Structured delivery is available only for content and uses explicitly covered by the active agreement. It is not a general extraction endpoint.

Crawler events are designed to buffer locally and flush approximately every five minutes. Final retention windows and data-processing terms will be documented before production onboarding.

Every licensed AI lab maps to a publisher-approved agreement and a separate technical entitlement. The access record includes:

• Buyer: the named legal entity and verified technical identities.
• Scope: covered content, permitted use, formats, territory, and any training or retrieval restrictions.
• Commercial state: effective date, expiry, renewal, suspension, and termination.
• Exclusivity: optional, explicit, and limited to the scope the publisher approves.

A publisher can authorize several labs at once. Each receives different credentials and can be revoked without changing any other lab’s state.