TS43Agents and crawling

Do authorized agents receive the intended access?

Access for authorized crawlers is tested with the full user-agent string and reconciled with logs.

Prepare evidence

In the app: Evidence import
You provide: Agents, HTTP responses and logs
Allow for: Depends on evidence needed

Prepare evidence

Understand

Allowed through. But does it get through?

Access for authorized crawlers is tested with the full user-agent string and reconciled with logs.

TSEP protocol criterion

TSEP distinguishes a test using a declared User-Agent from an observation attributed to the actual agent. Identification limits and missing evidence must remain explicit.

The rules and limits of this check are detailed at the bottom of this guide.

Check

The response, the refusal. And the actual trace.

In TSEP Control

From an analysis, “Investigate further” lets you prepare available supplementary observations. This guide’s workflow also lets you gather and import the evidence required by the rules.

Your contribution

Define the scope and expectations before collection. Gather the items below and supply the requested reviews.

With your own tools

Compare HTTP tests with server, CDN and firewall logs. Use identification methods documented by the operator; changing a User-Agent does not prove an actual visit by that agent.

Evidence to retain

Prior plan, reviewed full User-Agent, raw HTTP, native JSONL logs, bound supplied reviews, command/receipt and identity qualification.

What this result covers

First profile: assessor-initiated test client; no actual visit, identity, security or operator access inferred. Sources, logs, reviews and receipt are unauthenticated.

Act

The block located. The fix targeted.

Locate denial across the full chain and compare the test with available logs.

Recheck

Access restored? A verified visit?

After correction, gather new evidence with the same scope and expectations. Retain both dated observations to compare what changed.

Your reference points

To go further.

Statuses, sources and method.

Understand the statusesC · NC · NT · NA

The same legend in guides, analysis and demo. Each status depends on the control’s evidence and rules.

C · Verified: The criterion is met within the assessed scope. Keep dated evidence for the next comparison.
NC · To correct: Locate denial across the full chain and compare the test with available logs.
NT · Not established: Evidence or confirmations are insufficient to conclude. Supply what is missing; this status does not establish an SEO error.
NA · Not applicable: The control does not apply to the scope under its rules, with a retained justification. Missing evidence alone does not make it non-applicable.

Sources and references1 reference

OpenAI — Overview of crawlersdevelopers.openai.com · new tab

Detailed method and rules3 rules

Control method and criteria

Method
Declare operator access separately from test-client treatment; compare its full chain, reconcile logs and qualify its origin honestly under agent-access-v1.

Acceptance criteria
All three rules pass for the declared test scope, with planned response, qualified correlation and explicitly unverified operator identity; an intended denial can conform.

Non-applicability
No agent to test in the declared policy.

Detailed rules

TS43-A01 · Full client and expected treatment

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each URL and HTTP client within the previously declared test scope.

Evidence
Evidence, dates, bindings and observations under docs/agent-access-evidence.md.

Method
Read sources and bound review; check the dated HTTP chain then planned test-client response/body.

Acceptance criteria
Reviewed documentation and response agree with the test plan; retain independent contradictions despite other missing evidence.

When the result remains inconclusive
Missing client/source/review, incomplete chain, absent test expectation or required body reference.

Non-applicability
No automatic atomic exemption.

Assumptions
Supplied sources, context, log quality and reviews; test scope limited to its client.

Limitations
First profile: assessor-initiated test client; no actual visit, identity, security or operator access inferred. Sources, logs, reviews and receipt are unauthenticated.

TS43-A02 · Chain reconciled with logs

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each URL and HTTP client within the previously declared test scope.

Evidence
Evidence, dates, bindings and observations under docs/agent-access-evidence.md.

Method
Check export and layer/ID/cache/clock review, then identify each unique in-window line before literal comparison.

Acceptance criteria
Every hop has an attributed record with matching method, URL, client and status; any qualified difference fails despite another missing hop.

When the result remains inconclusive
Missing logs, client-supplied/reused ID, duplicate, other layer, unknown format or insufficient window/review.

Non-applicability
No automatic atomic exemption.

Assumptions
Supplied sources, context, log quality and reviews; test scope limited to its client.

Limitations
First profile: assessor-initiated test client; no actual visit, identity, security or operator access inferred. Sources, logs, reviews and receipt are unauthenticated.

TS43-A03 · Test origin without invented identity

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each URL and HTTP client within the previously declared test scope.

Evidence
Evidence, dates, bindings and observations under docs/agent-access-evidence.md.

Method
Check bound execution receipt and supplied attribution; distinguish assessor test from an operator visit.

Acceptance criteria
Assessor-test origin, not-verified operator identity and no real-visit claim; any bound contrary claim fails.

When the result remains inconclusive
Missing receipt/command/attribution, inconsistent dates, absent binding or ambiguous identity status.

Non-applicability
No automatic atomic exemption.

Assumptions
Supplied sources, context, log quality and reviews; test scope limited to its client.

Limitations
First profile: assessor-initiated test client; no actual visit, identity, security or operator access inferred. Sources, logs, reviews and receipt are unauthenticated.

Rule version 0.2.0-rc.1 · Public protocol reference · Versioned source

Put it into practice

← Back to controls