TS42Agents and crawling

Does every agent have an explicit policy?

Each automated agent is governed according to its documented purpose.

Prepare evidence

In the app: Evidence import
You provide: Agents, purposes and decisions
Allow for: Depends on evidence needed

Prepare evidence

Understand

One operator. Several agents.

Each automated agent is governed according to its documented purpose.

TSEP protocol criterion

TSEP requires a documented access decision for the studied agents and uses. Each operator’s documentation describes its agents; it does not determine your policy.

The rules and limits of this check are detailed at the bottom of this guide.

Check

A documented agent. A defined use.

In TSEP Control

You import evidence for this control. The engine assesses it within the declared scope; missing items remain visible.

Your contribution

Define the scope and expectations before collection. Gather the items below and supply the requested reviews.

With your own tools

For each operator, consult official documentation and distinguish crawling, training, search and user-requested access. Record the decision, owner and review date.

Evidence to retain

Mechanism/purpose inventory, dated operator sources bound to reviews, owner decision and tradeoffs, captured robots policy.

What this result covers

Supplied scope and bounded robots profile; no automatic registry. Sources, reviews and decisions are unauthenticated; no inferred visit, real identity, operator compliance, indexing or citation.

Act

The published policy. The permitted uses.

Decide access by documented purpose, then align the published policy and its review.

Recheck

The agent evolves. Does the decision too?

After correction, gather new evidence with the same scope and expectations. Retain both dated observations to compare what changed.

Your reference points

To go further.

Statuses, sources and method.

Understand the statusesC · NC · NT · NA

The same legend in guides, analysis and demo. Each status depends on the control’s evidence and rules.

C · Verified: The criterion is met within the assessed scope. Keep dated evidence for the next comparison.
NC · To correct: Decide access by documented purpose, then align the published policy and its review.
NT · Not established: Evidence or confirmations are insufficient to conclude. Supply what is missing; this status does not establish an SEO error.
NA · Not applicable: The control does not apply to the scope under its rules, with a retained justification. Missing evidence alone does not make it non-applicable.

Sources and references2 references

Detailed method and rules3 rules

Control method and criteria

Method
Document mappings and their review, retain the owner decision, then compare its implementation under agent-governance-v1; retain coupled purposes and unknowns.

Acceptance criteria
All three rules pass for each declared target and mechanism; qualified policy matches the accepted choice, with supplied review and responsibility.

Non-applicability
No agent policy in the declared scope, with a reason provided.

Detailed rules

TS42-A01 · Documented purposes and mechanisms

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each target URL and mechanism of the previously declared policy.

Evidence
Inputs, dates, hashes and detailed findings under docs/agent-usage-policy.md.

Method
Bind operator, actor, token, mechanism and purposes to dated UTF-8 sources; check hashes and supplied bound review.

Acceptance criteria
Every mapping has a complete, dated, bound supported review with explicit robots applicability. A bound contradiction fails.

When the result remains inconclusive
Missing, stale or ambiguous source; missing hash, binding, completeness or review.

Non-applicability
No automatic atomic exemption.

Assumptions
Supplied inventory; declared source interpretation and personal authority, without automatic authentication.

Limitations
Supplied scope and bounded robots profile; no automatic registry. Sources, reviews and decisions are unauthenticated; no inferred visit, real identity, operator compliance, indexing or citation.

TS42-A02 · Owner decisions and tradeoffs

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each target URL and mechanism of the previously declared policy.

Evidence
Inputs, dates, hashes and detailed findings under docs/agent-usage-policy.md.

Method
Check every reasoned choice, the separate declared owner/delegate decision and its plan binding; retain any arbitration of coupled purposes.

Acceptance criteria
Each mechanism/purpose pair has a choice and reason; the owner accepts the plan and its exact tradeoffs. A bound rejection fails.

When the result remains inconclusive
Missing/undeclared author, unbound decision, inconsistent dates, incomplete choice or arbitration.

Non-applicability
No automatic atomic exemption.

Assumptions
Supplied inventory; declared source interpretation and personal authority, without automatic authentication.

TS42-A03 · Captured policy and effective choice

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each target URL and mechanism of the previously declared policy.

Evidence
Inputs, dates, hashes and detailed findings under docs/agent-usage-policy.md.

Method
Group tokens case-insensitively, qualify mappings and compare Allow/Disallow with the effective choice; retain reading, tradeoffs, contradictions and unknowns.

Acceptance criteria
For each qualified group, the attributable client robots 200 policy matches the accepted choice. Any independent contradiction fails.

When the result remains inconclusive
Opposing purposes without arbitration, unsupported mechanism, ignored field or missing HTTP/source/decision evidence; other robots profiles unqualified.

Non-applicability
No automatic atomic exemption.

Assumptions
Supplied inventory; declared source interpretation and personal authority, without automatic authentication.

Rule version 0.2.0-rc.1 · Public protocol reference · Versioned source

Put it into practice

← Back to controls