Does robots.txt allow the right pages?

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Each origin and path matrix whose crawl policy is assessed.

Evidence
Complete matrix, prior intent, context, full robots request/response, profile and reasoned per-resource result.

Method
Freeze the profile and expected allowances/disallows before observation. Interpret robots status, groups, rules and redirects in the initial origin context; retain the lines determining each outcome.

Acceptance criteria
Every observed policy matches declared intent. Unexpected permission and unexpected disallow are contradictions. Intentional absence can be compatible under the profile; no universal 200 requirement.

When the result remains inconclusive
Absent profile/capture/timing, unsupported syntax or unresolved error/cache: inconclusive. An attributable contradiction on another resource remains fail. Null, unsupported or inconsistent directive_crawler makes policy interpretation inconclusive without falling back to the client group. An independent HTTP contradiction remains limited to the observed client.

Non-applicability
No atomic exemption when the control applies.

Assumptions
Choose the profile before measurement and keep it fixed for a campaign context; never select the favorable profile after observation. Distinguish RFC model, documented Google policy and unobserved actual cache. Optional directive_crawler=googlebot, frozen before capture, selects the Googlebot policy only on the representation received by crawler, the exact HTTP User-Agent. An explicit robots_policy with case-insensitive googlebot product_token is required, as for TS07. Without the field, historical behavior is preserved. Optional prior google-robotstxt-fields-v2 extends the closed registry of fields outside access scope using pinned google/robotstxt sources; old profiles remain unchanged with no automatic selection.

Limitations
semiAuto: matrix and profile suitability remain declared. The reference is bounded to one origin, documented profiles and supplied captures; no engine visit, ingestion or indexing is established. In explicit mode, every robots conclusion concerns the declared product on bytes received by the client, without proving the representation received by Googlebot. An HTTP status contrary to the plan (including 401/403/429/5xx when 200 is expected) remains fail for that client, never evidence of denial to Googlebot. Registry fields and sitemap are recorded without assessing usage preferences, TS42 or indexing and never delimit groups. Other fields and tolerances remain unsupported. Full line numbers and values are retained; reason previews disclose truncation.

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
Resources explicitly required in the declared scope, including selected rendering resources.

Evidence
Resource plan, robots, complete HTTP traces, reasoned per-resource review with declared identity, date, provenance, assistance and binding digest.

Method
Compare robots policy and HTTP trajectories with planned destinations/statuses, then review responses for obstacles. Bind each supplied review to bytes, dates, intent and context. Retain gaps separately.

Acceptance criteria
Every required resource and observed hop is allowed under the profile, reaches its planned response and has no obstacle according to supplied review. Robots blocking, a contradictory response or an evidenced obstacle gives fail even when other evidence is missing.

When the result remains inconclusive
Missing resource/review, unbound review, incomplete response, uncertainty or unsupported scope: inconclusive without erasing an established contradiction. Null, unsupported or inconsistent directive_crawler makes policy interpretation inconclusive without falling back to the client group. An independent HTTP contradiction remains limited to the observed client.

Non-applicability
No required resource in this scope, established by reasoned inventory review dated before the plan; an unjustified empty list is insufficient.

Assumptions
A test client does not authenticate as Googlebot; 200 alone cannot establish absence of a challenge. Software does not produce the review or authenticate reviewer identity. Optional directive_crawler=googlebot, frozen before capture, selects the Googlebot policy only on the representation received by crawler, the exact HTTP User-Agent. An explicit robots_policy with case-insensitive googlebot product_token is required, as for TS07. Without the field, historical behavior is preserved. Optional prior google-robotstxt-fields-v2 extends the closed registry of fields outside access scope using pinned google/robotstxt sources; old profiles remain unchanged with no automatic selection.

Limitations
Inventory sufficiency and human judgment are not fully automated. The reference covers one origin and attributable trajectories; it does not execute rendering or establish actual engine access. In explicit mode, every robots conclusion concerns the declared product on bytes received by the client, without proving the representation received by Googlebot. An HTTP status contrary to the plan (including 401/403/429/5xx when 200 is expected) remains fail for that client, never evidence of denial to Googlebot. Registry fields and sitemap are recorded without assessing usage preferences, TS42 or indexing and never delimit groups. Other fields and tolerances remain unsupported. Full line numbers and values are retained; reason previews disclose truncation.