TS05Access and indexing

Does your exclusion plan use the right method?

Avoid relying on robots.txt to remove a page from Google or protect private content. This check examines whether your declared methods match your goal.

Check my plan

In the app: Declared plan
You provide: Target, goal and mechanism
Allow for: Reading and confirmation

Check my plan

Understand

Three goals. Three decisions.

Avoid relying on robots.txt to remove a page from Google or protect private content. This check examines whether your declared methods match your goal.

According to Google

Robots.txt regulates crawling; it replaces neither index exclusion nor access protection. TSEP assesses the declared plan’s consistency here.

Introduction to robots.txt Source accessed on 2026-10-26

Check

Your goal. Your chosen method.

In TSEP Control

A declared-plan consistency result, a next action, a copyable summary and an export that can be replayed offline. Editing your answers creates a new version.

Your contribution

A URL, visibility and access goals, then your planned methods. You can answer “I do not know”. You confirm the recap before the check.

With your own tools

First write down the goal: limit crawling, remove from search or protect access. Then compare robots.txt, noindex and authentication with official guidance. Testing the actual site remains a separate step.

What this result covers

Your declared model only. No site visit, Google status check or protection test. A consistent plan is not proof that the page has disappeared from search results or that its content is private.

Act

Each decision. Its proper mechanism.

Choose a mechanism suited to the goal: limit crawling, exclude from the index or protect access.

Recheck

The intent changes. So does the plan.

Repeat the workflow with the same scope after correction. Date the new observation and retain earlier evidence.

Your reference points

To go further.

Statuses, sources and method.

Understand the statusesC · NC · NT · NA

The same legend in guides, analysis and demo. Each status depends on the control’s evidence and rules.

C · Verified: The criterion is met within the assessed scope. Keep dated evidence for the next comparison.
NC · To correct: Choose a mechanism suited to the goal: limit crawling, exclude from the index or protect access.
NT · Not established: Evidence or confirmations are insufficient to conclude. Supply what is missing; this status does not establish an SEO error.
NA · Not applicable: The control does not apply to the scope under its rules, with a retained justification. Missing evidence alone does not make it non-applicable.

Sources and references3 references

Detailed method and rules1 rule

Control method and criteria

Method
Compare a decision map dated before observation—crawl, indexing goal and method, noindex visibility, protection, and the scope of claims—with the supplied configuration record.

Acceptance criteria
A pass indicates only internal consistency of the supplied plan. Crawling, indexing and protection are separate; Disallow alone guarantees neither deindexing nor confidentiality. No actual index or server state is established.

Non-applicability
No relevant policy, with scope and decision documented.

Detailed rules

TS05-A01 · Supplied map: crawling, indexing and protection are distinct.

Protocol assessment mode : assisted, with supplied evidence and declarations. App availability is shown on the control card.

Scope
A crawl policy, indexing objective or confidentiality claim exists in scope.

Evidence
Configuration, prior intent map, dates, scope, methods/effects and claims with bases; both records cover the target.

Method
Read intent.ts05_policy, dated before observed_at, and the configuration record. Compare crawl methods/effects, indexing goal/method and noindex visibility, then protection and the bases for deindexing/confidentiality claims. The policy_version=2 profile also compares authentication, authorization and HTTP 404/410 removal. Exclusion hidden by Disallow is contradictory even without a guarantee claim. Results concern declared plan consistency only; neither deindexing nor confidentiality is established.

Acceptance criteria
Pass means only that the supplied map is internally consistent and separates the three axes. A deindexing guarantee based only on Disallow fails; an inaccessible noindex directive does not establish exclusion. No positive verdict proves index state, real crawler access or server confidentiality.

When the result remains inconclusive
Missing, undated, incomplete or future plan; unsupported method; unknown index state; blocked or unknown noindex visibility; incomplete removal request; indeterminate protection basis.

Non-applicability
This atomic test does not accept a partial exemption. Whole-control NA requires a complete, reasoned inventory review dated before observed_at, establishing absence of relevant policy for every target.

Assumptions
Records are supplied and unauthenticated. The interpreter tests their form and declared consistency, not their truth or implementation.

Limitations
Semi-automated mechanism over a synthetic model. It does not parse live robots.txt, meta, X-Robots-Tag or HTTP; observe visits, indexing or removals; or test access controls. Pass qualifies only the consistency of the supplied map.

Rule version 0.3.0-draft.8 · Public protocol reference · Versioned source

Put it into practice

← Back to controls