CRAWLING RULES · TS04
Does your robots.txt
let the right pages through?
Start with a page that should be crawlable. We’ll compare your expectations with the received rules, then with the page response.
This check covers the addresses you choose. It does not discover every site resource or establish indexing.
What TS04 checks here
TS04-A01 compares your expectations with robots policy. TS04-A02 adds required-resource responses and your supplied review. Passing the first rule alone does not validate the control.
Googlebot · predeclared google-robotstxt-fields-v2 profile. HTTP client TSEP-Control-TS04/1, without cookies. Same origin; up to three redirects, five seconds and 1024 KiB per response. Exceeded limits, missing evidence and unsupported syntax remain visible.
You declare the inventory. Reviewer identity and truth are not authenticated. Robots.txt does not protect private areas or guarantee actual crawling or indexing.
Google robots.txt documentationEngine · 0.3.0-draft.9
Import evidence in expert mode