A file received.
An intention to clarify.
Understand
Define which pages crawlers should be allowed to access.
Check
In this example, robots.txt contains a Disallow directive for /prive/.
Open the robots.txt excerpt
Inspect the complete sourceUser-agent: * Disallow: /prive/Act
Specify which crawlers should access which pages, then compare these intentions with the received rules.
Recheck
After a change, read the rules again and check access with the same scope. Keep both readings.
An observed directive alone does not prove a crawler’s effective access.