Search Bot Access Investigation Template
Diagnose why a public resource cannot be fetched by a search crawler using status, redirects, robots rules and verified request evidence.
When to use this template
Use this when a public page works in your browser but a search integration reports a fetch error or a verified crawler request is blocked.
Choose another approach when: You want to expose private routes, disable the firewall broadly, or treat a spoofed user-agent string as proof that a request came from an official crawler.
Gather these inputs
- The exact public URL and observed failure with timestamp.
- Read-only access to relevant response headers and appropriately limited logs.
- The provider’s current crawler documentation and verification information.
Copy and adapt the template
Edit the text below and copy it into your own document. Edits stay in this browser field and are not submitted or saved by this page.
Worked example
Illustrative example, not a Crany customer result. Replace every example detail with facts you can verify.
A fictional public guide returns 200 in a signed-in browser but redirects anonymous requests to login. The investigation identifies authentication as the difference; changing robots.txt would not fix it. If the guide is intended to be public, the owner can correct that route’s access policy through the normal review process. A private dashboard remains private. In another case, a claimed crawler user agent alone is insufficient evidence to add a broad firewall exception. The worksheet preserves the distinction between observed access and asserted identity.
How to use it
- Reproduce the failure for the same URL without changing security settings.
- Locate the blocking layer and verify any crawler identity using current provider guidance.
- Propose the smallest scoped correction and verify public access after an authorized change.
Check before using it
- Private application pages and secrets are outside the public discovery scope.
- A user-agent label is not treated as authentication.
- The proposed fix preserves protections unrelated to the public page.
What to measure
Track the exact response before and after a scoped fix and check later verified crawl evidence. A simulated fetch passing does not prove an official crawler visited, indexed or recommended the page.
Research and sources
The references document crawler roles and access controls. The diagnostic sequence is original operations guidance; it is not permission to disable a firewall or a guarantee of indexing.
- OpenAI: Overview of OpenAI Crawlers — checked 2026-10-23. Separate search crawler access from model-training crawler controls.
- Perplexity: Crawlers — checked 2026-10-23. Search crawling, user-triggered fetches and access verification.
- Google: Robots.txt introduction — checked 2026-10-23. Crawl access rules and the distinction between crawl control, indexing and private access control.
About Crany
Crany is an AI marketing co-founder in development for software founders and vibe coders, designed to strategize, execute approved marketing and improve based on results. This free resource is usable independently of the product. Join the public waitlist.