By Tangenix · Published 2026-10-23 · Free template

Robots.txt Change Review Template

Review crawl rules against public and private route intentions without treating robots.txt as authentication or a reliable removal mechanism.

When to use this template

Use this before changing robots.txt or investigating a crawler-access issue, with a clear list of routes that should remain publicly discoverable.

Choose another approach when: You need to secure private account data, remove already indexed content by blocking crawling alone, or bypass a site’s access restrictions.

Gather these inputs

• The current robots.txt fetched from the actual host.
• The intended public route groups and private access boundaries.
• The crawler documentation relevant to the policy being reviewed.

Copy and adapt the template

Edit the text below and copy it into your own document. Edits stay in this browser field and are not submitted or saved by this page.

Worked example

Illustrative example, not a Crany customer result. Replace every example detail with facts you can verify.

A fictional app intends /guides/ and /pricing to be public, while /workspace requires authentication.
A proposed broad Disallow: / rule would block intended public crawling and is rejected during review. The private workspace remains protected through actual authentication, not a line in robots.txt.
If a provider offers separate search and model-training crawler controls, those policies are reviewed separately. A generic crawler-like user-agent request is recorded only as a request test, not proof that the real provider’s crawler reached the site.

How to use it

1. Fetch the current file and list intended public paths before proposing a change.
2. Evaluate specific examples for each affected crawler group and preserve required public resources.
3. Verify the live rules after release while leaving authentication and private access protections intact.

Check before using it

• Robots.txt is not described as a security boundary.
• An intended public page is not accidentally covered by a broad block.
• Search crawling and training controls are not assumed to be identical across providers.

What to measure

Track whether intended crawl access matches the policy and investigate real crawler logs or provider reports when available. A policy file returning 200 does not establish indexing, recommendation or training behavior.

Research and sources

The sources distinguish crawl controls and provider-specific crawler roles. This review worksheet is original operational guidance and does not authorize weakening security, bypassing restrictions or changing a site’s training policy without a deliberate decision.

• Google: Robots.txt introduction — checked 2026-10-23. Crawl access rules and the distinction between crawl control, indexing and private access control.
• OpenAI: Overview of OpenAI Crawlers — checked 2026-10-23. Separate search crawler access from model-training crawler controls.
• Perplexity: Crawlers — checked 2026-10-23. Search crawling, user-triggered fetches and access verification.

This original template and worked example were prepared with AI assistance. Source documentation supports the principles stated above; it does not endorse Crany or guarantee this template’s performance.

About Crany

Crany is an AI marketing co-founder in development for software founders and vibe coders, designed to strategize, execute approved marketing and improve based on results. This free resource is usable independently of the product. Join the public waitlist.