How to use the Robots.txt Generator
- Choose the default policy.
- List paths to disallow.
- Optionally block AI crawlers and add your sitemap URL.
- Download and upload to your site root.
About this tool
robots.txt sits at the root of your domain and tells well-behaved crawlers which paths they may fetch. It's used to keep admin areas, cart pages and internal search results out of crawl budgets, and — increasingly — to opt out of AI model training crawlers.
robots.txt is a request, not a lock. It does not hide pages from people or stop malicious bots. Never rely on it to protect private data; use authentication.
Also note: blocking a page in robots.txt doesn't remove it from Google if other sites link to it. To de-index a page, allow crawling and add a noindex meta tag instead — you can generate one with our meta tag generator.
Frequently asked questions
Where does robots.txt go?
At https://yourdomain.com/robots.txt — subfolders are ignored.
Does blocking GPTBot affect Google rankings?
No. Google-Extended controls AI training only; Googlebot still indexes you.