Free tool

AI-ready robots.txt generator

Build a robots.txt that lets AI search engines find and cite your site, keeps your private paths private, and makes your own choice about AI training bots.

Loading…

AI search and answer bots

Allow these to appear in AI answers.

AI training bots

Blocking these doesn't remove you from AI search. It's a business choice.

Allow training Block training

Private paths (one per line)

Sitemap URL

robots.txt (upload to the root of your site)

After uploading, check it with the AI crawler checker.

What should an AI-ready robots.txt contain?

An AI-ready robots.txt explicitly allows the crawlers that power AI answers: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot, Claude-SearchBot, Googlebot (AI Overviews) and Bingbot (Copilot). It still disallows private paths like admin or checkout pages and points to your sitemap.

A crawler obeys only the most specific group that names it. If you add a group for OAI-SearchBot, it ignores the User-agent: * rules, so we copy your private paths into each group.
Training bots like GPTBot and Google-Extended can be blocked without affecting AI search visibility.
Firewalls can block bots even when robots.txt allows them. Test with the crawler checker after uploading.

Last updated: 14 October 2026

Is robots.txt the only problem?

Run the full audit to check firewall behaviour, JavaScript rendering, schema and more.

Run the full audit