Free robots.txt Generator

Build a valid robots.txt with one click. Allow Googlebot, block AI training bots like GPTBot and ClaudeBot, or write your own rules. Free, no signup.

1. Tell us about your site

Tap the eye for a beginner-friendly explanation.

How should crawlers behave?

2. Block AI training bots?

Tick a bot to add a Disallow rule. Many sites block training bots but allow live citation bots.

3. Your robots.txt

User-agent: *
Disallow:

Save this as robots.txt and upload it to the root of your site, so it lives at yoursite.com/robots.txt.

What is robots.txt?

robots.txtis a small text file at the root of your site that tells web crawlers what they can and can't access. Search engines, AI training bots, and SEO tools all read it before scraping your pages.

A good robots.txt does three things: lets the search engines you want (Google, Bing) crawl your public pages, blocks private sections like admin or staging, and points to your sitemap.xmlso search engines find all your URLs. If you don't want AI companies training on your content, you can also block bots like GPTBot, ClaudeBot, and Google-Extended in the same file.

How to use this generator

  1. 1. Add your web address. We use it to add the Sitemap line at the bottom.
  2. 2. Pick a mode. Allow all is the default and right for most sites. Block all is for staging and development. Custom lets you write your own rules.
  3. 3. Decide on AI bots. Tick the AI training bots you want to block. Most sites block GPTBot and Google-Extended.
  4. 4. Copy or download. Save the file as robots.txt and upload it to the root of your site. Then open yoursite.com/robots.txt to check it works.

Frequently asked questions

What is a robots.txt file?

A small text file at the root of your site (yoursite.com/robots.txt) that tells web crawlers — Google, Bing, AI training bots, SEO tools — which pages they can and can’t access. It’s the first file most crawlers check before doing anything else on your site.

Where do I put the robots.txt file?

At the root of your domain — the same place as sitemap.xml and llms.txt. The full address should be https://yoursite.com/robots.txt. Subdomains (blog.yoursite.com) need their own robots.txt; one robots.txt does not cover all subdomains.

Should I block AI training bots like GPTBot and ClaudeBot?

It depends on what you want. Blocking GPTBot, ClaudeBot, anthropic-ai, and Google-Extended stops your content being used to train AI models — useful if you sell content or want to protect IP. But it may reduce how often you’re cited in AI answers. Many sites block training bots but keep ChatGPT-User (the live search bot) so they can still appear in ChatGPT search.

What’s the difference between robots.txt, sitemap.xml, and llms.txt?

robots.txt tells crawlers what to skip. sitemap.xml is a complete list of your pages so search engines can index them all. llms.txt is a curated list of your most important pages, written for AI tools to cite. All three live at the root of your domain.

Will robots.txt actually stop bots from accessing my pages?

Most well-behaved bots (Google, Bing, GPTBot, ClaudeBot) honor robots.txt. Bad-faith bots can ignore it. So robots.txt is a strong signal, not a security wall. For pages you really need protected, use authentication, not robots.txt.

You'll also want