Skip to the content

Free generator, runs in your browser

Robots.txt Generator with AI crawler presets

Write a correct robots.txt in a minute. Pick a policy for AI crawlers, list the paths to keep crawlers out of, add your sitemap and copy the file.

Build a robots.txt

Choose the policy. The file updates as you go.

Every other crawler
AI crawlers

Model training

Collect content to train AI models.

AI search

Index pages so assistants can find and cite them.

User requests

Open a page because a person asked an assistant.

One path per line.

Paths inside a disallowed folder that stay crawlable.

Ignored by Google. Leave it empty unless your server needs it.

robots.txt

Why use the Robots.txt Generator

AI policy presets

Allow everything, allow AI search but block training, or block every AI crawler, in one click.

Every known crawler

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and more, grouped by purpose.

Correct syntax

Groups, paths and the Sitemap line are written the way RFC 9309 expects, with no typos to debug.

Private

Everything happens in your browser. Nothing is uploaded.

When to use the Robots.txt Generator

New sites

Start with a clean file that allows crawling and declares the sitemap.

AI training opt out

Keep training crawlers out while staying visible in AI search.

Admin areas

Keep crawlers out of login, cart and search result pages.

Staging sites

Write a block-everything file for a staging host, and remember to remove it.

Replacing a CMS default

Swap a platform default for a file that says what you mean.

Explaining policy

The generated comments make the file readable for the next person.

Understanding Robots.txt

How to use the generator

  1. 1Default rule. Choose whether all crawlers may crawl the site by default.
  2. 2AI crawlers. Pick a preset, then adjust each crawler if you need to.
  3. 3Disallowed paths. List paths such as /admin/ or /cart/, one per line.
  4. 4Sitemap. Add the absolute url of your sitemap, then copy or download the file to the root of your site.

What robots.txt cannot do

robots.txt is a request to well behaved crawlers, not access control. Anything private needs authentication. It also does not remove pages from search results; use noindex for that, on a page crawlers are allowed to fetch.

Choosing an AI policy

Training crawlers collect content for future models. Search and user crawlers let assistants find and open your pages to answer questions. Blocking training while allowing search is a common middle ground for publishers.

Privacy and availability

The generator runs entirely in your browser. Nothing you type is sent to Aback or anywhere else, and it keeps working offline once the page has loaded.

Copy or download the result and publish it on your own site. Then check the live file with the matching checker, because what crawlers see is what your server sends.

Related tools and guides

Robots.txt CheckerCheck that robots.txt is published, parses cleanly and does not block the pages you want found. See at a glance which search engines and AI crawlers the rules allow.AI Crawler CheckerFind out whether ChatGPT, Claude, Perplexity and other AI assistants can read your site. The checker reads your robots.txt rules for each AI crawler and requests the page with each crawler's user agent, so a firewall block shows up too.llms.txt GeneratorGive language models a clean map of your site. Fill in the name, a one sentence summary and the links that matter, and copy a valid llms.txt.Sitemap CheckerCheck that your sitemap exists, parses as valid XML, is declared in robots.txt and lists urls that actually work. A broken sitemap wastes the crawl budget you give it.Meta Tag GeneratorWrite the head of a page with a live preview of the search result and the social card. Copy the tags into your template when they look right.Security Headers GeneratorChoose the headers and their values, pick your platform, and copy a configuration for Nginx, Apache, Cloudflare, Netlify or Vercel.Guide: robots.txt for AI crawlersWhich AI crawlers exist, what each one does, and how to write robots.txt rules that block model training while keeping your site visible in ChatGPT, Claude and Perplexity answers.

Frequently asked questions

Where do I put the file?

At the root of the host, so that it is served at https://your-domain/robots.txt. Each subdomain needs its own file.

Will blocking GPTBot remove me from ChatGPT?

Blocking GPTBot stops training use. Whether ChatGPT can find and open your pages depends on OAI-SearchBot and ChatGPT-User, which the presets list separately.

Should I block /wp-admin/?

It is common and harmless, but keep /wp-admin/admin-ajax.php allowed if your theme uses it for public features.

Can I use wildcards?

Yes. * matches any sequence of characters and $ anchors the end of the url, for example Disallow: /*.pdf$.

What does Crawl-delay do?

It asks crawlers to wait between requests. Google ignores it. Bing and some others honour it. Leave it out unless your server needs it.

Is an empty Disallow the same as allowing everything?

Yes. Disallow: with no path allows everything. Disallow: / blocks everything.

How do I check the file after publishing?

Run the robots.txt checker on your site. It reads the live file and shows what each crawler is allowed to do.

↑ ↓ to move↵ to openesc to close