Robots.txt Generator

Compose a robots.txt with rules and a sitemap.

Robots.txt GeneratorRuns in your browser

robots.txt

User-agent: *
Disallow: /admin
Disallow: /api
Disallow: /draft

Sitemap: https://example.com/sitemap.xml

Runs entirely in your browser. Nothing you enter is uploaded.

How to use

  1. 1Enter the User-agent (use "*" for all crawlers).
  2. 2List Disallow and Allow paths one per line, and add a sitemap URL and Crawl-delay if needed.
  3. 3Copy the output and save it as robots.txt at the root of your site.

When it helps

  • Keeping crawlers away from admin screens, APIs or draft pages.
  • Pointing search engines to your sitemap when launching a new site.

FAQ

Does Disallow keep a page out of search results?

No. robots.txt controls crawling, and a URL linked from elsewhere can still be indexed. To keep a page out, use noindex and leave the page crawlable so the tag can be seen.

Does Google respect Crawl-delay?

Google ignores Crawl-delay. Some other search engines, such as Bing, support it.

Where does robots.txt go?

It must sit at the root of the host (for example https://example.com/robots.txt). A file in a subdirectory is not read.

Related tools

Hreflang Generator

Generate hreflang tags for multi-market, multi-locale pages.

Meta Tag Generator

Generate SEO title, description and Open Graph meta tags.

JSON-LD Schema

Build structured data for Organization or Article.

Automate this inside your content workflow.

tixigen manages SEO, metadata and multilingual publishing in one flow.

See features