robots.txt Generator

A robots.txt file is the first thing a well-behaved crawler reads before touching anything else on a domain, and its syntax is unforgiving: one Disallow: / left in from a staging environment quietly deindexes the whole site, and there is no warning when it happens. This tool builds the file from structured fields instead of raw text, so the output is always syntactically valid for whatever rules you actually intended.

Add one rule group per user-agent you need to target, list Allow and Disallow paths one per line, and optionally set a Crawl-delay and one or more Sitemap URLs. The output updates as you type and is ready to copy straight into a file named exactly robots.txt at your domain’s root.

Worked Example: Blocking Cart and Checkout Pages

Click Load Sample and the first rule group fills with user-agent * and 3 Disallow lines: /wp-admin/, /cart/, and /checkout/, plus a sitemap URL. The generated output is a single User-agent: * block with those 3 Disallow: lines, a blank line, then Sitemap: https://example.com/sitemap.xml — the exact shape most WordPress and e-commerce sites need out of the box. Pressing Load Sample again cycles to a group scoped to a single bot, Googlebot-Image, then to a group using wildcard query-string patterns (/*?*sort=) alongside Crawl-delay: 10.

The Controls: User-Agent, Paths and Crawl-Delay

• A path typed without a leading slash is corrected automatically: entering wp-admin/ in the Disallow box outputs Disallow: /wp-admin/, since every real robots.txt path is relative to the domain root.
• A rule group with no Allow, Disallow, or Crawl-delay value contributes nothing to the output. Newly added groups stay invisible in the generated file until you fill in at least one directive, so you can stage a second group mid-edit without it emitting a stray empty User-agent: * block.
• Crawl-delay must be digits only. A non-numeric value shows “Crawl-delay must be a non-negative whole number of seconds.” Google itself ignores this directive; Bing and Yandex still honour it.
• Sitemap URLs must be absolute. A relative path like /sitemap.xml triggers “Sitemap URLs must be absolute (start with http:// or https://).” because the Sitemap directive has no domain context of its own to resolve against.
• Google stops reading a robots.txt file at 500 KiB and ignores everything past that point, so an enormous list of Disallow paths eventually stops working silently rather than erroring. This tool has no size limit of its own, but Google’s does.

When Not to Use This: robots.txt Isn’t Access Control

• robots.txt is a public file and a request, not a lock. Anyone, including a browser, can fetch /robots.txt and read exactly which paths you consider sensitive enough to hide from crawlers.
• A page you Disallow can still appear in search results without its content if another site links to it — Google may index the URL from that link alone. Use a noindex meta tag or HTTP header, not robots.txt, to guarantee a page stays out of search results.
• Well-behaved crawlers respect Disallow; nothing forces a scraper or bot to. Real access control belongs behind authentication or a server-level rule, not a text file that only asks nicely.