Free Tool

4.9/5

Free Robots.txt Generator

Build a valid robots.txt file from a friendly form. Pick presets, add User-agent rules, set Crawl-delay, and point crawlers at your sitemap, with a live preview as you type.

Quick presets

Pick a preset to start, then customize. Existing rules will be replaced.

User-agent rules

User-agent

Allow paths (one per line)

Disallow paths (one per line)

Crawl-delay (seconds, optional)

Honored by Bing, Yandex, Yahoo, Baidu. Ignored by Google.

Sitemap URLs

(absolute URL, one per line)

robots.txt (live preview)

3 lines · 24 B

User-agent: *
Disallow:

Upload this file to the root of your domain so it's served at https://yourdomain.com/robots.txt.

Need branded short links for your sitemap and bot traps?

Track which crawlers (and which humans) hit your links with Flyn short URLs: free click counts, instant QR codes, and analytics breakdowns plus custom domains on Pro.

Shorten for free

How to Generate Robots.txt in 3 Steps

Pick a starting preset

Start from one of our presets, "Allow all", "Block all", "Block admin and API paths", or "Block sensitive directories", or build from scratch. Each preset is a sensible baseline you can edit.

Add User-agent rules and sitemaps

Add one row per crawler you want to target, Googlebot, Bingbot, AI bots like GPTBot, or the wildcard "*". Set Allow paths, Disallow paths, and an optional Crawl-delay. Add a Sitemap URL on its own line.

Copy and upload to /robots.txt

Copy the pretty-printed file with one click. Upload it to the root of your domain so it is served at https://yourdomain.com/robots.txt. Validate the live file with the Flyn Robots.txt Tester.

What Is a Robots.txt File?

Robots.txt is a plain-text file that lives at the root of your domain, for example, https://www.example.com/robots.txt , and tells web crawlers which parts of your site they may fetch. It is the front door of the Robots Exclusion Protocol (REP), an informal standard first proposed in 1994 and finally codified as RFC 9309 in 2022.

Crawlers fetch your robots.txt before they fetch any page on your site. If the file is missing or returns a 404, well-behaved bots assume nothing is off-limits. If it returns a 5xx server error, Google will pause crawling until the file is reachable again, which means a broken robots.txt can quietly stall your entire index. Getting it right matters: a single misplaced Disallow: / line will de-rank your whole site overnight.

This generator produces a syntactically valid robots.txt from a friendly form so you don't have to hand-roll the directives. It supports User-agent groups, Allow and Disallow patterns, wildcards and $ anchors, Crawl-delay, and one or more Sitemap URLs. The output renders live as you type, and you can copy the final file to your clipboard in one click.

Robots.txt Directives: The Quick Reference

The six directives you'll ever need, what each one does and which crawlers honor it.

`User-agent`

Names the crawler this rule group applies to. "User-agent: *" targets every bot; "User-agent: Googlebot" targets only Google's main crawler. You can list multiple User-agent lines before a single group of Allow/Disallow rules to apply the same rules to multiple bots.

`Disallow`

Blocks a path or pattern from being crawled by the user-agent above. "Disallow: /admin/" blocks every URL under /admin/. An empty "Disallow:" (no value) means "block nothing", useful for overriding wildcard rules. Paths are case-sensitive and must begin with a leading slash.

`Allow`

Explicitly permits crawling of a path, typically used to carve an exception out of a broader Disallow rule. "Disallow: /private/" plus "Allow: /private/public.html" blocks the whole directory except for one file. Allow is supported by Google, Bing, Yahoo, and most modern crawlers as an REP extension.

`Crawl-delay`

Asks the crawler to wait N seconds between successive requests. Honored by Bing, Yandex, Yahoo (Slurp), and Baidu, ignored by Google. Use it only if your server cannot keep up with normal crawl rates, and prefer a value of 1-10 seconds to avoid silently shrinking your index coverage.

`Sitemap`

Points crawlers at one or more XML sitemap files. Must use an absolute URL (https://www.example.com/sitemap.xml). Sitemap lines live at the file level, outside any User-agent group, and are read by every crawler that supports the directive. Adding it is a near-zero-effort SEO win.

`Host (deprecated)`

Originally a Yandex-only directive that named the canonical hostname. Yandex officially deprecated it in 2021 in favor of HTTP 301 redirects and rel=canonical. You can safely leave it out of every modern robots.txt, no major crawler still relies on it.

Common User-agents: Search Engines and AI Bots

The crawlers you'll most often want to target in your robots.txt.

Googlebot

Google's main web crawler. Also serves as a parent token: Googlebot-Image, Googlebot-News, and Googlebot-Video inherit rules from Googlebot unless they have their own group. Google-Extended is a separate opt-out for AI training (Bard, Vertex AI).

Bingbot

Microsoft Bing's crawler, also powering DuckDuckGo and Yahoo (in the US). Bingbot honors Crawl-delay, unlike Googlebot. Microsoft also runs MSNBot (legacy) and AdIdxBot (Bing Ads landing-page validation).

Slurp (Yahoo)

Yahoo's historic crawler. In most markets Yahoo Search results are now powered by Bing, so Slurp activity is regional (notably Japan). Honors Crawl-delay and the Allow directive.

DuckDuckBot

DuckDuckGo's own crawler, used to index a small share of results alongside Bing data. Lightweight traffic on most sites; blocking it usually has negligible SEO impact.

Baiduspider

Baidu's crawler, the dominant search engine in mainland China. Has its own quirks, including limited support for Allow and a stricter interpretation of Disallow patterns.

YandexBot

Yandex's crawler, the largest search engine in Russia and several CIS countries. Supports Allow, wildcards, $, Crawl-delay, and historically the now-deprecated Host directive.

GPTBot, Claude-Web, PerplexityBot

AI-training and answer-engine crawlers. Each one accepts a standard User-agent block. Many publishers block these to prevent unattributed content reuse in chatbots. Note that not every AI bot respects robots.txt, robots.txt is a polite request, not enforcement.

CCBot (Common Crawl)

Common Crawl powers the bulk training datasets behind many AI models. Blocking CCBot removes your site from those datasets going forward but does not retroactively scrub it from existing crawls.

Robots.txt vs Meta Robots vs X-Robots-Tag

These three controls do related but different things. Robots.txt controls crawling, whether the bot is allowed to fetch a URL. The <meta name="robots"> tag and the X-Robots-Tag HTTP header control indexing, whether the URL can appear in search results once it has been crawled.

These two layers are independent. A page can be crawled but noindex'd (Google fetches it, reads the tag, and silently drops it from the index). A page can also be blocked in robots.txt and still appear in search results as a bare URL if other sites link to it, because Google knows the URL exists but isn't allowed to fetch the actual page to see a noindex directive. To reliably remove a page from search, allow it in robots.txt and add noindex.

The choice between meta robots and X-Robots-Tag is mostly a delivery question. Meta tags live in HTML <head> and only work for HTML responses. The X-Robots-Tag HTTP header works on any response, PDFs, images, JSON endpoints, making it the right tool for non-HTML assets.

Robots.txt Best Practices

What we've learned from auditing thousands of production robots.txt files.

Don't block CSS, JS, or image files

Google needs to render your pages to score them for Core Web Vitals and mobile-friendliness. Blocking /assets/, /static/, or /*.css will hurt rankings. Always allow your build output and CDN paths.

Use one wildcard "*" group for the common case

Most sites need only a single "User-agent: *" group with a handful of Disallow rules. Add bot-specific groups only when you want to give a particular crawler narrower or broader access than the default.

Anchor file-extension rules with $

Use "Disallow: /*.pdf$" rather than "/*.pdf" to avoid accidentally blocking directories that contain ".pdf" in their name. The dollar anchors the pattern to the end of the URL.

Always include a Sitemap directive

Even if you submit your sitemap manually to Google and Bing, listing it in robots.txt helps every other crawler, AI bots, niche search engines, and verification tools, discover it without manual setup.

Never put sensitive paths in Disallow

Anyone can read your robots.txt. Listing /private-data/ or /api/internal/ in Disallow advertises those paths to attackers. Use authentication for real privacy.

Test before you deploy

A single "Disallow: /" left in production de-ranks your entire site. Run the file through the Flyn Robots.txt Tester (or Google Search Console's robots report) before pushing it live.

Crawl Rules + Branded Links: Better Together

Generate a clean robots.txt with the form above, then use Flyn to share trackable short links with every click counted free; Pro adds analytics breakdowns and custom domains so campaign URLs route through a domain you control.

Live preview

See the exact robots.txt update in real time as you add rules, no submit button, no surprises at copy time.

Sensible presets

One click to start from "Allow all", "Block all", or "Block admin and API paths", the patterns most sites need on day one.

Live in your browser

Everything runs client-side. We never see your draft rules, your paths, or your sitemap URLs.

Pairs with the tester

Generate here, then drop the file on your server and validate it with the free Flyn Robots.txt Tester in seconds.

Frequently Asked Questions

What is a robots.txt file and where does it go?

A robots.txt file is a plain-text file that tells web crawlers which parts of your site they are allowed to fetch. It must live at the root of your domain and be served at exactly /robots.txt, for example, https://www.example.com/robots.txt. It cannot live in a subdirectory or on a subdomain that doesn't match the host you want to control. The file is part of the Robots Exclusion Protocol (REP), an informal standard dating to 1994 and re-codified as RFC 9309 in 2022.

What is the basic syntax of robots.txt (User-agent, Allow, Disallow)?

A robots.txt file is built from groups of directives. Each group starts with one or more User-agent lines that name the crawler (e.g., "User-agent: Googlebot" or "User-agent: *" for all bots), followed by Allow and Disallow lines that whitelist or block path patterns. "Disallow: /admin/" blocks every URL under /admin/, and "Allow: /admin/public/" carves out an exception. A blank line ends a group. Order does not matter, for Google, the longest matching path wins, with Allow breaking ties.

How do wildcards (*) work in robots.txt?

Google, Bing, and most major crawlers support two wildcard characters inside Disallow and Allow patterns. The asterisk (*) matches any sequence of characters, including an empty one. For example, "Disallow: /*?sort=" blocks every URL that contains the ?sort= query string, regardless of the path before it. Wildcards are an extension, not part of the original 1994 spec, so very old or niche crawlers may ignore them. The User-agent line is not a wildcard pattern, only the bare star "*" means "all bots".

What does the dollar sign ($) do in robots.txt?

The dollar sign anchors a pattern to the end of the URL. It tells the crawler the match must end exactly here. "Disallow: /*.pdf$" blocks /report.pdf and /docs/whitepaper.pdf but not /pdf-archive/index.html. Without the $, the pattern .pdf would also match URLs that just contain ".pdf" anywhere in the path. Combined with the asterisk, $ is the cleanest way to block specific file types: /*.xls$, /*.zip$, /*.json$. Like the asterisk, $ is a Google/Bing extension and not in the 1994 standard.

What is Crawl-delay and which bots actually respect it?

Crawl-delay is a non-standard directive that asks a crawler to wait N seconds between successive requests. "Crawl-delay: 10" requests a 10-second gap. Bing, Yandex, Yahoo (Slurp), and Baidu honor it. Google does not, Googlebot ignores Crawl-delay completely and instead manages its own crawl rate; you can adjust Googlebot via the Crawl Rate setting in legacy Google Search Console. For most sites, the right answer is to let modern crawlers self-throttle rather than guessing a delay.

How do I point crawlers to my sitemap from robots.txt?

Add a Sitemap directive on its own line: "Sitemap: https://www.example.com/sitemap.xml". The URL must be absolute (include the full https://), relative paths are not allowed. You can list multiple sitemaps, one per line, including sitemap index files. The Sitemap directive is independent of any User-agent block, every crawler that reads your robots.txt will see it. Adding it is the lowest-effort way to ensure new crawlers (including AI bots) can discover your sitemap without manual submission.

What's the difference between robots.txt and the noindex meta tag?

Robots.txt controls crawling, whether the bot is allowed to fetch the URL at all. Noindex (in a <meta name="robots"> tag or the X-Robots-Tag HTTP header) controls indexing, whether the URL appears in search results. These two are independent: a page can be crawled but noindex'd, or blocked by robots.txt and still indexed if other sites link to it. Importantly, a URL blocked in robots.txt cannot be deindexed via noindex, because Google never crawls it to read the tag. To remove a page from search, allow crawling first, then add noindex.

Should I use robots.txt to hide sensitive or private data?

No. Robots.txt is a public file that anyone can read at /robots.txt, listing /admin/ or /private-keys/ in your Disallow rules tells attackers exactly where to look. It also does not prevent access; well-behaved bots respect it, but malicious crawlers and scrapers do not. For real privacy, use authentication (HTTP Basic, OAuth, session cookies), IP allowlists, or take the content off the public internet entirely. Robots.txt is a politeness signal for search engines, not a security boundary.

How do I validate my robots.txt file before publishing it?

Test it with our companion tool, the Flyn Robots.txt Tester, which fetches the live file from any URL, parses every directive, and lets you check specific paths against any user-agent to confirm they are allowed or blocked. Google Search Console also includes a robots.txt report under Settings that shows the last fetch result, errors, and warnings. Test before deployment by hosting the draft file on a staging URL and pointing the tester at it, a misconfigured production robots.txt can take your entire site out of the index.

How does Google actually read and apply robots.txt rules?

Googlebot fetches /robots.txt at the start of every crawl session and caches it for up to 24 hours. When deciding whether to crawl a URL, it walks all groups whose User-agent matches Googlebot (or a more specific token like Googlebot-Image), gathers all Allow and Disallow patterns, and picks the rule with the longest matching path. If Allow and Disallow have equal length, Allow wins. If robots.txt is unreachable (5xx), Google temporarily pauses crawling; if it returns 404, Google treats it as "no restrictions".

What are the most common robots.txt mistakes that hurt SEO?

Five big ones: (1) Blocking CSS, JS, or image files, Google needs these to render and rank your pages. (2) Leaving "Disallow: /" on a production site after a launch, which de-indexes everything. (3) Using robots.txt to hide thin or duplicate content that should be canonicalized or noindex'd instead. (4) Wrong path patterns, "Disallow: admin" without the leading slash matches nothing in Google's parser. (5) Forgetting that the file is case-sensitive on paths: "Disallow: /Admin" does not block requests to /admin.

Is robots.txt case-sensitive?

The filename itself must be lowercase: robots.txt, never Robots.txt or ROBOTS.TXT, crawlers will not look at the uppercase variants. Directive names (User-agent, Allow, Disallow, Sitemap, Crawl-delay) are case-insensitive, so "user-agent" and "USER-AGENT" both work. However, path values are case-sensitive because URLs are case-sensitive: "Disallow: /Admin" only blocks /Admin and does nothing for /admin or /ADMIN. Always test paths in the exact case they appear on your live site.

More Free SEO Tools

Pair this generator with our other technical SEO tools, free, instant, no sign-up required.

Robots.txt Tester

Validate the robots.txt you just built, fetch the live file and test any URL against any user-agent.

XML Sitemap Validator

Validate the sitemap you point to from robots.txt, structure, URLs, and lastmod fields.

Canonical URL Checker

Audit canonical tags on any URL, find duplicates and missing rel=canonical.

Meta Tag Generator

Generate SEO meta tags, Open Graph, and Twitter Cards with a live Google preview.

URL Shortener

Create branded short links with analytics.

Ready to ship your robots.txt?

Build it here in seconds, upload it to your domain root, then validate it with the Flyn Robots.txt Tester. Pair it with Flyn short links that count every click across every campaign, with country and device breakdowns on Pro.

Start free, no credit card Read the API Docs