Free Tool

4.9/5

Free Robots.txt Tester

Test any site's robots.txt instantly. Check if a URL is allowed or blocked for Googlebot, Bingbot, GPTBot, and other crawlers, and see exactly which rule matched.

Website URL to check *

We fetch /robots.txt from the site's origin and parse every directive.

Want to track every link you share?

Create Flyn short links that count every click free; Pro shows from where and on what device. Every link becomes a data point.

Shorten for free

How to Test a Robots.txt in 3 Steps

Enter a website URL

Paste the URL of any site. The tool fetches /robots.txt from the origin and parses every directive, User-agent blocks, Disallow rules, Allow rules, Sitemap references, and Crawl-delay.

Review parsed rules

See every user-agent block with its rules, all sitemap references, and a syntax-highlighted preview of the raw file. Syntax errors, unknown directives, and risky configurations are flagged.

Test a specific URL

Enter any path like /admin/users and pick a user-agent (Googlebot, GPTBot, etc.) to instantly see whether it's allowed or blocked, plus the exact rule that matched.

What Is Robots.txt and Why Does It Matter?

Robots.txt is a plain-text file placed at the root of a website (for example, https://example.com/robots.txt) that tells web crawlers which parts of the site they are allowed to fetch. It's part of the Robots Exclusion Protocol (REP), an informal standard from 1994 that every major search engine honors. Every serious site has one.

Why does it matter for SEO? Because a single line in the wrong place can de-rank your entire site. Disallow: / in your production robots.txt blocks every crawler from every page. More subtle mistakes, blocking CSS, blocking paginated URLs, blocking the same path your sitemap references, quietly erode rankings without any error in Search Console. Robots.txt is also how you point crawlers to your sitemap, control AI training bots like GPTBot and Claude-Web, and set per-bot crawl rates.

Common Robots.txt Directives

Directive

Purpose

Example

User-agent

Names the crawler the rules apply to. Use * for all crawlers.

User-agent: Googlebot

Disallow

Blocks the crawler from fetching a path. Most-specific rule wins.

Disallow: /admin/

Allow

Explicitly permits a path, used to carve exceptions inside a disallowed directory.

Allow: /admin/public.html

Sitemap

Absolute URL to your XML sitemap. Multiple Sitemap lines are allowed.

Sitemap: https://example.com/sitemap.xml

Crawl-delay

Seconds to wait between requests. Honored by Bing, Yandex, Yahoo, not Google.

Crawl-delay: 10

Wildcards: * matches any sequence, $ anchors the end (e.g., Disallow: /*.pdf$).

Robots.txt vs Meta Robots vs X-Robots-Tag

These three tools sound similar but do very different things. Pick the right one for each situation.

robots.txt

Blocks crawling. Site-wide file. Cannot prevent indexing of URLs linked from elsewhere.

/robots.txt

Meta robots

Blocks indexing. Per-page. Crawler must be allowed to fetch the page to see it.

X-Robots-Tag

Blocks indexing via HTTP header. Works for non-HTML files like PDFs and images.

X-Robots-Tag: noindex

Robots.txt Best Practices

Always include a Sitemap reference

Add "Sitemap: https://www.example.com/sitemap.xml" to robots.txt. Any crawler will find it, not just the ones you submit to manually via Search Console.

Never block CSS or JavaScript

Google needs to render pages to understand them. Blocking /assets/ or /static/ breaks rendering and hurts rankings. Let Googlebot fetch everything the browser fetches.

Don't rely on robots.txt to hide sensitive pages

Robots.txt is public, anyone can read yours at /robots.txt. Listing /admin/ or /private/ there actually advertises these paths. Use authentication, not robots.txt.

Use noindex, not robots.txt, to remove from search

A page blocked by robots.txt can still appear in search results (Google sees it's linked, just not the content). To truly remove from search, allow crawling and use the noindex meta tag.

Test changes before deploying

A single character can de-rank your site. Validate syntax, test critical paths against Googlebot, and ideally stage the new file on a test environment first.

Block AI training crawlers if needed

To opt out of AI training, add User-agent blocks for GPTBot, Claude-Web, Google-Extended, CCBot, PerplexityBot, and Bytespider with Disallow: /. Not all bots honor this.

Crawler Rules + Link Tracking: Better Together

Test your robots.txt here for free, then use Flyn short links to count every click, with Pro breakdowns showing from where and on what device.

Track every click

Every Flyn short link tallies clicks in real time; Pro adds device, country, referrer, and time-series breakdowns.

Branded links

Use your own domain for short links on Pro. Build trust and increase CTR with branded URLs.

Editable destinations

Change where your short links point anytime. No more broken links when URLs change.

Instant redirects

Flyn redirects complete in under 50ms globally. Your users never notice the redirect.

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file is a plain-text file placed at the root of a domain (e.g., https://example.com/robots.txt) that tells web crawlers which parts of the site they are allowed to access. It uses a simple directive-based syntax: User-agent identifies which crawler the rule applies to, Disallow blocks specific paths, Allow explicitly permits paths, and Sitemap points to an XML sitemap. Robots.txt is part of the Robots Exclusion Protocol (REP), an informal standard dating to 1994.

Does Google always follow robots.txt?

Major search engines including Google, Bing, Yahoo, Yandex, and DuckDuckGo honor robots.txt directives for crawling, they will not fetch disallowed URLs. However, robots.txt is not a security mechanism. Google has stated that if a disallowed URL is linked from other sites, it may still appear in search results (without snippet or content) because Google knows the URL exists. Use the noindex meta tag or password protection to keep pages out of search results.

What's the difference between robots.txt and the noindex meta tag?

Robots.txt blocks crawling (the crawler will not fetch the page). Noindex blocks indexing (the crawler fetches the page but will not include it in search results). Importantly, a page blocked by robots.txt cannot be deindexed via noindex, because Google never crawls the page to see the noindex directive. To remove a page from search, allow crawling and add a noindex tag, or serve an X-Robots-Tag: noindex HTTP header.

How do I block specific bots like GPTBot, Claude-Web, or PerplexityBot?

To block AI crawlers, add a User-agent block for each bot you want to exclude. Example: "User-agent: GPTBot\nDisallow: /" blocks OpenAI's GPTBot from your entire site. Common AI bot user-agents include GPTBot (OpenAI), Claude-Web and ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google's AI training), CCBot (Common Crawl), and Bytespider (ByteDance/TikTok). Note that some AI bots ignore robots.txt, robots.txt is a polite request, not enforcement.

Can robots.txt hurt my SEO?

Yes, a misconfigured robots.txt is one of the most common SEO disasters. A single line like "Disallow: /" in your production robots.txt will de-rank your entire site. Other common mistakes: blocking CSS or JavaScript files (Google needs these to render pages), blocking paginated URLs that have canonical tags, blocking faceted search pages that then cannot pass signals via canonicals. Always test changes in a staging environment before deploying.

How often do search engines re-check robots.txt?

Google typically fetches robots.txt once every 24 hours and caches it. If the file is unreachable (5xx error), Google will temporarily stop crawling your site. If it returns a 4xx (including 404), Google treats it as "no restrictions" and crawls freely. To force Google to re-check your robots.txt sooner, resubmit your sitemap or use the URL Inspection tool in Google Search Console.

Does robots.txt support wildcards?

Google, Bing, and most major crawlers support two wildcard characters: asterisk (*) matches any sequence of characters, and dollar sign ($) anchors the end of the URL. For example, "Disallow: /*.pdf$" blocks all PDF files, and "Disallow: /search?*" blocks any URL starting with /search?. Wildcards are not part of the original 1994 standard, so some older crawlers may not support them.

What is crawl-delay and does Google honor it?

Crawl-delay is a directive that asks crawlers to wait N seconds between requests. Example: "Crawl-delay: 10" asks for a 10-second gap. Bing, Yandex, and Yahoo honor crawl-delay, but Google does not, it ignores the directive. To control Googlebot's crawl rate, use the Crawl rate settings in Google Search Console. For most sites, letting Google manage its own crawl rate is better than setting a manual delay.

Is robots.txt case-sensitive?

The file name must be lowercase: robots.txt, not Robots.txt or ROBOTS.TXT. The directive names (User-agent, Disallow, Allow, Sitemap) are case-insensitive. However, the path values in Disallow and Allow rules ARE case-sensitive, because URLs are case-sensitive. "Disallow: /Admin" will not block requests to /admin. The user-agent name string is matched case-insensitively by most crawlers.

Should I add my sitemap to robots.txt?

Yes. Adding "Sitemap: https://www.example.com/sitemap.xml" to robots.txt is a best practice, it tells any crawler (not just the ones you submit sitemaps to manually) where to find your sitemap. You can add multiple Sitemap lines if you have several sitemap files. The Sitemap directive must be an absolute URL (including https://), not a relative path.

Can I test a robots.txt file before uploading it to production?

Yes, this tool lets you test the actual live robots.txt file on any URL. To test a draft file before deployment, paste it into Google Search Console's robots.txt Tester (legacy tool still available for verified properties). You can also test specific URL/user-agent combinations with this tool's URL tester: enter a path like /admin/users and pick a user-agent to see if it's allowed or blocked.

What is the difference between Disallow and Allow?

Disallow blocks access to a path. Allow explicitly permits access, useful for carving out exceptions within a disallowed directory. Example: "Disallow: /private/\nAllow: /private/public-file.html" blocks the /private/ directory but allows that one file. When multiple rules match, Google applies the most-specific rule (longest path); if lengths tie, Allow wins. Not all crawlers implement Allow, it was added later as an extension.

More Free SEO Tools

Explore other free tools from Flyn to audit and improve your site.

Ready to take control of your links?

Test robots.txt for free, then create trackable short links with Flyn: click counts and QR codes free, analytics breakdowns and branded domains on Pro, all in one platform.

Shorten for free Read the API Docs