Home › SEO Tools › Robots.txt Validator

Robots.txt Validator & Tester

Test any URL against your robots.txt rules, validate syntax, and check resource accessibility — all in one free tool.

Paste your robots.txt content above
How to use: Paste your robots.txt content, enter the URL you want to check, choose the user agent (or use * for all), and click Test URL. The tool will tell you whether the URL is allowed or blocked and which rule applies.
Note: Due to browser CORS restrictions, live fetch simulates robots.txt fetching from the domain root (/robots.txt). For accurate live testing of your own site, use the URL Tester tab with your actual robots.txt content.
Ready
1
    Why check resources? If Google can't access your CSS, JS, or images, it can't render your page properly — which can hurt your rankings. This tool checks each resource URL against your robots.txt rules.

    How This Robots.txt Tester Works

    Everything runs 100% in your browser — no data is sent to any server.

    1. Paste robots.txt
    Copy your robots.txt content and paste it into the editor or URL tester.
    2. Choose User Agent
    Select which bot you want to simulate — Googlebot, Bingbot, or any custom agent.
    3. Enter URL
    Enter the full URL you want to test and click the Test button.
    4. See Results
    Get instant results showing allowed/blocked status and the exact rule causing it.

    What Is a Robots.txt File — and Why Does It Actually Matter?

    A robots.txt file sits at the root of your website — always at yourdomain.com/robots.txt — and acts as a polite instruction sheet for web crawlers. When Googlebot, Bingbot, or any other crawler visits your site, the very first thing it checks is this file. It tells bots which pages they can crawl and which ones to skip.

    Here's the thing most people get wrong: robots.txt doesn't prevent pages from appearing in search results. It only stops crawlers from visiting those URLs. A page can still get indexed if another site links to it, even if it's disallowed in robots.txt. If you truly want a page out of Google's index, you need a noindex meta tag combined with proper crawl access.

    For SEO in 2026, robots.txt is as important as ever — especially with Google's rendering pipeline. If your CSS, JavaScript, or font files are accidentally blocked, Googlebot can't render your pages properly, which directly impacts how it understands and ranks your content.

    The Basic Structure

    Every valid robots.txt follows the same pattern: a User-agent line that names the bot, followed by one or more Allow or Disallow rules. Groups are separated by blank lines. The asterisk * matches all bots. Rules for a named bot like Googlebot always take priority over wildcard rules when both apply.

    A Disallow: / blocks the entire site. An empty Disallow: allows everything. When there's no robots.txt at all, bots assume full access — which is fine for most public websites.

    Robots.txt Directives — Plain English Glossary

    These are all the directives you'll encounter when reading or writing a robots.txt file. Understanding each one helps you write rules that actually do what you intend.

    User-agent
    Specifies which crawler the following rules apply to. Use * for all bots, or name a specific crawler like Googlebot.
    Disallow
    Tells a crawler not to visit URLs that match the given path. An empty value means nothing is disallowed — the bot can crawl everything.
    Allow
    Overrides a broader Disallow rule for a specific path. Commonly used to allow a file inside a blocked folder (e.g., allowing admin-ajax.php inside /wp-admin/).
    Sitemap
    Points crawlers directly to your XML sitemap. Can appear anywhere in the file and applies globally — not just to the nearest User-agent group.
    Crawl-delay
    Asks a crawler to wait N seconds between requests. Not supported by Googlebot (use Google Search Console crawl rate settings instead). Honored by Bingbot and others.
    Wildcard (*)
    A special character in path rules that matches any sequence of characters. /search/* blocks all URLs starting with /search/, regardless of what follows.
    End anchor ($)
    Forces the pattern to match only at the end of a URL. Disallow: /*.pdf$ blocks only URLs ending in .pdf, not URLs that contain .pdf somewhere in the middle.
    Rule priority
    When multiple rules match a URL, the most specific rule wins (longest matching path). If two rules have equal length, Allow takes precedence over Disallow.

    7 Robots.txt Mistakes That Quietly Hurt Your SEO

    Most robots.txt errors aren't obvious — they don't break your site, they just silently stop Google from doing its job. These are the ones we see most often when auditing sites.

    Blocking CSS and JavaScript files If Googlebot can't load your stylesheets and scripts, it sees a broken version of your page — and ranks it accordingly. Check every /wp-includes/ and /assets/ disallow rule you have.
    Using robots.txt to hide thin or duplicate content Blocking a URL with robots.txt doesn't remove it from Google's index — it just stops Googlebot from reading it. For pages you don't want indexed, use a noindex tag on the page itself, not a Disallow rule.
    Accidentally blocking the whole site with Disallow: / This is the most catastrophic and surprisingly common error — often introduced during a site migration or staging setup that never got reversed before launch. Run a validation check before pushing any robots.txt change live.
    Relative Sitemap URLs Writing Sitemap: /sitemap.xml instead of the full absolute URL (https://yourdomain.com/sitemap.xml) is technically invalid. Some crawlers handle it fine; others ignore it entirely.
    Forgetting that rules are case-sensitive Disallow: /Admin/ and Disallow: /admin/ are different rules. URLs are case-sensitive in robots.txt. Write your paths exactly as they appear on your server.
    Duplicate User-agent blocks for the same bot Having two separate User-agent: Googlebot blocks in one file creates ambiguity. Different crawlers interpret this differently — consolidate all rules for a bot into a single group.
    Not using this checker after every change Even a single typo in a path pattern can open up or block large sections of your site. Always validate your robots.txt with a tester like this one before deploying — especially after migrations, theme changes, or CMS updates.

    Robots.txt vs Meta Noindex vs Canonical — When to Use Which

    These three tools often get confused because they all deal with controlling what Google sees. They work very differently though, and using the wrong one can create invisible SEO problems.

    Method Stops Crawling? Removes from Index? Best Used For
    robots.txt Disallow Yes No Saving crawl budget — keeping bots away from admin pages, internal search, duplicate parameter URLs
    Meta noindex No Yes Removing a page from search results while still letting it be crawled (so Google can read the tag)
    Canonical tag No Consolidates signals Duplicate or near-duplicate content — tells Google which version is the "real" URL to rank
    Disallow + noindex combined Yes Won't work This is a trap — if Googlebot is blocked from the page, it can't read the noindex tag. Don't combine these.

    Frequently Asked Questions About Robots.txt

    These are the questions that come up most often — from beginners setting up their first file to developers debugging complex crawl issues.

    Does robots.txt prevent pages from being indexed in Google?
    No — and this is probably the most common misunderstanding. A Disallow rule stops Googlebot from visiting the URL, but the page can still appear in search results if Google discovers it through a link from another site. The URL shows up in results with a message like "no information is available for this page." To actually remove a page from Google's index, you need a noindex meta tag on the page — and that page must be crawlable so Google can read the tag.
    What happens if there's no robots.txt file on a website?
    Nothing bad. When a crawler looks for robots.txt and gets a 404 response, it simply assumes the entire site is open to crawling. This is the default behavior. You don't need a robots.txt file unless you actually want to restrict something — like protecting admin pages, avoiding crawl waste on search result pages, or pointing crawlers to your sitemap.
    How do I check if a URL is blocked by robots.txt?
    Use the URL Tester tab on this page — paste your robots.txt content, enter the URL you want to check, choose the user agent (Googlebot, Bingbot, etc.), and click Test. The tool will show you exactly whether the URL is allowed or blocked and which rule is responsible. You can also use Google Search Console's URL Inspection tool for live verification against Google's actual copy of your robots.txt.
    Can I use robots.txt to block a specific file type, like PDF files?
    Yes. Use a wildcard pattern with the end anchor: Disallow: /*.pdf$ blocks all URLs ending in .pdf. The dollar sign anchors the match to the end of the URL, so it won't accidentally block something like /pdf-guide/. You can apply this approach to any file extension — .doc, .xls, .zip, and so on.
    Is robots.txt case-sensitive?
    The directive names (User-agent, Disallow, Allow, Sitemap) are case-insensitive — you can write them in any case. But the path values are case-sensitive on most servers. Disallow: /Admin/ and Disallow: /admin/ are treated as two different rules. Always match the exact case of your actual URL paths.
    How often does Googlebot re-fetch robots.txt?
    Google typically caches your robots.txt for up to 24 hours, though in practice it often refreshes more frequently for active sites. If you make a critical change — like removing an accidental block — you can request a re-fetch through Google Search Console. Don't expect robots.txt changes to take effect instantly in Google's crawl behavior.
    What is the difference between Allow and Disallow in robots.txt?
    Disallow tells a bot not to crawl a path. Allow explicitly permits a path — its main purpose is to carve out exceptions inside a broader Disallow rule. For example, if you block all of /wp-admin/ but need Googlebot to access /wp-admin/admin-ajax.php (which powers live search and AJAX features), you add an Allow rule for that specific file. When both rules match a URL equally, Allow wins over Disallow.
    Should the Sitemap be inside a User-agent block or outside?
    Sitemap directives are global and apply to all crawlers regardless of where they appear in the file. The convention is to place them at the bottom of the file, outside any User-agent group — but placing them inside a group doesn't technically change their behavior. Use absolute URLs (starting with https://), and you can list multiple Sitemap lines if you have more than one sitemap file.