ABC Tools

How robots.txt Rules Decide What Gets Crawled

After reading this you can predict, for any URL and any crawler, whether a given robots.txt file allows or blocks it, and explain which single rule made the decision.

What a robots.txt file actually does

A robots.txt file sits at the root of a host, for example https://example.com/robots.txt. It tells crawlers which URL paths they may fetch. It is a request, not a wall: well-behaved crawlers like Googlebot read it and obey, but the file cannot force compliance and does not require a password.

Here is the hook. Suppose your file contains these two lines under one group:

  • Disallow: /admin/
  • Allow: /admin/help

Does Googlebot crawl /admin/help/faq.html? Reading top to bottom you might guess "blocked", because the disallow comes first. That guess is wrong. The rules are evaluated by how specifically each one matches the path, not by their order in the file. This article shows the exact procedure Google documents so you never have to guess.

When to test, and when the file is the wrong fix

Test a robots.txt file whenever traffic to a section drops after a deploy, whenever Search Console reports "Blocked by robots.txt", or before you ship a change to the file itself. A single stray Disallow: / can remove an entire site from crawling within a day.

robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other sites link to it, because Google can index the URL without fetching its content. To keep a page out of results, allow crawling and add a noindex directive, or protect it behind authentication.

So if your goal is "this page must never show up in Google", robots.txt is the wrong tool. If your goal is "stop wasting crawl budget on 40,000 faceted search URLs", robots.txt is exactly right.

The matching rule in plain terms

Google resolves an Allow or Disallow decision in three stages.

Group selection
Each block starting with one or more User-agent lines is a group. The crawler uses the single group whose user-agent token is the longest match for its own name. If nothing matches, it uses the User-agent: * group. Only one group applies; the others are ignored.
Longest-match precedence
Within that group, compare the path against every Allow and Disallow pattern. The rule with the longest matching pattern wins. Length is counted in characters of the pattern, so /admin/help (11 characters) beats /admin/ (7 characters).
Allow wins ties
If the longest matching Allow and the longest matching Disallow are the same length, Allow wins and the URL is crawlable.

Two wildcards change what "matches" means. * stands for any sequence of characters, including none. $ anchors the pattern to the end of the URL path. So /*.pdf$ matches /reports/2024.pdf but not /reports/2024.pdf?v=2, because the query string comes after the anchored .pdf.

The decision, written as a rule

For a path p, let L_A be the length of the longest Allow pattern that matches p, and L_D the length of the longest Disallow pattern that matches. Treat a missing match as length -1.

\text{crawl}(p) = \begin{cases} \text{allowed} & L_A \ge L_D \\ \text{blocked} & L_A \lt L_D \end{cases}

The symbol L_A is the character count of the winning allow pattern, L_D the character count of the winning disallow pattern. The \ge encodes both rules at once: Allow wins when it is longer, and it also wins on a tie. An empty Disallow: line (nothing after the colon) matches nothing, so it leaves L_D = -1 and permits everything.

A worked example you can reproduce

Testing three URLs against one file

Paste this file, choose user-agent Googlebot, and test the three paths below. These are the tool's demo defaults.

  • User-agent: *
  • Disallow: /admin/
  • Allow: /admin/help
  • Disallow: /*.pdf$
  1. /admin/help/faq.html: the Disallow: /admin/ pattern matches (7 chars), and Allow: /admin/help also matches (11 chars). Since L_A = 11 \ge L_D = 7, the URL is allowed.
  2. /admin/settings: only Disallow: /admin/ matches (7 chars). No allow matches, so L_A = -1 \lt L_D = 7. The URL is blocked.
  3. /downloads/guide.pdf: Disallow: /*.pdf$ matches because * covers /downloads/guide and the path ends in .pdf. No allow matches, so L_D = 7 wins. The URL is blocked.

Note that /*.pdf$ counts as 7 characters for precedence. The winning rule is reported for each URL, so you can see the reasoning, not just the verdict.

For each URL, the taller bar wins, except on a tie where Allow wins. Only the first URL has an Allow match tall enough to beat its Disallow.

Explore how length decides the winner

With the file from the worked example, lengthening the Allow pattern from /admin (6 chars) to /admin/help (11 chars) flips /admin/help/faq.html from blocked to allowed once the Allow length reaches or passes the Disallow length of 7. Shortening it below 7 blocks the URL again.

Reading and interpreting the results

The tool shows one row per URL with a verdict and the exact rule that produced it. Read the rule, not just the color. Three patterns of confusion are worth naming.

First, a URL reported as allowed may still fail to appear in search for reasons robots.txt never touches: a noindex tag, a 404 status, or a canonical pointing elsewhere. robots.txt only answers "may I fetch this".

Second, the reported rule tells you where to edit. If /downloads/guide.pdf is blocked by /*.pdf$ and you want it crawled, you add a longer allow such as Allow: /downloads/guide.pdf (21 characters), which beats the 7-character disallow.

Third, changing the user-agent can change everything, because a different group may apply. Test the specific crawler you care about.

Common mistakes

Match paths, not full URLs. Rules apply to the path and query after the host, so write Disallow: /search, not Disallow: https://example.com/search. A full URL with a scheme rarely matches anything.

Case matters. /Admin/ and /admin/ are different paths, and a rule for one does not cover the other. Trailing slashes matter too: Disallow: /reports (8 chars) blocks both /reports and /reports-2024, while Disallow: /reports/ (9 chars) blocks only URLs inside that folder.

Blank lines split groups. If you separate two User-agent lines with a blank line, you create two groups instead of one shared group, and each gets its own rules. Keep user-agents that should share rules on consecutive lines.

Finally, a robots.txt that returns a 500 error is treated by Google as "disallow everything" for the duration of the outage. A 404, by contrast, is treated as "allow everything". Serving reliability matters as much as the rules.

Related tools

Once you know which URLs are crawlable, list the allowed ones in a sitemap with the XML Sitemap Generator. If a URL you expected to be crawlable still misbehaves, trace its hops with the Redirect Chain Checker, since a redirect to a blocked path produces the same "not crawled" symptom.

Frequently asked questions

Does the order of rules in robots.txt matter?

No. Within the applicable group, Google picks the longest matching pattern regardless of line order, and Allow wins ties. Reordering an Allow and a Disallow of the same specificity changes nothing.

Why is my blocked page still showing in Google?

robots.txt blocks crawling, not indexing. If other pages link to the URL, Google can index it without fetching it, often showing the URL with no description. Use a noindex tag on a crawlable page to remove it from results.

How do I block one crawler but allow others?

Create a group with that crawler's exact user-agent token and its own rules. Because only the single best-matching group applies, a User-agent: Googlebot-Image group overrides the User-agent: * group for that crawler alone.

Is my pasted file sent anywhere?

No. The test runs locally in your browser, so nothing you paste leaves your machine.

What happens if robots.txt is missing?

A 404 on the robots.txt URL is treated as "everything allowed". Crawlers proceed as if no restrictions exist.

Try it yourself in the tool

Related tools

Redirect Chain Checker

Follow every hop: 301s, 302s, loops and chain length.

XML Sitemap Generator

Turn a list of URLs into a valid XML sitemap.

Keyword Density Checker

Word and phrase frequencies of any text, with stopword filter.

Meta Tag & Open Graph Generator

Generate meta tags and preview the Google, Facebook and X cards.

SERP Snippet Preview

Pixel-accurate preview of your Google result on desktop and mobile.

UTM Campaign URL Builder

Build tagged campaign URLs with correct encoding.