robots.txt Tester
Paste a robots.txt file, choose a user-agent and test any list of URLs against it. The tool applies the same rules Google documents — group selection, longest-match precedence, * and $ wildcards — and shows exactly which rule allowed or blocked each URL.
Runs 100% in your browser — nothing you paste leaves your device.
Read the full guide to this tool
Notes
- The most specific (longest) matching rule wins, and on a tie Allow beats Disallow — order in the file does not matter.
- robots.txt controls crawling, not indexing: a blocked URL can still appear in results if other sites link to it. Use noindex for pages that must stay out.
- Runs locally in your browser; nothing you paste is sent to any server.
Explore the idea
Related tools
Redirect Chain Checker
Follow every hop: 301s, 302s, loops and chain length.
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap.
Keyword Density Checker
Word and phrase frequencies of any text, with stopword filter.
Meta Tag & Open Graph Generator
Generate meta tags and preview the Google, Facebook and X cards.
SERP Snippet Preview
Pixel-accurate preview of your Google result on desktop and mobile.
UTM Campaign URL Builder
Build tagged campaign URLs with correct encoding.