Robots.txt Validator & Tester

Validate robots.txt syntax, inspect parsed rule groups, and test whether a URL path would be allowed or blocked for any user-agent.

Robots.txt Content
Validation Results
✓ No syntax errors or warnings found
URL Tester
BLOCKED— Matched disallow: /admin/ (line 4)
Parsed Rule Groups
User-agent:Googlebotline 2
allow/public/:3
disallow/admin/:4
disallow/api/:5
crawl-delay10:6
User-agent:*line 8
disallow/private/:9
allow/:10
Sitemaps
sitemaphttps://example.com/sitemap.xml:12

How It Works

1

Paste your robots.txt

Paste the contents of your robots.txt file into the editor. A sample is pre-loaded to get started.

2

Review validation results

The tool checks syntax, groups rules by User-agent, and flags errors and warnings.

3

Test specific URLs

Enter a user-agent name and URL path to check whether that path would be allowed or blocked.

FAQ

User-agent, Allow, Disallow, Sitemap, Crawl-delay, and Host. Unknown directives trigger a warning.
Patterns support * (wildcard) and $ (end anchor) per the robots.txt specification. The most specific matching rule wins — longer patterns take priority.
Yes. 'Disallow: /' blocks all paths for the specified user-agent. To allow everything, use 'Allow: /' or leave Disallow empty.
No. All parsing and validation runs in your browser. To validate your site's Open Graph tags, try the OG Tag Validator.

Related Tools

DNS LookupCORS TesterSSL CheckerOG Tag ValidatorHTTP Status CodesSitemap ValidatorUser Agent Parser

Building on or debugging infrastructure?

Plan your stack and deployment before it becomes a problem — run a Discovery session.

Built & Maintained by Varstatt

Varstatt is a one-person product studio run by Jurij Tokarski, product engineer since 2011. These tools are free and open — no signup, no catch.