Robots.txt Tester
Fetch robots.txt and test a path
What this tool does
Fetches robots.txt and helps you review crawl instructions for search-engine bots.
How to use it
Enter the website URL and, when available, a path you want to test.
What the results mean
Blocked paths may be intentionally hidden from crawlers. Robots.txt controls crawling, not guaranteed indexing.