Robots & Sitemap fetches and validates the robots.txt file for a domain, summarises the crawl directives, and checks that the referenced sitemap is reachable and well-formed.
Enter a domain name (example.org) and run the tool. The probe fetches /robots.txt, parses the directives, and validates any sitemap it references.
Input: Domain — e.g. example.org.
The output shows the parsed robots.txt rules grouped by user-agent, the crawl-delay if set, and the result of fetching each declared sitemap (reachable / unreachable, URL count). A misconfigured Disallow can accidentally hide your site from search engines; an unreachable sitemap means crawlers cannot discover new content efficiently.
→ Verify robots.txt is not accidentally blocking important pages.
→ Confirm a referenced sitemap is reachable and well-formed.
→ Audit crawl directives after a site restructure.
→ Check that private paths are disallowed from crawling.
A Disallow rule blocks crawling, not indexing — a blocked URL can still appear in results if it is linked from other sites, just without a snippet. To truly keep a URL out of search, use noindex instead; use robots.txt only to manage crawl budget and keep crawlers out of private paths.
Yes. A Sitemap: directive in robots.txt helps crawlers find your sitemap without you submitting it to every search console. It is a good complement to, not a replacement for, direct submission in Google Search Console.
crawl-delay asks crawlers to wait N seconds between requests. Googlebot ignores it; Bing and some others honour it. It is a weak signal — for real rate control use server-side throttling rather than relying on crawl-delay.