Sitemap robots crawlability auditor

Paste a sitemap and robots.txt to see which listed URLs a chosen crawler is allowed to fetch.

Privacy: runs in your browser

This tool processes your input in this browser. Inputs and results are excluded from analytics. Draft saving is optional and stays on this device; delete saved text with the draft controls. Loading text from a URL is optional and contacts that server; its URL and your IP address are visible to the server. When a button sends your result to another tool, it waits in this tab until that tool opens it, for up to a minute, and is then deleted.

Copy link includes the inputs you choose to share in the address. Anyone with that link can reopen them. Files and generated results are excluded. Password and key tools share settings only.

Options and local file

Optional settings apply only to this tab.

Your result

More
Keyboard shortcuts

Sharing and starting over

A copied link includes what you entered and your settings, so anyone with it can see them. Leave private text out. Files and results are not included; random draws are made again.

Start over puts the tool back to its starting values and closes any open file or device. Drafts you saved stay on this device until you delete them.

Sitemap robots crawlability auditor explained

Paste a sitemap and the robots.txt text for its origin to see which listed URLs the selected crawler may request under those rules. The report identifies the matching rule and duplicates. Specific crawler groups take priority over the wildcard group; equally specific Allow rules win ties.

Next: URL parser

Worked example

Disallow: /private/ blocks /private/draft for the wildcard group. Allow: /private/public$ permits exactly that path when it is the most specific match.

Before you rely on the result

A local rules check, not a live crawl or indexing verdict. It does not inspect HTTP responses, noindex, canonical tags, JavaScript or access restrictions. Sitemap indexes must be opened as individual child sitemaps first. All input URLs must share one origin; no URL is contacted.

Next: XML formatter

Cite this page

ToolOctopus. “Sitemap robots crawlability auditor.” Updated . https://tooloctopus.com/sitemap-robots-crawlability-auditor.

Add an access date if your instructions require one.

Citation formatting uses citeproc-js by Frank Bennett and Citation Style Language styles. License and source code.

Find a tool

Type a task, a unit pair, a sum or a time zone. ↓ to choose, Enter to open.