Robots.txt generator

Build a robots.txt file with a user-agent group, blocked paths and an optional sitemap.

On your device · No account

Privacy details

Robots.txt generator processes your input in this browser. Inputs and results are excluded from analytics. Recent tools save tool names, visit counts and last-visit times. Saved tools store only their names. Draft saving is optional and stays on this device; delete saved text with the draft controls. Loading text from a URL is optional and contacts that server; its URL and your IP address are visible to the server. An optional “Use in” action keeps a handoff in this tab. The next tool removes it on arrival and ignores it if more than a minute has passed.

Copy link includes the inputs you choose to share in the address. Anyone with that link can reopen them. Files and generated results are excluded. Password and key tools share settings only.

Disallowed paths, one per line

More options

User-agent

Sitemap URL (optional)

Your result

Reset reloads the tool with its starting values and releases open files and device controls. Saved drafts stay on this device until you delete them.

Links include your inputs and settings. Anyone with the link can see them, so leave out private text. Files and generated results are left out; random draws run again.

Keyboard shortcuts

Updated 2026-09-19

Using Robots.txt generator

Robots.txt generator writes one user-agent group and checks path syntax before export. An empty disallow list leaves crawling unrestricted for the selected agent.

Next: Meta tag generator

A worked example

A Disallow line for /private/ asks matching crawlers not to request paths with that prefix.

Before you use the result

Robots.txt is public and voluntary. It does not protect private files or guarantee removal from search. Publish it at the site root; access control and indexing directives are separate decisions.

Cite this page

Citation style

ToolOctopus. “Robots.txt generator.” Updated 2026-09-19. https://tooloctopus.com/robots-txt-generator.

Add an access date if your instructions require one.

Citation formatting uses citeproc-js by Frank Bennett and Citation Style Language styles. Licence and source code.