How it works
Robots.txt generator writes one user-agent group and checks path syntax before export. An empty disallow list leaves crawling unrestricted for the selected agent.
Worked example
A Disallow line for /private/ asks matching crawlers not to request paths with that prefix.
Before you rely on the result
Robots.txt is public and voluntary. It does not protect private files or guarantee removal from search. Publish it at the site root; access control and indexing directives are separate decisions.
Cite this page
ToolOctopus. “Robots.txt generator.” Updated . https://tooloctopus.com/robots-txt-generator.
Add an access date if your instructions require one.
Citation formatting uses citeproc-js by Frank Bennett and Citation Style Language styles. License and source code.