Robots.txt Generator
Robots.txt generator online tool.
Processes your input locally. Review generated configuration before publishing.
Loading tool…
What this tool does
Create crawler directives with allow and disallow paths and an optional sitemap URL. Robots rules are not access controls.
How to use it
- Set user agent (initial value: *).
- Set disallow paths (one per line).
- Set allow paths (one per line).
- Set sitemap url (optional).
- Choose Process, inspect the result, then use the download control. A reported error means no successful output was produced.
Limits and result checks
Processes your input locally. Review generated configuration before publishing.
Common questions
Can robots.txt hide a private page?
No. Crawlers may ignore the file, and a disallowed URL can still appear in search if other sites link to it. Use authentication for private resources and an accessible noindex response when appropriate.
Where can I read the data-handling details for Robots.txt Generator?
Inputs are processed in browser memory and are not uploaded by this plugin. Generated downloads remain on your device. The site privacy policy explains memory cleanup and hosting logs.