robots.txt Generator
Choose a crawl policy, add disallow paths and your sitemap, then copy a valid robots.txt. Free, no sign-up.
Where it goes
Save the output as robots.txt in your site's root so it is served at
https://yoursite.com/robots.txt. Adding a Sitemap: line helps search
engines discover every page — point it at your XML sitemap. Remember robots.txt controls
crawling, not indexing; to keep a page out of results, use a noindex tag.
Frequently asked questions
What is a robots.txt file?
robots.txt is a small text file at the root of your site (example.com/robots.txt) that tells search-engine crawlers which paths they may or may not request. It is a crawling hint, not a security control.
How do I use this generator?
Pick a policy, list any paths to disallow (one per line), add your sitemap URL, then copy the output and save it as robots.txt in your site root.
Should I block pages I want kept private?
No — robots.txt only asks polite crawlers not to fetch a path; it does not hide it. For truly private pages use authentication or a noindex tag, not robots.txt.
Is this tool free?
Yes — it runs in your browser, is free, and needs no sign-up.