Robots.txt Generator
Build a properly formatted robots.txt file to tell search engine crawlers which parts of your site they can and cannot access.
How to Use the Robots.txt Generator
- Enter the user-agent (leave as * to target all crawlers) and your sitemap URL.
- List any paths you want to explicitly allow or disallow, one per line.
- Check "Block all crawlers" if you want to keep a staging site out of search results entirely.
- Click "Generate robots.txt", then copy the text or download the file.
- Upload the downloaded file to the root directory of your domain (e.g. example.com/robots.txt).
Why Use This Tool?
What does robots.txt actually do?
robots.txt is a plain text file placed at your site's root that tells well-behaved search engine crawlers which URLs or directories they should avoid crawling. It does not remove pages from search results by itself — for that you need a noindex meta tag or a removal request — but it does help control crawl budget and keep sensitive or duplicate paths out of the crawl queue.
Where does the file need to be placed?
Always at the root of your domain, e.g. https://example.com/robots.txt. Placing it in a subfolder means crawlers will never find it.
Should I block all crawlers?
Only on staging or development sites that should never appear in search results. Blocking crawlers on a live production site will stop search engines from indexing any of your pages, which is almost always the opposite of what you want.
Is this tool safe to use for a live website?
Yes. Everything is generated locally in your browser — nothing is uploaded to any server — so you can review the output carefully before uploading it to your own site. It's completely free with no signup required.