Robots.txt Generator
Generate robots.txt
What Is This Tool?
The Robots.txt Generator builds a valid robots.txt file from user-agent blocks with Allow/Disallow rules and an optional sitemap URL. Search engines and crawlers read this file to decide which parts of your site they may crawl.
How to Use
Set the User-agent for a block (use * to target all crawlers), then add Disallow or Allow path rules.
Click "Add User-agent Block" for additional crawler-specific rule sets.
Optionally add your Sitemap URL, then click Generate.
Copy or download the result and upload it as robots.txt at your site's root.
Features
- Multiple User-agent blocks with independent rule sets
- Allow and Disallow rules per block
- Optional Sitemap directive appended at the end
- Add/remove blocks and paths dynamically
- Download as a ready-to-upload robots.txt file
- 100% client-side — nothing leaves your browser
Examples
Output:
User-agent: *
Disallow: /admin/
Disallow: /account/
Sitemap: https://example.com/sitemap.xml
Common Use Cases
- Blocking crawlers from indexing admin or account areas
- Pointing search engines to your XML sitemap
- Setting different crawl rules for specific bots
- Quickly scaffolding a robots.txt for a new site
Frequently Asked Questions
Where do I put the generated robots.txt file?
Upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Crawlers look for it at that exact path.
What does User-agent: * mean?
The asterisk is a wildcard that matches all crawlers that don't have a more specific block defined elsewhere in the file.
Does robots.txt guarantee pages won't be indexed?
No. Disallow only requests that compliant crawlers not fetch those paths; it does not guarantee removal from search results. Use a "noindex" meta tag or HTTP header for that.
Can I target specific bots like Googlebot?
Yes, add a separate block with that bot's exact user-agent name (for example "Googlebot") and its own set of Allow/Disallow rules.
Do I need a Sitemap line?
It's optional but recommended. It gives crawlers a direct pointer to your XML sitemap so they can discover pages more efficiently.
Is my site information sent anywhere?
No. The file is generated entirely in your browser using JavaScript. Nothing is uploaded or logged.