robots.txt Generator
SEO Tools
Loading tool...
About This Tool
Free, private, serverless robots.txt generator. Visual rule builder for web crawlers. Add user agents, disallow/allow paths with a friendly UI. Includes presets for Googlebot, GPTBot, ChatGPT-User, AdsBot, and CCBot. Optional sitemap and host fields. Download or copy the result. No data leaves your browser — 100% client-side.
How to Use
- Add crawler rules — set user-agent and disallow/allow paths
- Or click Load Preset for common configurations
- Optionally enter Sitemap URL and Host
- Click Generate to build your robots.txt
- Copy or download the result
Frequently Asked Questions
What are the preset rules?
The preset includes rules for all crawlers (*), Googlebot (allow all), AdsBot-Google (block), GPTBot (block AI training), ChatGPT-User (block), and CCBot (block).
What is the Sitemap field for?
Adding your sitemap URL tells search engines where to find your XML sitemap for better indexing.
Can I block AI crawlers?
Yes, the preset includes rules to block GPTBot, ChatGPT-User, and CCBot which are commonly used for AI training data collection.
What are the use cases?
SEO configuration, controlling search engine indexing, blocking unwanted crawlers, protecting private content, managing AI bot access, and website launch preparation.
Guides & Tutorials
In-depth guides for every tool. Learn how to process your data privately, for free, right in your browser.
Read more