Robots.txt Generator
Generate and validate robots.txt files to control how search engines crawl your website.
Basic Settings
Sitemap URL, host and crawl delay
Predefined Rules
Common crawling rules for your robots.txt
Blocked Paths
Add paths you want to block from crawling
Quick add:
Custom Rules
Advanced: add any directives manually
Live Preview
Your generated robots.txt
Validate Robots.txt
Paste your robots.txt content or enter your domain to fetch it automatically
Rules per User-Agent
Blocked paths ()
Allowed paths ()
No Allow/Disallow rules defined for this agent.
Sitemap References
Warnings
Errors
What is robots.txt?
Everything you need to know about controlling search engine crawlers.
Purpose of robots.txt
The robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site. It is placed in the root directory of your website and is the first file crawlers look for before indexing your content.
Always include a Sitemap
Adding a Sitemap directive to your robots.txt helps search engines discover all the pages on your site. The Sitemap URL should point to a valid XML sitemap, which greatly improves crawl efficiency and indexing speed.
Be careful with Disallow
Blocking too many paths can prevent search engines from indexing important content. Only disallow pages that should not appear in search results, such as admin areas, login pages, and temporary files. Always test changes before deploying.
Not a security measure
Robots.txt is a guideline, not a security tool. Malicious bots can ignore it entirely. Never rely on robots.txt to hide sensitive information. Use proper authentication, access control, and noindex meta tags for pages that must not be publicly accessible.
Want a comprehensive SEO analysis?
Robots.txt is just one piece of the puzzle. Check all SEO aspects of your website.
Run Full SEO AnalysisOther free SEO tools
Check other tools for analyzing your website