Robots.txt Generator
Build a robots.txt file with per-bot rules, blocked paths, sitemap link, and a crawl simulator.
Live Robots.txt Output
About this tool
Controlling exactly which search engines can crawl which parts of a site often means hand-writing Disallow rules and hoping the syntax is right. This generator gives you a full control panel: set a global allow/block policy, grant or deny specific bots like Googlebot, Google Image, Bingbot, Yahoo Slurp, Baiduspider, and Yandex individually, and list blocked paths with wildcard support — the tool auto-formats missing leading slashes.
Add a crawl-delay, sitemap URL, and canonical host, then use the built-in crawl simulator to test any path and instantly see whether it would be blocked or allowed under your current rules, before you ever publish the file.
Where to publish it
Download or copy the generated file and place it at the root of your domain as /robots.txt, the standard location every well-behaved crawler checks first.