Robots.txt Generator

Build your robots.txt

Start from a preset, then adjust the rules. The file on the right updates live.

Quick presets
Rules (paths start with / — use * as a wildcard and $ for "ends with")
No sitemap yet? Create one with our XML Sitemap Generator.

Your robots.txt


            

Test a URL

How to use robots.txt the right way

  • Block only what has no search value — admin areas, cart and checkout pages, internal search results and filter URLs.
  • Never block CSS or JavaScript that your pages need; Google must render pages like a visitor.
  • Always add your sitemap so crawlers find all important pages.
  • Don't use robots.txt to hide private pages — the file is public. Protect them with a login instead.

After uploading, check your homepage with the SEO Analyzer to confirm it is crawlable.

FAQ

Frequently asked questions

What is a robots.txt file?
robots.txt is a small text file at the root of your website (yourwebsite.com/robots.txt) that tells search engine crawlers which parts of the site they may or may not crawl. It is the first file Google checks when it visits your site.
Does Disallow remove a page from Google?
No. Disallow only stops crawling. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag instead.
Should I block AI bots like GPTBot?
It is your choice. Blocking AI crawlers (GPTBot, Google-Extended, CCBot, ClaudeBot and others) stops your content being used to train AI models, but may also reduce how often AI assistants mention your brand. Use the "Block AI crawlers" preset if you want to opt out.
Where do I upload robots.txt?
Upload it to the main (root) folder of your website so it opens at https://yourwebsite.com/robots.txt. It must be named exactly robots.txt in lowercase. Each subdomain needs its own file.
How do I test my robots.txt?
Use the "Test a URL" box on this page to check whether a path is allowed for Googlebot. After uploading, the SEO Analyzer will also tell you if your page is blocked.
Need more than a tool?

Let SVC Digital fix it for you

Our team turns these reports into results — technical SEO fixes, faster websites, content that ranks and campaigns that bring leads.