Free tool
Robots.txt Generator
Tell search engines and AI crawlers which parts of your site they can visit, without memorising the syntax.
Rule 1
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Upload this file to the root of your domain, e.g. https://example.com/robots.txt
Robots.txt best practices
- Never block your CSS and JavaScript files; Google needs them to render your pages.
- Block low-value pages like internal search results, carts and admin areas.
- Always include your sitemap URL so crawlers can find every page.
- Test changes in Google Search Console before relying on them.
FAQs
What is robots.txt?
robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may or may not request.
Does Disallow remove a page from Google?
Not necessarily. Disallow stops crawling, but a blocked page can still be indexed if other sites link to it. Use a noindex meta tag to keep a page out of search results.
Should I block AI crawlers?
It's a business decision. Blocking GPTBot, ClaudeBot and similar bots stops them using your content for AI training, but may reduce how often AI assistants reference your site.
More free tools
Website Grader
Score any website for SEO, speed, mobile, trust and AI search, with a fix list.
Meta Tag Generator
Generate SEO title, description, Open Graph and Twitter tags in seconds.
Google SERP Preview
See how your page title and description will look in Google results.
Keyword Density Checker
Find the most used words and phrases in any piece of content.
Want experts to do it for you?
Book a free strategy call. We'll audit your current marketing and show you exactly where the growth is.