Robots.txt Generator

Build a valid robots.txt for your website. Wizard or full controls, smart SEO score, undo/redo, import existing files, JSON backup and one-click download β€” all in your browser.

Smart SEO Score
Complete the form to see your score
0/100 Needs work

Smart Setup Wizard

✨ Answer a few questions and we'll build a safe, SEO-friendly robots.txt for you. You can fine-tune anything in Advanced Mode afterward.

We'll suggest {domain}/sitemap.xml or {domain}/sitemap_index.xml automatically. Nothing is fetched.

Live preview robots.txt

Reflects current settings. Full output appears after you generate.
Statistics live
0Groups
0Rules
0Allow
0Disallow
0Warnings
Add at least one user-agent group.

How to Use the Robots.txt Generator

  1. Try Easy Mode first β€” answer 4 quick questions and we generate a safe config automatically.
  2. Or switch to Advanced Mode to control every group, rule, crawl-delay and global directive.
  3. Watch the Smart SEO Score β€” it updates live and flags dangerous configs (blocking Google, blocking assets, conflicts).
  4. Import an existing robots.txt β€” paste it in via the Import button and it's parsed into editable groups.
  5. Use presets for quick starts, and undo/redo freely while experimenting.
  6. Generate, review the output, then Copy, Print, Download, or Export as JSON for backup.

Easy Mode vs Advanced Mode

Easy Mode is a beginner-friendly wizard. It asks about your platform, whether Google should crawl, and whether AI crawlers should be blocked β€” then generates a safe, sensible configuration. Advanced Mode exposes the full editor for every group and directive. Your choice is remembered on this browser.

How robots.txt works

The file lives at the root of your site: https://example.com/robots.txt. Crawlers fetch it before crawling and follow the rules for their user-agent. A * group applies to everyone. Allow rules always take precedence over Disallow rules at the same specificity β€” and Google uses the longest match wins rule.

Google-Extended is not Googlebot

Googlebot controls Google Search indexing. Google-Extended controls whether your content trains Gemini models. Blocking Google-Extended has no effect on search indexing. Blocking Googlebot removes your site from Google Search entirely. This tool treats them separately in the AI crawler preset.

Allow vs Disallow

Disallow tells crawlers to stay out of a path. Allow explicitly lets them back in β€” usually paired with a broader Disallow to carve out an exception. For example, block /admin/ but allow /admin/public/.

Wildcards

Google supports * (any sequence) and $ (end of URL). So Disallow: /*.pdf$ blocks all PDFs, and Disallow: /*?sort= blocks sorted URLs. Use sparingly β€” an over-broad pattern can deindex valuable pages.

Crawl-delay

This is not part of the official robots.txt spec. Google ignores it entirely; Bing, Yandex and others may respect it. If you need rate limiting, use it sparingly or rely on your CDN and Search Console settings instead.

Is my data uploaded?

No. Everything runs in your browser. Settings are auto-saved to this browser's localStorage only β€” nothing is sent to a server.

Frequently Asked Questions

What is robots.txt?A plain-text file at your site root that tells crawlers which paths they may or may not crawl. It is not a security mechanism β€” it only requests good behaviour from compliant crawlers.
Can robots.txt remove pages from Google?No. It only stops crawling, but pages may still be indexed if they are linked. To remove a page from the index, use noindex meta tags or Search Console removal tools.
Does robots.txt affect Google ranking?Indirectly. Blocking crawlers from pages you want indexed hurts. Blocking low-quality pages can help crawl budget. Accurate sitemap references improve discovery.
Should I block AI crawlers?That is a policy choice. The preset targets GPTBot, CCBot, ClaudeBot, PerplexityBot, Bytespider and similar agents. Google Search indexing is not affected β€” only Google-Extended is added (AI training opt-out).
Why does the WordPress preset warn about assets?It blocks /wp-includes/ and /wp-content/plugins/, which is standard for crawl-budget reasons. If your pages depend on assets from those paths, Googlebot may not render them. If SEO is critical, whitelist the exact CSS/JS files you need with Allow: lines.
Why is my robots.txt not blocking a page?Check that the file is at your site root, returns HTTP 200, uses plain text, and that the pattern matches the URL. Allow rules override Disallow at the same specificity.
What does Undo/Redo do?It steps back and forward through recent configuration snapshots (structural changes, presets, imports). Text edits inside fields are captured when you pause typing.
Where is my draft saved?In this browser's localStorage. Nothing leaves your device. Reset clears both the fields and the saved draft.