codeyaan

robots.txt Generator

SEO & Config

Create robots.txt files for search engine crawling rules

Runs in browser Instant No tracking

Configuration

Output

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://example.com/sitemap.xml

How to use

  1. 1Add user-agent rules (use * for all crawlers)
  2. 2Specify paths to disallow or allow
  3. 3Add your sitemap URL
  4. 4Copy or download the generated robots.txt file

FAQ

What is robots.txt?

robots.txt is a file placed at the root of a website that tells search engine crawlers which pages or sections they can or cannot access.

Does robots.txt block indexing?

No, robots.txt only controls crawling. Pages blocked by robots.txt can still appear in search results if other pages link to them. Use meta noindex tags to prevent indexing.

Where should I place robots.txt?

robots.txt must be placed at the root of your domain: https://example.com/robots.txt. It won't work in subdirectories.

Related tools

Meta Tag Generator

Generate HTML meta tags and Open Graph tags for SEO

.gitignore Builder

Build .gitignore files from popular templates for any stack

Chmod Calculator

Calculate Unix file permissions in numeric and symbolic notation