Robots.txt Generator
SEO-03Build a valid robots.txt file to control how crawlers index your site.
Configuration
Crawler access
Disallow paths (one per line)
Sitemap URL (optional)
Crawl-delay seconds (optional)
robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /cgi-bin