Free Tools

Robots.txt Generator

Create a robots.txt file to control how search engines crawl and index your website.

Basic Configuration
Common Rules
Custom Rules
Generated Robots.txt
# Robots.txt generated by Free Tools
# https://freetoools.com/tools/robots-txt-generator

User-agent: *

Instructions:

  1. Copy the generated robots.txt content above
  2. Create a new file named "robots.txt" in your website's root directory
  3. Paste the content into the file
  4. Upload the file to your web server
  5. Test by visiting: yoursite.com/robots.txt