Search Engine Standard · Block AI Scrapers

Robots.txt Generator

Create clean, standards-compliant directives for Googlebot, Bingbot, and AI web scrapers. Customize allowed paths and download your ready-to-use file.

Robots Configuration:

Crawler Directives

Default Access for All Robots (User-agent: *)

Block AI Training Bots & Scrapers

Adds Disallow directives for OpenAI (GPTBot), Anthropic (Claude-Web), Common Crawl (CCBot), Google Extended, and ByteDance bots.

Disallow Specific Paths or Files (4)

/admin//api//private//*?*sort=

Crawl Delay (Seconds)

XML Sitemap Location

Generated robots.txt

Public Root File

# robots.txt generated by ToolHub (iformat.io)
User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /*?*sort=

# Block Artificial Intelligence & Web Scraping Bots
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /

# XML Sitemap Index
Sitemap: https://iformat.io/sitemap.xml

Where to place this file?

Save this file in the root directory of your website domain so it is accessible at:https://yourdomain.com/robots.txt

Robots.txt Best Practices

File Placement

The file must strictly live at the root of your domain (e.g. `https://yourdomain.com/robots.txt`). It cannot be placed in a subdirectory.

Blocking AI Training Bots

Protect your intellectual property by blocking automated training spiders like GPTBot, CCBot, Anthropic, and Google-Extended.

Sitemap Reference

Always include the absolute URL of your XML sitemap at the bottom of the robots.txt file so crawlers can discover your URL inventory.

Frequently Asked Questions

Does robots.txt hide private pages from the public?

No! Robots.txt is a public advisory file. It tells compliant search crawlers what NOT to index. Anyone can read your robots.txt file. For private data, use password protection, authentication, or server-level access control.

What is the difference between Disallow: / and Disallow: ?

`Disallow: /` blocks bots from accessing every single page on the site. An empty `Disallow:` directive instructs bots that they have permission to crawl all pages on the site.