Skip to content
Theme
Request a shortlist
Hivex tools · WebsiteFree · No account

robots.txt of www.nytimes.com.

See which search engines and AI crawlers may fetch a page, and the exact line of robots.txt that decides it.

Trynytimes.comgithub.comhivex.si

Looking up…

Hivex index

Short names, still free to register.

Starting something new? Explore short, brandable .si names with confirmed or likely-free status, then check live before registering.

Browse free names

How the rules are read.

  • A crawler looks for groups with its own name in User-agent:, case does not matter, and combines them; with none, it uses the * group.
  • Of the Allow: and Disallow: lines that match the path, the longest wins. A tie goes to Allow. * matches any run of characters and $ the end of the address.
  • No file (404) means everything may be fetched; a server error (5xx) means crawlers treat everything as blocked. /robots.txt itself is always allowed.

AI crawlers, by what they do.

  • Training: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), CCBot (Common Crawl), Applebot-Extended. Blocking them asks that your pages not be used to train models.
  • Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking them can keep your site out of those assistants' answers.
  • On a user's request: ChatGPT-User and Claude-User fetch a page someone asked about; OpenAI notes robots.txt may not apply to those.

Blocked is not hidden.

robots.txt stops crawling, not indexing: a blocked page can still show in search results, without a description, if other pages link to it. To keep a page out of results, let it be crawled and add noindex. Never rely on robots.txt for anything private; the file itself is public.

Questions people ask.

How does robots.txt decide whether a page is blocked?

A crawler uses the group for its own name, or the * group if it has none. Of the Allow and Disallow lines that match the path, the longest one wins; when an Allow and a Disallow are equally long, Allow wins (RFC 9309).

How do I block AI crawlers but keep Google?

Give each AI crawler its own group with Disallow: /, for example User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended, User-agent: CCBot. Googlebot keeps following its own rules, so search is unaffected.

Does Disallow keep a page out of Google?

No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out, allow crawling and add a noindex meta tag or header.

What happens if robots.txt is missing or broken?

A missing file (404) means crawlers may fetch everything. A server error (5xx) makes crawlers treat the whole site as blocked until it is fixed.

Is there a size limit?

Crawlers must read at least 500 KiB, and Google stops there: rules further down are ignored.

From code, or an AI assistant.

Hivex's free JSON API and MCP server check domains, DNS and registration records from your own code or from AI assistants that speak MCP. No key needed.

The API guideThe MCP serverOpenAPI schema

More Website tools

All free tools

Sources

  1. RFC 9309: Robots Exclusion Protocol
  2. OpenAI: overview of OpenAI crawlers
  3. Anthropic: how site owners can block Anthropic's crawlers
  4. Google Search Central: how Google interprets robots.txt

By Hivex. Updated 10 October 2026. Addresses are fetched live from Hivex's servers; page contents are never read or stored.