Skip to content Hivex tools · Website Free · No account
Hivex / Tools / robots.txt tester / https://github.com/ robots.txt of github.com. See which search engines and AI crawlers may fetch a page, and the exact line of robots.txt that decides it.
How the rules are read. A crawler looks for groups with its own name in User-agent:, case does not matter, and combines them; with none, it uses the * group. Of the Allow: and Disallow: lines that match the path, the longest wins. A tie goes to Allow. * matches any run of characters and $ the end of the address. No file (404) means everything may be fetched; a server error (5xx) means crawlers treat everything as blocked. /robots.txt itself is always allowed. AI crawlers, by what they do. Training : GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), CCBot (Common Crawl), Applebot-Extended. Blocking them asks that your pages not be used to train models.Search : OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking them can keep your site out of those assistants' answers.On a user's request : ChatGPT-User and Claude-User fetch a page someone asked about; OpenAI notes robots.txt may not apply to those.Blocked is not hidden. robots.txt stops crawling, not indexing: a blocked page can still show in search results, without a description, if other pages link to it. To keep a page out of results, let it be crawled and add noindex. Never rely on robots.txt for anything private; the file itself is public.
Questions people ask. How does robots.txt decide whether a page is blocked? A crawler uses the group for its own name, or the * group if it has none. Of the Allow and Disallow lines that match the path, the longest one wins; when an Allow and a Disallow are equally long, Allow wins (RFC 9309).
How do I block AI crawlers but keep Google? Give each AI crawler its own group with Disallow: /, for example User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended, User-agent: CCBot. Googlebot keeps following its own rules, so search is unaffected.
Does Disallow keep a page out of Google? No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out, allow crawling and add a noindex meta tag or header.
What happens if robots.txt is missing or broken? A missing file (404) means crawlers may fetch everything. A server error (5xx) makes crawlers treat the whole site as blocked until it is fixed.
Is there a size limit? Crawlers must read at least 500 KiB, and Google stops there: rules further down are ignored.
Hivex index
Short names, still free to register.
Starting something new? Hivex keeps a live index of short, brandable .si names nobody has claimed yet, each checked with the registry.
Browse free names From code, or an AI assistant. More Website tools Redirect checker Every hop of a URL, and all four forms of the address. HTTP header checker All response headers, with the security headers checked. Security headers checker A grade for HSTS, CSP, framing and cookies, with fixes. SSL checker Trust, expiry, the chain and TLS versions of any site. Open Graph checker How a link looks when shared, with the tags checked. Is it down? Down for everyone or just you, and where it fails. Sitemap checker Limits, dates and hosts checked, a sample requested live. Who hosts this website Web host, CDN, DNS host, mail provider and registrar. Tech stack checker CMS, framework, hosting and trackers of any site. Domain checker Is a name free to register? Checked live on any extension. All free tools Partners
Domains worth owning.
Checked live · No account needed · Premium sales with AtomPay protection
© 2026 hivex.si. Availability changes all the time, so confirm at the registrar before you pay. Not affiliated with ARNES or register.si. Some domain and product links earn Hivex a commission, at no extra cost to you.
https://github.com/robots.txt
Checked 19:32:44 UTC · Fetched from Hivex's servers
Tested path /
Groups 8
Sitemaps 0
Size 16.3 KiB Note: No AI crawler listed below is blocked from /.Note: No Sitemap: line. Listing the sitemap here helps every search engine find it.Note: Crawl-delay is set; Google ignores it, Bing and some others follow it.Good: Google may crawl /.Who may fetch / RFC 9309 rules: longest match wins
Crawler / Decided by Search engines Googlebot Google Search Allowed no rule matches (group *) Bingbot Bing (and search built on it) Allowed no rule matches (group Bingbot) AI training Google-Extended Google: Gemini training and grounding Allowed
The file https://github.com/robots.txt
1 # If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots 2 # We also provide an extensive API: https://docs.github.com 3 User-agent: GPTBot 4 User-agent: OAI-SearchBot 5 User-agent: ClaudeBot 6 User-agent: anthropic-ai 7 User-agent: PerplexityBot 8 Crawl-delay: 1 9 Allow: /$ 10 Allow: /about no rule matches (group *)
GPTBot OpenAI: model training Allowed line 9: Allow: /$
ClaudeBot Anthropic: model training Allowed line 9: Allow: /$
CCBot Common Crawl: open dataset many models train on Allowed no rule matches (group *)
Applebot-Extended Apple: AI training Allowed no rule matches (group *)
AI search OAI-SearchBot OpenAI: ChatGPT search Allowed line 9: Allow: /$ Claude-SearchBot Anthropic: Claude's search Allowed no rule matches (group *) PerplexityBot Perplexity: search index Allowed line 9: Allow: /$
AI, fetched for a user ChatGPT-User OpenAI: pages a ChatGPT user asks for Allowed no rule matches (group *) Claude-User Anthropic: pages a Claude user asks for Allowed no rule matches (group *)
To test another page, put its path in the address you check, such as example.com/private/page.
11 Allow: /customer-stories
12 Allow: /enterprise
13 Allow: /features
14 Allow: /marketplace
15 Allow: /mcp
16 Allow: /newsroom
17 Allow: /pricing
18 Allow: /resources
19 Allow: /security
20 Allow: /solutions
21 Allow: /topics/download*
22 Allow: /topics/comments*
23 Allow: /marketplace/actions/download*
24 Disallow: /*/*/pulse
25 Disallow: /*/*/projects
26 Disallow: /*/*/forks
27 Disallow: /*/*/issues/new
28 Disallow: /*/*/milestones/new
29 Disallow: /*/*/issues/search
30 Disallow: /*/*/commits/
31 Disallow: /*/*/branches
32 Disallow: /*/*/contributors
33 Disallow: /*/*/tags
34 Disallow: /*/*/stargazers
35 Disallow: /*/*/watchers
36 Disallow: /*/*/network
37 Disallow: /*/*/graphs
38 Disallow: /*/*/compare
39 Disallow: /*/tree/
40 Disallow: /gist/
41 Disallow: /*/download
42 Disallow: /*/revisions
43 Disallow: /*/commits/*?author
44 Disallow: /*/commits/*?path
45 Disallow: /*/comments
46 Disallow: /*/archive/
47 Disallow: /*/blame/
48 Disallow: /*/raw/
49 Disallow: /*/cache/
50 Disallow: /.git/
51 Disallow: */.git/
52 Disallow: /*.git$
53 Disallow: /search/advanced
54 Disallow: /search$
55 Disallow: /*.atom$
56 Disallow: /ekansa/Open-Context-Data
57 Disallow: /ekansa/opencontext-*
58 Disallow: */tarball/
59 Disallow: */zipball/
60 Disallow: /*commits?author=*
61 Disallow: /*report-abuse?report=*
62 Disallow: /account-login
63 Disallow: /Explodingstuff/
64 Disallow: /copilot/
65 Disallow: /copilot/c/
66 Disallow: /marketplace/*/*?tab=readme$
67 Disallow: /marketplace/*?*?tab=readme$
68 Disallow: /marketplace/actions?tab=readme$
69 Disallow: /marketplace/free-trials?tab=readme$
70 Disallow: /marketplace/manage?tab=readme$
71 Disallow: /marketplace/models?tab=readme$
72 Disallow: /marketplace/new?tab=readme$
73 Disallow: /marketplace/publishers?tab=readme$
74 Disallow: /marketplace/recently-added?tab=readme$
75 Disallow: /marketplace/*/*?tab=transparency$
76 Disallow: /marketplace/*?*?tab=transparency$
77 Disallow: /marketplace/actions?tab=transparency$
78 Disallow: /marketplace/free-trials?tab=transparency$
79 Disallow: /marketplace/manage?tab=transparency$
80 Disallow: /marketplace/models?tab=transparency$
81 Disallow: /marketplace/new?tab=transparency$
82 Disallow: /marketplace/publishers?tab=transparency$
83 Disallow: /marketplace/recently-added?tab=transparency$
84 Disallow: /*?source=*
85 Disallow: /*&source=*
86 Disallow: /*?ref_cta=*
87 Disallow: /*&ref_cta=*
88 Disallow: /*?plan=*
89 Disallow: /*&plan=*
90 Disallow: /*?return_to=*
91 Disallow: /*&return_to=*
92 Disallow: /*?ref_loc=*
93 Disallow: /*&ref_loc=*
94 Disallow: /*?setup_organization=*
95 Disallow: /*&setup_organization=*
96 Disallow: /*?source_repo=*
97 Disallow: /*&source_repo=*
98 Disallow: /*?ref_page=*
99 Disallow: /*&ref_page=*
100 Disallow: /*?referrer=*
101 Disallow: /*&referrer=*
102 Disallow: /*?report=*
103 Disallow: /*&report=*
104 Disallow: /*?author=*
105 Disallow: /*&author=*
106 Disallow: /*?since=*
107 Disallow: /*&since=*
108 Disallow: /*?until=*
109 Disallow: /*&until=*
110 Disallow: /*?tab=*
111 Disallow: /*&tab=*
112 Disallow: /*?q=*
113 Disallow: /*&q=*
114 Disallow: /topics/download*?source=*
115 Disallow: /topics/download*&source=*
116 Disallow: /topics/download*?ref_cta=*
117 Disallow: /topics/download*&ref_cta=*
118 Disallow: /topics/download*?plan=*
119 Disallow: /topics/download*&plan=*
120 Disallow: /topics/download*?return_to=*
121 Disallow: /topics/download*&return_to=*
122 Disallow: /topics/download*?ref_loc=*
123 Disallow: /topics/download*&ref_loc=*
124 Disallow: /topics/download*?setup_organization=*
125 Disallow: /topics/download*&setup_organization=*
126 Disallow: /topics/download*?source_repo=*
127 Disallow: /topics/download*&source_repo=*
128 Disallow: /topics/download*?ref_page=*
129 Disallow: /topics/download*&ref_page=*
130 Disallow: /topics/download*?referrer=*
131 Disallow: /topics/download*&referrer=*
132 Disallow: /topics/download*?report=*
133 Disallow: /topics/download*&report=*
134 Disallow: /topics/download*?author=*
135 Disallow: /topics/download*&author=*
136 Disallow: /topics/download*?since=*
137 Disallow: /topics/download*&since=*
138 Disallow: /topics/download*?until=*
139 Disallow: /topics/download*&until=*
140 Disallow: /topics/download*?tab=*
141 Disallow: /topics/download*&tab=*
142 Disallow: /topics/download*?q=*
143 Disallow: /topics/download*&q=*
144 Disallow: /topics/comments*?source=*
145 Disallow: /topics/comments*&source=*
146 Disallow: /topics/comments*?ref_cta=*
147 Disallow: /topics/comments*&ref_cta=*
148 Disallow: /topics/comments*?plan=*
149 Disallow: /topics/comments*&plan=*
150 Disallow: /topics/comments*?return_to=*
151 Disallow: /topics/comments*&return_to=*
152 Disallow: /topics/comments*?ref_loc=*
153 Disallow: /topics/comments*&ref_loc=*
154 Disallow: /topics/comments*?setup_organization=*
155 Disallow: /topics/comments*&setup_organization=*
156 Disallow: /topics/comments*?source_repo=*
157 Disallow: /topics/comments*&source_repo=*
158 Disallow: /topics/comments*?ref_page=*
159 Disallow: /topics/comments*&ref_page=*
160 Disallow: /topics/comments*?referrer=*
161 Disallow: /topics/comments*&referrer=*
162 Disallow: /topics/comments*?report=*
163 Disallow: /topics/comments*&report=*
164 Disallow: /topics/comments*?author=*
165 Disallow: /topics/comments*&author=*
166 Disallow: /topics/comments*?since=*
167 Disallow: /topics/comments*&since=*
168 Disallow: /topics/comments*?until=*
169 Disallow: /topics/comments*&until=*
170 Disallow: /topics/comments*?tab=*
171 Disallow: /topics/comments*&tab=*
172 Disallow: /topics/comments*?q=*
173 Disallow: /topics/comments*&q=*
174 Disallow: /marketplace/actions/download*?source=*
175 Disallow: /marketplace/actions/download*&source=*
176 Disallow: /marketplace/actions/download*?ref_cta=*
177 Disallow: /marketplace/actions/download*&ref_cta=*
178 Disallow: /marketplace/actions/download*?plan=*
179 Disallow: /marketplace/actions/download*&plan=*
180 Disallow: /marketplace/actions/download*?return_to=*
181 Disallow: /marketplace/actions/download*&return_to=*
182 Disallow: /marketplace/actions/download*?ref_loc=*
183 Disallow: /marketplace/actions/download*&ref_loc=*
184 Disallow: /marketplace/actions/download*?setup_organization=*
185 Disallow: /marketplace/actions/download*&setup_organization=*
186 Disallow: /marketplace/actions/download*?source_repo=*
187 Disallow: /marketplace/actions/download*&source_repo=*
188 Disallow: /marketplace/actions/download*?ref_page=*
189 Disallow: /marketplace/actions/download*&ref_page=*
190 Disallow: /marketplace/actions/download*?referrer=*
191 Disallow: /marketplace/actions/download*&referrer=*
192 Disallow: /marketplace/actions/download*?report=*
193 Disallow: /marketplace/actions/download*&report=*
194 Disallow: /marketplace/actions/download*?author=*
195 Disallow: /marketplace/actions/download*&author=*
196 Disallow: /marketplace/actions/download*?since=*
197 Disallow: /marketplace/actions/download*&since=*
198 Disallow: /marketplace/actions/download*?until=*
199 Disallow: /marketplace/actions/download*&until=*
200 Disallow: /marketplace/actions/download*?tab=*
201 Disallow: /marketplace/actions/download*&tab=*
202 Disallow: /marketplace/actions/download*?q=*
203 Disallow: /marketplace/actions/download*&q=*
204
205 User-agent: Bytespider
206 Disallow: /
207
208 User-agent: AdsBot-Google
209 User-agent: AdsBot-Google-Mobile
210 Disallow: /*/*/pulse
211 Disallow: /*/*/projects
212 Disallow: /*/*/forks
213 Disallow: /*/*/issues/new
214 Disallow: /*/*/milestones/new
215 Disallow: /*/*/issues/search
216 Disallow: /*/*/commits/
217 Disallow: /*/*/branches
218 Disallow: /*/*/contributors
219 Disallow: /*/*/tags
220 Disallow: /*/*/stargazers
221 Disallow: /*/*/watchers
222 Disallow: /*/*/network
223 Disallow: /*/*/graphs
224 Disallow: /*/*/compare
225 Disallow: /*/tree/
226 Disallow: /gist/
227 Disallow: /*/download
228 Disallow: /*/revisions
229 Disallow: /*/commits/*?author
230 Disallow: /*/commits/*?path
231 Disallow: /*/comments
232 Disallow: /*/archive/
233 Disallow: /*/blame/
234 Disallow: /*/raw/
235 Disallow: /*/cache/
236 Disallow: /.git/
237 Disallow: */.git/
238 Disallow: /*.git$
239 Disallow: /search/advanced
240 Disallow: /search$
241 Disallow: /*.atom$
242 Disallow: /ekansa/Open-Context-Data
243 Disallow: /ekansa/opencontext-*
244 Disallow: */tarball/
245 Disallow: */zipball/
246 Disallow: /*commits?author=*
247 Disallow: /*report-abuse?report=*
248 Disallow: /account-login
249 Disallow: /Explodingstuff/
250 Disallow: /copilot/
251 Disallow: /copilot/c/
252 Disallow: /marketplace/*/*?tab=readme$
253 Disallow: /marketplace/*?*?tab=readme$
254 Disallow: /marketplace/actions?tab=readme$
255 Disallow: /marketplace/free-trials?tab=readme$
256 Disallow: /marketplace/manage?tab=readme$
257 Disallow: /marketplace/models?tab=readme$
258 Disallow: /marketplace/new?tab=readme$
259 Disallow: /marketplace/publishers?tab=readme$
260 Disallow: /marketplace/recently-added?tab=readme$
261 Disallow: /marketplace/*/*?tab=transparency$
262 Disallow: /marketplace/*?*?tab=transparency$
263 Disallow: /marketplace/actions?tab=transparency$
264 Disallow: /marketplace/free-trials?tab=transparency$
265 Disallow: /marketplace/manage?tab=transparency$
266 Disallow: /marketplace/models?tab=transparency$
267 Disallow: /marketplace/new?tab=transparency$
268 Disallow: /marketplace/publishers?tab=transparency$
269 Disallow: /marketplace/recently-added?tab=transparency$
270 Disallow: /*?source=*
271 Disallow: /*&source=*
272 Disallow: /*?ref_cta=*
273 Disallow: /*&ref_cta=*
274 Disallow: /*?plan=*
275 Disallow: /*&plan=*
276 Disallow: /*?return_to=*
277 Disallow: /*&return_to=*
278 Disallow: /*?ref_loc=*
279 Disallow: /*&ref_loc=*
280 Disallow: /*?setup_organization=*
281 Disallow: /*&setup_organization=*
282 Disallow: /*?source_repo=*
283 Disallow: /*&source_repo=*
284 Disallow: /*?ref_page=*
285 Disallow: /*&ref_page=*
286 Disallow: /*?referrer=*
287 Disallow: /*&referrer=*
288 Disallow: /*?report=*
289 Disallow: /*&report=*
290 Disallow: /*?author=*
291 Disallow: /*&author=*
292 Disallow: /*?since=*
293 Disallow: /*&since=*
294 Disallow: /*?until=*
295 Disallow: /*&until=*
296 Disallow: /*?tab=*
297 Disallow: /*&tab=*
298 Disallow: /*?q=*
299 Disallow: /*&q=*
300
301 User-agent: bingbot
302 Disallow: /ekansa/Open-Context-Data
303 Disallow: /ekansa/opencontext-*
304 Disallow: /account-login
305 Disallow: */tarball/
306 Disallow: */zipball/
307 Disallow: /Explodingstuff/
308 Disallow: /copilot/
309 Disallow: /copilot/c/
310
311 User-agent: adidxbot
312 Disallow: /ekansa/Open-Context-Data
313 Disallow: /ekansa/opencontext-*
314 Disallow: /account-login
315 Disallow: */tarball/
316 Disallow: */zipball/
317 Disallow: /Explodingstuff/
318 Disallow: /copilot/
319 Disallow: /copilot/c/
320
321 User-agent: BingPreview
322 Disallow: /ekansa/Open-Context-Data
323 Disallow: /ekansa/opencontext-*
324 Disallow: /account-login
325 Disallow: */tarball/
326 Disallow: */zipball/
327 Disallow: /Explodingstuff/
328 Disallow: /copilot/
329 Disallow: /copilot/c/
330
331 User-agent: baidu
332 crawl-delay: 1
333
334
335 User-agent: *
336 Allow: /*?tab=achievements&achievement=*
337 Allow: /marketplace/*?tab=readme$
338 Allow: /marketplace/*?tab=transparency$
339 Allow: /topics/download*
340 Allow: /topics/comments*
341 Allow: /marketplace/actions/download*
342 Disallow: /*/*/pulse
343 Disallow: /*/*/projects
344 Disallow: /*/*/forks
345 Disallow: /*/*/issues/new
346 Disallow: /*/*/milestones/new
347 Disallow: /*/*/issues/search
348 Disallow: /*/*/commits/
349 Disallow: /*/*/branches
350 Disallow: /*/*/contributors
351 Disallow: /*/*/tags
352 Disallow: /*/*/stargazers
353 Disallow: /*/*/watchers
354 Disallow: /*/*/network
355 Disallow: /*/*/graphs
356 Disallow: /*/*/compare
357 Disallow: /*/tree/
358 Disallow: /gist/
359 Disallow: /*/download
360 Disallow: /*/revisions
361 Disallow: /*/commits/*?author
362 Disallow: /*/commits/*?path
363 Disallow: /*/comments
364 Disallow: /*/archive/
365 Disallow: /*/blame/
366 Disallow: /*/raw/
367 Disallow: /*/cache/
368 Disallow: /.git/
369 Disallow: */.git/
370 Disallow: /*.git$
371 Disallow: /search/advanced
372 Disallow: /search$
373 Disallow: /*.atom$
374 Disallow: /ekansa/Open-Context-Data
375 Disallow: /ekansa/opencontext-*
376 Disallow: */tarball/
377 Disallow: */zipball/
378 Disallow: /*commits?author=*
379 Disallow: /*report-abuse?report=*
380 Disallow: /account-login
381 Disallow: /Explodingstuff/
382 Disallow: /copilot/
383 Disallow: /copilot/c/
384 Disallow: /marketplace/*/*?tab=readme$
385 Disallow: /marketplace/*?*?tab=readme$
386 Disallow: /marketplace/actions?tab=readme$
387 Disallow: /marketplace/free-trials?tab=readme$
388 Disallow: /marketplace/manage?tab=readme$
389 Disallow: /marketplace/models?tab=readme$
390 Disallow: /marketplace/new?tab=readme$
391 Disallow: /marketplace/publishers?tab=readme$
392 Disallow: /marketplace/recently-added?tab=readme$
393 Disallow: /marketplace/*/*?tab=transparency$
394 Disallow: /marketplace/*?*?tab=transparency$
395 Disallow: /marketplace/actions?tab=transparency$
396 Disallow: /marketplace/free-trials?tab=transparency$
397 Disallow: /marketplace/manage?tab=transparency$
398 Disallow: /marketplace/models?tab=transparency$
399 Disallow: /marketplace/new?tab=transparency$
400 Disallow: /marketplace/publishers?tab=transparency$