Skip to content Hivex tools · Website Free · No account
Hivex / Tools / robots.txt tester / http://github.com/ robots.txt of github.com. See which search engines and AI crawlers may fetch a page, and the exact line of robots.txt that decides it.
Hivex index
Short names, still free to register.
Starting something new? Explore short, brandable .si names with confirmed or likely-free status, then check live before registering.
Browse free names How the rules are read. A crawler looks for groups with its own name in User-agent:, case does not matter, and combines them; with none, it uses the * group. Of the Allow: and Disallow: lines that match the path, the longest wins. A tie goes to Allow. * matches any run of characters and $ the end of the address. No file (404) means everything may be fetched; a server error (5xx) means crawlers treat everything as blocked. /robots.txt itself is always allowed. AI crawlers, by what they do. Training : GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini), CCBot (Common Crawl), Applebot-Extended. Blocking them asks that your pages not be used to train models.Search : OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking them can keep your site out of those assistants' answers.On a user's request : ChatGPT-User and Claude-User fetch a page someone asked about; OpenAI notes robots.txt may not apply to those.Blocked is not hidden. robots.txt stops crawling, not indexing: a blocked page can still show in search results, without a description, if other pages link to it. To keep a page out of results, let it be crawled and add noindex. Never rely on robots.txt for anything private; the file itself is public.
Questions people ask. How does robots.txt decide whether a page is blocked? A crawler uses the group for its own name, or the * group if it has none. Of the Allow and Disallow lines that match the path, the longest one wins; when an Allow and a Disallow are equally long, Allow wins (RFC 9309).
How do I block AI crawlers but keep Google? Give each AI crawler its own group with Disallow: /, for example User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended, User-agent: CCBot. Googlebot keeps following its own rules, so search is unaffected.
Does Disallow keep a page out of Google? No. It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out, allow crawling and add a noindex meta tag or header.
What happens if robots.txt is missing or broken? A missing file (404) means crawlers may fetch everything. A server error (5xx) makes crawlers treat the whole site as blocked until it is fixed.
Is there a size limit? Crawlers must read at least 500 KiB, and Google stops there: rules further down are ignored.
From code, or an AI assistant. More Website tools Redirect checker Every hop of a URL, and all four forms of the address. HTTP header checker All response headers, with the security headers checked. Security headers checker A grade for HSTS, CSP, framing and cookies, with fixes. SSL checker Trust, expiry, the chain and TLS versions of any site. Open Graph checker How a link looks when shared, with the tags checked. Is it down? Down for everyone or just you, and where it fails. Sitemap checker Limits, dates and hosts checked, a sample requested live. Who hosts this website Web host, CDN, DNS host, mail provider and registrar. Tech stack checker CMS, framework, hosting and trackers of any site. Domain checker Is a name free to register? Checked live on any extension. All free tools Partners
Domains worth owning.
Checked live · No account needed · Premium sales with AtomPay protection
© 2026 hivex.si. Availability changes all the time, so confirm at the registrar before you pay. Not affiliated with ARNES or register.si. Some domain and product links earn Hivex a commission, at no extra cost to you.
http://github.com/robots.txt
Checked 09:23:49 UTC · Fetched from Hivex's servers
Tested path /
Groups 6
Sitemaps 1
Size 14.2 KiB Note: No AI crawler listed below is blocked from /.Note: Crawl-delay is set; Google ignores it, Bing and some others follow it.Good: Google may crawl /.Who may fetch / RFC 9309 rules: longest match wins
Crawler / Decided by Search engines Googlebot Google Search Allowed no rule matches (group *) Bingbot Bing (and search built on it) Allowed no rule matches (group Bingbot) AI training Google-Extended Google: Gemini training and grounding Allowed no rule matches (group *) GPTBot OpenAI: model training
The file Redirected to https://github.com/robots.txt
1 # If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots 2 # We also provide an extensive API: https://docs.github.com 3 Sitemap: https://github.com/sitemap_marketing.xml 4 5 User-agent: GPTBot 6 User-agent: OAI-SearchBot 7 User-agent: ClaudeBot 8 User-agent: anthropic-ai 9 User-agent: PerplexityBot 10 Crawl-delay: 1 no rule matches (group GPTBot)
ClaudeBot Anthropic: model training Allowed no rule matches (group ClaudeBot)
CCBot Common Crawl: open dataset many models train on Allowed no rule matches (group *)
Applebot-Extended Apple: AI training Allowed no rule matches (group *)
AI search OAI-SearchBot OpenAI: ChatGPT search Allowed no rule matches (group OAI-SearchBot) Claude-SearchBot Anthropic: Claude's search Allowed no rule matches (group *) PerplexityBot Perplexity: search index Allowed no rule matches (group PerplexityBot)
AI, fetched for a user ChatGPT-User OpenAI: pages a ChatGPT user asks for Allowed no rule matches (group *) Claude-User Anthropic: pages a Claude user asks for Allowed no rule matches (group *)
To test another page, put its path in the address you check, such as example.com/private/page.
11 Allow: /*?tab=achievements&achievement=*
12 Allow: /marketplace/*?tab=readme$
13 Allow: /marketplace/*?tab=transparency$
14 Disallow: /account-login
15 Disallow: /copilot/
16 Disallow: /copilot/c/
17 Disallow: /ekansa/Open-Context-Data
18 Disallow: /ekansa/opencontext-*
19 Disallow: /Explodingstuff/
20 Disallow: /gist/
21 Disallow: /search/advanced
22 Disallow: /search$
23 Disallow: /*/*/pulse_committer_data.json$
24 Disallow: /*/*/pulse_committer_data.json?*
25 Disallow: /*/archive/
26 Disallow: /*/blame/
27 Disallow: /*/cache/
28 Disallow: /*/raw/
29 Disallow: /*/tree/
30 Disallow: */tarball/
31 Disallow: */zipball/
32 Disallow: /.git/
33 Disallow: */.git/
34 Disallow: /*.git$
35 Disallow: /*.atom$
36 Disallow: /*report-abuse?report=*
37 Disallow: /*/*/commits/
38 Disallow: /marketplace/*/*?tab=readme$
39 Disallow: /marketplace/*/*?tab=transparency$
40 Disallow: /marketplace/*?*?tab=readme$
41 Disallow: /marketplace/*?*?tab=transparency$
42 Disallow: /marketplace/actions?tab=readme$
43 Disallow: /marketplace/actions?tab=transparency$
44 Disallow: /marketplace/free-trials?tab=readme$
45 Disallow: /marketplace/free-trials?tab=transparency$
46 Disallow: /marketplace/manage?tab=readme$
47 Disallow: /marketplace/manage?tab=transparency$
48 Disallow: /marketplace/models?tab=readme$
49 Disallow: /marketplace/models?tab=transparency$
50 Disallow: /marketplace/new?tab=readme$
51 Disallow: /marketplace/new?tab=transparency$
52 Disallow: /marketplace/publishers?tab=readme$
53 Disallow: /marketplace/publishers?tab=transparency$
54 Disallow: /marketplace/recently-added?tab=readme$
55 Disallow: /marketplace/recently-added?tab=transparency$
56 Disallow: /biztools$
57 Disallow: /biztools?*
58 Disallow: /biztools/*
59 Disallow: /businesses$
60 Disallow: /businesses?*
61 Disallow: /businesses/*
62 Disallow: /enterprises$
63 Disallow: /enterprises?*
64 Disallow: /enterprises/*
65 Disallow: /organizations$
66 Disallow: /organizations?*
67 Disallow: /organizations/*
68 Disallow: /settings$
69 Disallow: /settings?*
70 Disallow: /settings/*
71 Disallow: /stafftools$
72 Disallow: /stafftools?*
73 Disallow: /stafftools/*
74 Disallow: /users/*/billing$
75 Disallow: /users/*/billing?*
76 Disallow: /users/*/billing/*
77 Disallow: /writing$
78 Disallow: /writing?*
79 Disallow: /writing/*
80 Disallow: /*/*/branches$
81 Disallow: /*/*/branches?*
82 Disallow: /*/*/branches/*
83 Disallow: /*/*/comments$
84 Disallow: /*/*/comments?*
85 Disallow: /*/*/comments/*
86 Disallow: /*/*/compare$
87 Disallow: /*/*/compare?*
88 Disallow: /*/*/compare/*
89 Disallow: /*/*/contributors$
90 Disallow: /*/*/contributors?*
91 Disallow: /*/*/contributors/*
92 Disallow: /*/*/contributors_list$
93 Disallow: /*/*/contributors_list?*
94 Disallow: /*/*/contributors_list/*
95 Disallow: /*/*/contributors-list$
96 Disallow: /*/*/contributors-list?*
97 Disallow: /*/*/contributors-list/*
98 Disallow: /*/*/download$
99 Disallow: /*/*/download?*
100 Disallow: /*/*/download/*
101 Disallow: /*/*/forks$
102 Disallow: /*/*/forks?*
103 Disallow: /*/*/forks/*
104 Disallow: /*/*/graphs$
105 Disallow: /*/*/graphs?*
106 Disallow: /*/*/graphs/*
107 Disallow: /*/*/issues/new$
108 Disallow: /*/*/issues/new?*
109 Disallow: /*/*/issues/new/*
110 Disallow: /*/*/issues/search$
111 Disallow: /*/*/issues/search?*
112 Disallow: /*/*/issues/search/*
113 Disallow: /*/*/milestones/new$
114 Disallow: /*/*/milestones/new?*
115 Disallow: /*/*/milestones/new/*
116 Disallow: /*/*/network$
117 Disallow: /*/*/network?*
118 Disallow: /*/*/network/*
119 Disallow: /*/*/projects$
120 Disallow: /*/*/projects?*
121 Disallow: /*/*/projects/*
122 Disallow: /*/*/pulse$
123 Disallow: /*/*/pulse?*
124 Disallow: /*/*/pulse/*
125 Disallow: /*/*/pulse-new$
126 Disallow: /*/*/pulse-new?*
127 Disallow: /*/*/pulse-new/*
128 Disallow: /*/*/pulse_committer_data$
129 Disallow: /*/*/pulse_committer_data?*
130 Disallow: /*/*/pulse_committer_data/*
131 Disallow: /*/*/pulse_diffstat_summary$
132 Disallow: /*/*/pulse_diffstat_summary?*
133 Disallow: /*/*/pulse_diffstat_summary/*
134 Disallow: /*/*/revisions$
135 Disallow: /*/*/revisions?*
136 Disallow: /*/*/revisions/*
137 Disallow: /*/*/stargazers$
138 Disallow: /*/*/stargazers?*
139 Disallow: /*/*/stargazers/*
140 Disallow: /*/*/tags$
141 Disallow: /*/*/tags?*
142 Disallow: /*/*/tags/*
143 Disallow: /*/*/watchers$
144 Disallow: /*/*/watchers?*
145 Disallow: /*/*/watchers/*
146 Disallow: /*?source=*
147 Disallow: /*&source=*
148 Disallow: /*?ref_cta=*
149 Disallow: /*&ref_cta=*
150 Disallow: /*?plan=*
151 Disallow: /*&plan=*
152 Disallow: /*?return_to=*
153 Disallow: /*&return_to=*
154 Disallow: /*?ref_loc=*
155 Disallow: /*&ref_loc=*
156 Disallow: /*?setup_organization=*
157 Disallow: /*&setup_organization=*
158 Disallow: /*?source_repo=*
159 Disallow: /*&source_repo=*
160 Disallow: /*?ref_page=*
161 Disallow: /*&ref_page=*
162 Disallow: /*?referrer=*
163 Disallow: /*&referrer=*
164 Disallow: /*?report=*
165 Disallow: /*&report=*
166 Disallow: /*?author=*
167 Disallow: /*&author=*
168 Disallow: /*?since=*
169 Disallow: /*&since=*
170 Disallow: /*?until=*
171 Disallow: /*&until=*
172 Disallow: /*?tab=*
173 Disallow: /*&tab=*
174 Disallow: /*?q=*
175 Disallow: /*&q=*
176
177 User-agent: Bytespider
178 Disallow: /
179
180 User-agent: AdsBot-Google
181 User-agent: AdsBot-Google-Mobile
182 Disallow: /account-login
183 Disallow: /copilot/
184 Disallow: /copilot/c/
185 Disallow: /ekansa/Open-Context-Data
186 Disallow: /ekansa/opencontext-*
187 Disallow: /Explodingstuff/
188 Disallow: /gist/
189 Disallow: /search/advanced
190 Disallow: /search$
191 Disallow: /*/*/pulse_committer_data.json$
192 Disallow: /*/*/pulse_committer_data.json?*
193 Disallow: /*/archive/
194 Disallow: /*/blame/
195 Disallow: /*/cache/
196 Disallow: /*/raw/
197 Disallow: /*/tree/
198 Disallow: */tarball/
199 Disallow: */zipball/
200 Disallow: /.git/
201 Disallow: */.git/
202 Disallow: /*.git$
203 Disallow: /*.atom$
204 Disallow: /*report-abuse?report=*
205 Disallow: /*/*/commits/
206 Disallow: /marketplace/*/*?tab=readme$
207 Disallow: /marketplace/*/*?tab=transparency$
208 Disallow: /marketplace/*?*?tab=readme$
209 Disallow: /marketplace/*?*?tab=transparency$
210 Disallow: /marketplace/actions?tab=readme$
211 Disallow: /marketplace/actions?tab=transparency$
212 Disallow: /marketplace/free-trials?tab=readme$
213 Disallow: /marketplace/free-trials?tab=transparency$
214 Disallow: /marketplace/manage?tab=readme$
215 Disallow: /marketplace/manage?tab=transparency$
216 Disallow: /marketplace/models?tab=readme$
217 Disallow: /marketplace/models?tab=transparency$
218 Disallow: /marketplace/new?tab=readme$
219 Disallow: /marketplace/new?tab=transparency$
220 Disallow: /marketplace/publishers?tab=readme$
221 Disallow: /marketplace/publishers?tab=transparency$
222 Disallow: /marketplace/recently-added?tab=readme$
223 Disallow: /marketplace/recently-added?tab=transparency$
224 Disallow: /biztools$
225 Disallow: /biztools?*
226 Disallow: /biztools/*
227 Disallow: /businesses$
228 Disallow: /businesses?*
229 Disallow: /businesses/*
230 Disallow: /enterprises$
231 Disallow: /enterprises?*
232 Disallow: /enterprises/*
233 Disallow: /organizations$
234 Disallow: /organizations?*
235 Disallow: /organizations/*
236 Disallow: /settings$
237 Disallow: /settings?*
238 Disallow: /settings/*
239 Disallow: /stafftools$
240 Disallow: /stafftools?*
241 Disallow: /stafftools/*
242 Disallow: /users/*/billing$
243 Disallow: /users/*/billing?*
244 Disallow: /users/*/billing/*
245 Disallow: /writing$
246 Disallow: /writing?*
247 Disallow: /writing/*
248 Disallow: /*/*/branches$
249 Disallow: /*/*/branches?*
250 Disallow: /*/*/branches/*
251 Disallow: /*/*/comments$
252 Disallow: /*/*/comments?*
253 Disallow: /*/*/comments/*
254 Disallow: /*/*/compare$
255 Disallow: /*/*/compare?*
256 Disallow: /*/*/compare/*
257 Disallow: /*/*/contributors$
258 Disallow: /*/*/contributors?*
259 Disallow: /*/*/contributors/*
260 Disallow: /*/*/contributors_list$
261 Disallow: /*/*/contributors_list?*
262 Disallow: /*/*/contributors_list/*
263 Disallow: /*/*/contributors-list$
264 Disallow: /*/*/contributors-list?*
265 Disallow: /*/*/contributors-list/*
266 Disallow: /*/*/download$
267 Disallow: /*/*/download?*
268 Disallow: /*/*/download/*
269 Disallow: /*/*/forks$
270 Disallow: /*/*/forks?*
271 Disallow: /*/*/forks/*
272 Disallow: /*/*/graphs$
273 Disallow: /*/*/graphs?*
274 Disallow: /*/*/graphs/*
275 Disallow: /*/*/issues/new$
276 Disallow: /*/*/issues/new?*
277 Disallow: /*/*/issues/new/*
278 Disallow: /*/*/issues/search$
279 Disallow: /*/*/issues/search?*
280 Disallow: /*/*/issues/search/*
281 Disallow: /*/*/milestones/new$
282 Disallow: /*/*/milestones/new?*
283 Disallow: /*/*/milestones/new/*
284 Disallow: /*/*/network$
285 Disallow: /*/*/network?*
286 Disallow: /*/*/network/*
287 Disallow: /*/*/projects$
288 Disallow: /*/*/projects?*
289 Disallow: /*/*/projects/*
290 Disallow: /*/*/pulse$
291 Disallow: /*/*/pulse?*
292 Disallow: /*/*/pulse/*
293 Disallow: /*/*/pulse-new$
294 Disallow: /*/*/pulse-new?*
295 Disallow: /*/*/pulse-new/*
296 Disallow: /*/*/pulse_committer_data$
297 Disallow: /*/*/pulse_committer_data?*
298 Disallow: /*/*/pulse_committer_data/*
299 Disallow: /*/*/pulse_diffstat_summary$
300 Disallow: /*/*/pulse_diffstat_summary?*
301 Disallow: /*/*/pulse_diffstat_summary/*
302 Disallow: /*/*/revisions$
303 Disallow: /*/*/revisions?*
304 Disallow: /*/*/revisions/*
305 Disallow: /*/*/stargazers$
306 Disallow: /*/*/stargazers?*
307 Disallow: /*/*/stargazers/*
308 Disallow: /*/*/tags$
309 Disallow: /*/*/tags?*
310 Disallow: /*/*/tags/*
311 Disallow: /*/*/watchers$
312 Disallow: /*/*/watchers?*
313 Disallow: /*/*/watchers/*
314 Disallow: /*?source=*
315 Disallow: /*&source=*
316 Disallow: /*?ref_cta=*
317 Disallow: /*&ref_cta=*
318 Disallow: /*?plan=*
319 Disallow: /*&plan=*
320 Disallow: /*?return_to=*
321 Disallow: /*&return_to=*
322 Disallow: /*?ref_loc=*
323 Disallow: /*&ref_loc=*
324 Disallow: /*?setup_organization=*
325 Disallow: /*&setup_organization=*
326 Disallow: /*?source_repo=*
327 Disallow: /*&source_repo=*
328 Disallow: /*?ref_page=*
329 Disallow: /*&ref_page=*
330 Disallow: /*?referrer=*
331 Disallow: /*&referrer=*
332 Disallow: /*?report=*
333 Disallow: /*&report=*
334 Disallow: /*?author=*
335 Disallow: /*&author=*
336 Disallow: /*?since=*
337 Disallow: /*&since=*
338 Disallow: /*?until=*
339 Disallow: /*&until=*
340 Disallow: /*?tab=*
341 Disallow: /*&tab=*
342 Disallow: /*?q=*
343 Disallow: /*&q=*
344
345 User-agent: bingbot
346 User-agent: adidxbot
347 User-agent: BingPreview
348 Disallow: /ekansa/Open-Context-Data
349 Disallow: /ekansa/opencontext-*
350 Disallow: /account-login
351 Disallow: */tarball/
352 Disallow: */zipball/
353 Disallow: /Explodingstuff/
354 Disallow: /copilot/
355 Disallow: /copilot/c/
356
357 User-agent: baidu
358 crawl-delay: 1
359
360 User-agent: *
361 Allow: /*?tab=achievements&achievement=*
362 Allow: /marketplace/*?tab=readme$
363 Allow: /marketplace/*?tab=transparency$
364 Disallow: /account-login
365 Disallow: /copilot/
366 Disallow: /copilot/c/
367 Disallow: /ekansa/Open-Context-Data
368 Disallow: /ekansa/opencontext-*
369 Disallow: /Explodingstuff/
370 Disallow: /gist/
371 Disallow: /search/advanced
372 Disallow: /search$
373 Disallow: /*/*/pulse_committer_data.json$
374 Disallow: /*/*/pulse_committer_data.json?*
375 Disallow: /*/archive/
376 Disallow: /*/blame/
377 Disallow: /*/cache/
378 Disallow: /*/raw/
379 Disallow: /*/tree/
380 Disallow: */tarball/
381 Disallow: */zipball/
382 Disallow: /.git/
383 Disallow: */.git/
384 Disallow: /*.git$
385 Disallow: /*.atom$
386 Disallow: /*report-abuse?report=*
387 Disallow: /*/*/commits/
388 Disallow: /marketplace/*/*?tab=readme$
389 Disallow: /marketplace/*/*?tab=transparency$
390 Disallow: /marketplace/*?*?tab=readme$
391 Disallow: /marketplace/*?*?tab=transparency$
392 Disallow: /marketplace/actions?tab=readme$
393 Disallow: /marketplace/actions?tab=transparency$
394 Disallow: /marketplace/free-trials?tab=readme$
395 Disallow: /marketplace/free-trials?tab=transparency$
396 Disallow: /marketplace/manage?tab=readme$
397 Disallow: /marketplace/manage?tab=transparency$
398 Disallow: /marketplace/models?tab=readme$
399 Disallow: /marketplace/models?tab=transparency$
400 Disallow: /marketplace/new?tab=readme$