cgitsb/robots.txt atzarā 696a33b66f8638d5c3e0464d66760c6d60ff24a5 - cgitsb - quitesimple

crtxcr/cgitsb

Jason A. Donenfeld 23debef621 robots.txt: disallow access to snapshots

My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.

Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>

2013-08-12 13:14:10 -06:00

4 rindas

47 B

Plaintext

Neapstrādāts Vainot Vēsture

	`User-agent: *`
	`Disallow: //snapshot/`
	`Allow: /`