crtxcr/cgitsb
1
0
Vork 0
Code Kwesties Pull-aanvragen Publicaties Wiki Activiteit

robots.txt: disallow access to snapshots

Bladeren bron 
My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.

Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
This commit is contained in:
Jason A. Donenfeld 2013-05-28 14:17:00 +02:00
bovenliggende 830eb6f6ff
commit 23debef621
2 gewijzigde bestanden met toevoegingen van 4 en 0 verwijderingen
Statistieken weergeven Download Patch-bestand Download Diff-bestand Expand all files Collapse all files

1
Makefile
Bestand weergeven

@ -78,6 +78,7 @@ install: all
$(INSTALL) -m 0644 cgit.css $(DESTDIR)$(CGIT_DATA_PATH)/cgit.css
$(INSTALL) -m 0644 cgit.png $(DESTDIR)$(CGIT_DATA_PATH)/cgit.png
$(INSTALL) -m 0644 favicon.ico $(DESTDIR)$(CGIT_DATA_PATH)/favicon.ico
$(INSTALL) -m 0644 robots.txt $(DESTDIR)$(CGIT_DATA_PATH)/robots.txt
$(INSTALL) -m 0755 -d $(DESTDIR)$(filterdir)
$(COPYTREE) filters/* $(DESTDIR)$(filterdir)

3
robots.txt Normal file
Bestand weergeven

@ -0,0 +1,3 @@
User-agent: *
Disallow: /*/snapshot/*
Allow: /