1
0

Added fetchers concept: seperate scripts to fetch the feeds

Fetchers claim to be a certain client. They try to send the same
headers as the original client. That's better than a simple curl request
with a fake user agent, because curl doesn't send the other headers like
the original client and therefore its traffic stands out.
Dieser Commit ist enthalten in:
2017-08-11 12:57:30 +02:00
Ursprung 8a80aa0d6d
Commit 3a723b9440
6 geänderte Dateien mit 40 neuen und 9 gelöschten Zeilen

2
fetchers/firefox_agents Normale Datei
Datei anzeigen

@@ -0,0 +1,2 @@
Mozilla/5.0 (X11; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0
Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0