Added fetchers concept: seperate scripts to fetch the feeds

Fetchers claim to be a certain client. They try to send the same
headers as the original client. That's better than a simple curl request
with a fake user agent, because curl doesn't send the other headers like
the original client and therefore its traffic stands out.
这个提交包含在:
2017-08-11 12:57:30 +02:00
父节点 8a80aa0d6d
当前提交 3a723b9440
共有 6 个文件被更改,包括 40 次插入9 次删除

1
fetchers/chrome_agents 普通文件
查看文件

@ -0,0 +1 @@
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36