Added fetchers concept: seperate scripts to fetch the feeds

Fetchers claim to be a certain client. They try to send the same
headers as the original client. That's better than a simple curl request
with a fake user agent, because curl doesn't send the other headers like
the original client and therefore its traffic stands out.
这个提交包含在:
2017-08-11 12:57:30 +02:00
父节点 8a80aa0d6d
当前提交 3a723b9440
共有 6 个文件被更改,包括 40 次插入9 次删除

11
fetchers/firefox 可执行文件
查看文件

@@ -0,0 +1,11 @@
#!/bin/sh
set -x
#Tries more or less to look like Firefox
if [ $# -ne 2 ] ; then
echo "usage: $0 url output" 1>&2
exit 1
fi
#better randomize
useragent=$(shuf -n 1 $RANDRSS_ROOT/fetchers/firefox_agents)
curl "$1" -H "User-Agent: $useragent" -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' --compressed -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' > $2