randrss/fetchers/chrome

#!/bin/sh
#Tries more or less to look like Chrome
if [ $# -ne 2 ] ; then
echo "usage: $0 url output" 1>&2
exit 1
fi
if find "$RANDRSS_ROOT/fetchers/chrome_version" -mtime +1 | grep -q chrome_version ; then
curl "https://en.wikipedia.org/w/index.php?title=Template:Latest_stable_software_release/Google_Chrome&action=edit" | grep version1 | awk '{print $4}' | sed -e 's/[^0-9.]//g' > /"$RANDRSS_ROOT/fetchers/chrome_version"
fi
VERSION=$(cat "$RANDRSS_ROOT/fetchers/chrome_version")
useragent=$(shuf -n 1 $RANDRSS_ROOT/fetchers/chrome_agents  | sed -e "s/VERSION/$VERSION/g")

curl "$1" -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H "$useragent" -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-User: ?1' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3' -H 'Sec-Fetch-Site: same-origin' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' --compressed > $2
Added fetchers concept: seperate scripts to fetch the feeds Fetchers claim to be a certain client. They try to send the same headers as the original client. That's better than a simple curl request with a fake user agent, because curl doesn't send the other headers like the original client and therefore its traffic stands out. 2017-08-11 12:57:30 +02:00			`#!/bin/sh`
			`#Tries more or less to look like Chrome`
			`if [ $# -ne 2 ] ; then`
			`echo "usage: $0 url output" 1>&2`
			`exit 1`
			`fi`
fetchers: update browser version automatically. updated headers to look more like recent versions 2019-08-27 13:12:40 +02:00			`if find "$RANDRSS_ROOT/fetchers/chrome_version" -mtime +1 \| grep -q chrome_version ; then`
simplify and fix fetching chrome version after some wikipedia format change 2019-11-10 09:50:31 +01:00			`curl "https://en.wikipedia.org/w/index.php?title=Template:Latest_stable_software_release/Google_Chrome&action=edit" \| grep version1 \| awk '{print $4}' \| sed -e 's/[^0-9.]//g' > /"$RANDRSS_ROOT/fetchers/chrome_version"`
fetchers: update browser version automatically. updated headers to look more like recent versions 2019-08-27 13:12:40 +02:00			`fi`
			`VERSION=$(cat "$RANDRSS_ROOT/fetchers/chrome_version")`
			`useragent=$(shuf -n 1 $RANDRSS_ROOT/fetchers/chrome_agents \| sed -e "s/VERSION/$VERSION/g")`
Added fetchers concept: seperate scripts to fetch the feeds Fetchers claim to be a certain client. They try to send the same headers as the original client. That's better than a simple curl request with a fake user agent, because curl doesn't send the other headers like the original client and therefore its traffic stands out. 2017-08-11 12:57:30 +02:00
fetchers: update browser version automatically. updated headers to look more like recent versions 2019-08-27 13:12:40 +02:00			`curl "$1" -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H "$useragent" -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-User: ?1' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3' -H 'Sec-Fetch-Site: same-origin' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' --compressed > $2`