Bash Multi-Get using wget
If you were ever looking at using wget with a proxy – then this is probably the script for you. I haven&squo;t spent much time testing this and it requires a helper program written in perl to get the timer resolution to handle properly. The idea behind this script is to be able to traverse a list of proxies so that you can test accessing your site from a bunch of “different places at once”.
This could also help to simulate a load if put in a loop, this will however only be run from one system so it does not necessarily guarantee a solid load. It also does not guarantee that there won&squo;t be issues caused by using said script.
Nevertheless, have fun playing with this script.
#!/usr/bin/bash FILE= DELAY= TIMEOUT= function Usage { echo "Please use [list] [timeout=] [delay=]" ; exit 0 ; } function Worker { if [ -z "$5" ]; then sleep 5$ fi if [ ! -z "$4" ]; then export http_proxy=`echo "http://$2:$3"` wget $1 --timeout=$4 -Y on -q on --cache=off --output-file=$1.tmp; if [ "$?" == 0 ]; then echo "Success:$1 http://$2:$3">> $1.tmp else echo "Fail:$1 http://$2:$3">> $1.tmp fi else export http_proxy=`echo "http://$2:$3"` wget $1 -Y on -q on --cache=off --output-file $1.tmp; if [ "$?" == 0 ]; then echo "Success:$1 http://$2:$3">> $1.tmp else echo "Fail:$1 http://$2:$3">> $1.tmp fi fi } function GetFile { wget #put proxy list here -q on --cache=off --output-document=list.tmp FILE=list.tmp } if [ -z "$1" ]; then Usage; fi if [ -z "$2" ]; then GetFile; else FILE=$2 fi if [ ! -f "$FILE" ]; then wget # put proxy list here -q on --cache=off --output-document=list.tmp FILE=list.tmp echo "Warning: using web proxy list because list file invalid" ; fi TIMEOUT= DELAY= rm go.log if [[ $3 =~ ^timeout=[0-9]*$ ]]; then TIMEOUT=`echo $3|cut -d"=" -f2` echo "Setting Timeout to: "$TIMEOUT >> go.log ; elif [[ $4 =~ ^timeout=[0-9]*$ ]]; then TIMEOUT=`echo $4|cut -d"=" -f2` echo "Setting Timeout to: "$TIMEOUT >> go.log ; fi if [[ $3 =~ ^delay=[0-9]*$ ]]; then DELAY=`echo $3|cut -d"=" -f2` echo "Setting Delay to: "$DELAY >> go.log; elif [[ $4 =~ ^delay=[0-9]*$ ]]; then DELAY=`echo $4|cut -d"=" -f2` echo "Setting Delay to: "$DELAY >> go.log; fi while read F do if [[ $F =~ ^([0-9]{1,3}.){3}[0-9]{1,3}:[0-9]{1,4}$ ]]; then HOST=`echo $F|cut -d":" -f1` PORT=`echo $F|cut -d":" -f2` if [[ ! -z $DELAY ]]; then echo "Sleeping for $DELAY" >>go.log ; perl perlsleep.pl fi Worker $1 $HOST $PORT $TIMEOUT $DELAY & echo "Host: $HOST Port: $PORT" >> go.log fi done <$FILE