Bash Multi Get using wget

Bash Multi-Get using wget

If you were ever looking at using wget with a proxy – then this is probably the script for you. I haven&squo;t spent much time testing this and it requires a helper program written in perl to get the timer resolution to handle properly. The idea behind this script is to be able to traverse a list of proxies so that you can test accessing your site from a bunch of “different places at once”.

This could also help to simulate a load if put in a loop, this will however only be run from one system so it does not necessarily guarantee a solid load. It also does not guarantee that there won&squo;t be issues caused by using said script.
Nevertheless, have fun playing with this script.

#!/usr/bin/bash
FILE=
DELAY=
TIMEOUT=
function Usage {
echo "Please use   [list] [timeout=] [delay=]" ;
exit 0 ;
}
function Worker {
if [ -z "$5" ]; then
sleep 5$
fi
if [ ! -z "$4" ]; then
export http_proxy=`echo "http://$2:$3"`
wget $1 --timeout=$4 -Y on -q on  --cache=off --output-file=$1.tmp;
if [ "$?" == 0 ]; then
echo "Success:$1 http://$2:$3">> $1.tmp
else
echo "Fail:$1 http://$2:$3">> $1.tmp
fi
else
export http_proxy=`echo "http://$2:$3"`
wget $1 -Y on -q on --cache=off --output-file $1.tmp;
if [ "$?" == 0 ]; then
echo "Success:$1 http://$2:$3">> $1.tmp
else
echo "Fail:$1 http://$2:$3">> $1.tmp
fi
fi
}
function GetFile {
wget #put proxy list here -q on --cache=off --output-document=list.tmp
FILE=list.tmp
}
if [ -z "$1" ]; then
Usage;
fi
if [ -z "$2" ]; then
GetFile;
else
FILE=$2
fi
if [ ! -f "$FILE" ]; then
wget # put proxy list here -q on --cache=off --output-document=list.tmp
FILE=list.tmp
echo "Warning: using web proxy list because list file invalid" ;
fi
TIMEOUT=
DELAY=
rm go.log
if [[ $3 =~ ^timeout=[0-9]*$ ]]; then
TIMEOUT=`echo $3|cut -d"=" -f2`
echo "Setting Timeout to: "$TIMEOUT >> go.log ;
elif [[ $4 =~ ^timeout=[0-9]*$ ]]; then
TIMEOUT=`echo $4|cut -d"=" -f2`
echo "Setting Timeout to: "$TIMEOUT >> go.log ;
fi
if [[ $3 =~ ^delay=[0-9]*$ ]]; then
DELAY=`echo $3|cut -d"=" -f2`
echo "Setting Delay to: "$DELAY >> go.log;
elif [[ $4 =~ ^delay=[0-9]*$ ]]; then
DELAY=`echo $4|cut -d"=" -f2`
echo "Setting Delay to: "$DELAY >> go.log;
fi
while read F
do
if [[ $F =~ ^([0-9]{1,3}.){3}[0-9]{1,3}:[0-9]{1,4}$ ]]; then
HOST=`echo $F|cut -d":" -f1`
PORT=`echo $F|cut -d":" -f2`
if [[ ! -z $DELAY ]]; then
echo "Sleeping for $DELAY" >>go.log ;
perl perlsleep.pl
fi
Worker $1 $HOST $PORT $TIMEOUT $DELAY &
echo "Host: $HOST Port: $PORT" >> go.log
fi
done <$FILE
ttessier

About ttessier

Professional Developer and Operator of SwhistleSoft
This entry was posted in Bash Scripting. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *