Transparent (Proxy) Avast Mirror

Recently I was confronted with the situation of many users Avast Antivirus installations updating from the internet, and downloading many MB’s of files on a limited internet connection. Also, due to the speed of the connection, and the fact that they are going through a captival portal, the downloads take considerable time, and are counted towards their download quota. Due to a limitation in the captive portal, I can’t change it so that the Avast downloads aren’t counted in their download limit, but if they don’t login to the portal, then the download won’t count towards their quota.

So part of the solution was going to be to make the downloads happen quicker. It was already going through a Squid proxy, which should have made this happen, but it still meant the downloads would be slow the first time, so one user would be charged for the downloads, and not subsequent users. And due to the fact that it load balances the servers, the chances of all users hitting the same avast download server is next to nothing. My searches on the net lead me to http://files.avast.com/files/eng/mirror.zip which is an application from the Avast Distributed Network Manager that sets up a central server for managing avast in a corporate setting. After playing around with the mirror.exe file that generates a mirror, and discovering that adding “restrict_products=av_pro” to the mirror.ini file, I could limit the mirror generation to just the normal avast files for program and virus definition updates.

So now I had an Avast mirror, I now needed to do 2 things, find a way to generate an avast mirror, and update it without using the windows only, mirror.exe program, and find a way to make Avast clients update from the mirror, and not the internet.

The first problem was solved with a bit of poking around in the files downloaded by mirror.exe, and discovering that it checks a couple of “stamp” files on the server, if the stamp files are new, it downloads the file it is a stamp for, in particular, mirror.def, which contains a complete list of all the files needed for a mirror, with the MD5 sum of the file, and filesize, as well as the “products” that utilise the file. This file makes it easy to generate a mirror!! Below is the script that generates a mirror. There are a few things it doesn’t do that mirror.exe does. It doesn’t generate the 400.vps files, which Linux clients use to update from (all other clients should use the partial update files to save downloading 20Mb+ every time). It also doesn’t generate a new mirror.def file that only contains the files that this partial contains, instead it uses the same mirror.def file that it used to generate the mirror from the master servers. It also doesn’t modify the servers.def file. The main reason for this is that these files are signed, and we don’t yet know how to sign them ourselves, so the clients would reject any modifications. Also, unless someone who will be behind the proxy, and thus using your mirror, attempts to create their own more complete mirror, mirror.def being wrong doesn’t matter. Also, changing the servers.def file to point to your mirror as mirror.exe does, would prevent users from later updating Avast on another network that doesn’t contain your mirror.

The next part of the solution is how to redirect requests for Avast updates to your mirror, without Avast knowing it isn’t getting it’s updates from the real servers, so that no modifications to Avast are needed.
So this part actually worked out a lot easier than initially thought. All users currently go through a Squid proxy, transparently, due to filtering and the obvious benefit of caching on a limited internet connection. I thought that with Avast attempting it’s updates from a variety of servers throughout the world, and with both IP and domain name attempts (it alternates between the 2 so that DNS outages, or DNS hijacks, can’t prevent it from updating), it would be difficult to write a generic filter to redirect them. Then I realised that every request from avast, was to a directory, iavs4x. Suddenly I didn’t need to worry about which server it was attempting to connect to, just the directory. So I wrote a Squid Redirector that rewrites any request that is to the directory, iavs4x, to go to the server that hosts the mirror. You can setup ACL’s so that not every URL goes through the redirector, but as I already use a redirector for adzapping, I didn’t pursue this option. The script is below, change the server variable to reflect your mirror server.

avast_mirror.sh

#!/bin/bash

#### Avast Mirror
#
# Files that should be in mirror but won't be
# 400.vps 400.vps.md5
#
# Extra files we need that aren't in mirror.def
# jollyroger.vpu.stamp jollyroger.vpu servers.def servers.def.stamp servers.def.vpu.stamp servers.def.lkg  servers.def.vpu

# BASEURL (Select a random from mirrors file? with fallback, currently just default)
BASEURL="http://files.avast.com/iavs4x/"
mirror_def=mirror.def
BASEDIR=/home/mirror
TMP_URL_LIST=/tmp/mirror_files

# Filter for mirror.def
restrict_products='av_pro'

# Set DEBUG to anything other than '' to get debugging info
DEBUG=''


cd $BASEDIR

# Files updated from stamp files
function stamp_update {
	file=$1
	stampfile=$1.stamp
	old_stamp=$(cat $stampfile)
	file_stamp=$file_stamp.new
	wget -q -O $file_stamp $BASEURL$stampfile
	new_stamp=$(cat $file_stamp)
	if test "x$old_stamp" != "x$new_stamp" ; then
		echo "$file out of date."
		echo "Updating..."
		mv $file_stamp $stampfile
		mv -f $file $file.old
		wget -nv $BASEURL$file
		echo "$file updated"
	else
		if test -n "$DEBUG" ; then echo "$file up to date.";fi
	fi
}

stamp_update mirror.def
stamp_update jollyroger.vpu
stamp_update servers.def
stamp_update servers.def.vpu

# Read in mirror.def, check all files and add out of date ones to queue
	rm -f $TMP_URL_LIST
	# Add "extra" files that are always downloaded here
	#echo  $BASEURL"servers.def.lkg" > $TMP_URL_LIST
	filelist=$(cat $mirror_def |grep $restrict_products|cut -f 2 -d =|cut -f 1-3 -d ,)
	for file in $filelist
	do
		info=( `echo "$file" | tr -s ',' ' '` )
		filename=${info[0]}
		filesize=${info[1]}
		filemd5=${info[2]}
		if test ! -f $filename
			then
			if test -n "$DEBUG" ; then echo;fi
			echo "Adding new $filename to download queue"
			echo $BASEURL$filename >> $TMP_URL_LIST
			continue
		fi
		curfilesize=$(stat -c%s "$filename")
		if test "0$curfilesize" -lt "0$filesize" 
			then
			if test -n "$DEBUG" ; then echo;fi
			echo "Adding incomplete $filename to download queue"
			echo $BASEURL$filename >> $TMP_URL_LIST
			continue
		fi

		curfilemd5=$(md5sum "$filename" | cut -f 1 -d ' '|tr [:lower:] [:upper:])
		if test "x$filemd5" != "x$curfilemd5"
			then
			mv -f $filename $filename.old
			if test -n "$DEBUG" ; then echo;fi
			echo "Adding corrupt $filename to download queue"
			echo $BASEURL$filename >> $TMP_URL_LIST
			continue
		fi
		if test -n "$DEBUG" ; then echo -n ".";fi
		#echo "$filename already downloaded and complete"		
	done
	if test -n "$DEBUG" ; then echo;fi
	if test -s $TMP_URL_LIST
		then		
		echo "Downloading out of date mirror files"
		#Download files
		# --no-dns-cache means it'll load balance randomly accross different servers for each file
		wget -c -nv -N --no-dns-cache -i /tmp/mirror_files --progress=dot |tee -a mirror.log
	fi

avast_redirect.pl

#!/usr/bin/perl
$|=1;
$server='10.1.0.1';
while (<>) {
    @X = split;
    $url = $X[0];
    if ($url =~ /^http:\/\/[^\/]*\/iavs4x\// && $url !~ m/$server/o) {
        $url =~ s/^http:\/\/[^\/]*\/iavs4x\//http:\/\/$server\/iavs4x\//o;
        print "$url\n";
    } else {
        print "\n";
    }
}

2 thoughts on “Transparent (Proxy) Avast Mirror

  1. This is why I moved to NOD32, especially for multiple computers.
    You can set one install as a webhost for virus definitions, as well as creating automatic disk copies of them as well.
    With Amnet still providing ‘unmetered’ WAIX this has saved me a reasonable amount of quota.

  2. Unfortunately as all the client computers on the network this is implemented on, are constantly changing and not under my control, any changes to the client side render it impractical. It all needs to be transparent and server side.

Comments are closed.