Sunday, December 21, 2008

Ditching apt-cacher-ng for Squid

apt-cacher-ng hasn't exactly worked out as well as I had hoped. I kept having a problem where the Releases file from the apt repository would be reported as corrupted. I could go into the cache directory and manually remove that file, forcing it to be downloaded again, but that would only help with the system I was updating at the moment. The next system that I tried to update would have the same problem.

I've read a number of articles about just using Squid as an apt cache. I avoided this at first because Squid isn't really made for that. For example squid has no way of knowing when a specific version of a package has been made obsolete by a newer version. I assume apt-cacher-ng and apt-proxy know how to do this (maybe not). Also Squid isn't intended to cache arbitrarily sized objects for indefinite periods of time. I decided to give it a try though, since a number of people seem to be having success with this.

I had to tweak Squid's default configuration a bit to make it suitable for apt-caching.
The Squid configuration file is well documented, but below are the directives of interest:

refresh_pattern deb$ 1577846 100% 1577846
refresh_pattern Packages.gz$ 1440 100% 1440
cache_dir ufs /var/spool/squid 15000 2 8
maximum_object_size 409600 KB

The first line says to cache anything ending ('$' is the end-of-line anchor) in "deb" for 3 years. There are some packages that rarely get updated if ever, so I want to make they stay in the cache the entire time I'm using a given distribution release. I figure I'll probably be on a particular distribution release for no longer than three years.

The second line says to cache anything ending in "Packages.gz" for one day.

The "cache_dir" line says to put Squid's cache in /var/spool/squid (I believe this is the default). It also says to let it grow no larger than 15,000 megabytes. My system partition gets very little use, so 15Gb is no problem for me. The second two numbers tell Squid how to structure the cache. "2" says to create two level 1 directories and "8" says to create 8 level 2 directories. The default is 16 and 256. I read an article where the author was having a problem with the hard drive never spinning down because Squid was rescanning the cache every few seconds. The author said reducing the number of L1 and L2 directories helped. If anyone can find this article, please post a link in the comments.

The "maximum_object_size" tells squid to cache objects up to 400Mb. There shouldn't be any debs even close to that large. Squid's default is much smaller than this.

In order to get apt to use the proxy, I created a file called "proxy" in /etc/apt/apt.conf.d/. You can call the file whatever you want. The file contains the line:

Acquire::http::Proxy "http://hoth:3128";

"hoth" is the hostname of the Squid server. Make sure there are no "Acquire" directives in any other apt configuration file:

/etc/apt/ # grep -rn Acquire *

I discovered that on some systems there was an "Acquire" directive explicitly telling apt not to use a proxy.

So far, Squid has been working pretty well. There was one instance where apt failed to completely download a package, but after running apt-get dist-upgrade again, the package successfully downloaded.