kaashif's blog

Programming, software freedom and Unix


Playing around with distcc

Today, I decided to install Gentoo on a spare machine I had lying around, since I was bored. Obviously, the first issue I ran into was that emerge x11-base/xorg-server was taking a really long time to run since the Xorg server is a pretty bloated program. Then emerge firefox was taking forever too.

One solution (for Firefox anyway) was to use the provided binary packages, this meant firefox-bin for Firefox. But this means I abandon all of the nice features (for me, that means USE flags) that Gentoo offers. If I’m going to download a load of binaries that I can’t customize, why not just install Debian?

So the solution is to speed up compilation. That means putting more CPU cores to work. But my poor old ThinkPad only has 2 cores! This is where distcc comes in.

What is distcc?

It’s a way of distributing compilation jobs to build servers which are supposed to have huge beefy CPUs. This saves a lot of time.

To explain how to set it up, I only need quote the distcc man page:

1 For each machine, download distcc, unpack, and install.

2 On each of the servers, run distccd --daemon with --allow
  options to restrict access.

3 Put the names of the servers in your environment:
  $ export DISTCC_HOSTS='localhost red green blue'

4 Build!
  $ make -j8 CC=distcc

This is an OK explanation of the general process, but the specifics can get a bit more fiddly than that.

My setup

Because I didn’t want to put too much effort into standardising the OSs of my machines, my ~~pile of shitty laptops~~ build cluster runs a whole host of different Linux distros. There’s Debian, Slackware, Ubuntu, Arch. I really wanted to use my OpenBSD server in here, but sadly distcc doesn’t abstract away the build server OS that much: the files it sends just get compiled by the server’s compiler then sent back - nothing too fancy.

My instructions will focus on Debian, since my Xeon desktop and dual-Opteron server both run Debian - all the other machines probably contribute less than 10% of the total CPU power.

Installing and configuring on Gentoo should be basically the same if you’re using systemd.

Installing

The package and ebuild are both named distcc, so just use apt or emerge as appropriate. Also install distcc-pump, for an extra speed boost: it offloads even more of the build process to the server (take a look at this).

Configuring the client

There are a few things that cause distcc to fail.

On the client (where the jobs get sent from), in /etc/distcc/hosts, there may be a line reading +zeroconf. If you don’t use Zeroconf, just delete this line. It will likely cause distcc to fail to distribute jobs even though it’s supposed to realise you’re not using Zeroconf. Type the IP addresses or hostnames (if they will resolve correctly) here, separated by spaces. For each IP, add ,cpp,lzo to the end, so that pump mode will work correctly and compression will be used to send files.

So an example config might look like:

192.168.0.32,cpp,lzo big-server,cpp,lzo my-desktop,cpp,lzo

Then go to /etc/portage/make.conf and add distcc distcc-pump to the FEATURES variable (or create it if it doesn’t exist).

Change MAKE_OPTS to -jN -lM where N is double the total number of CPU cores available and M is however many jobs you want to run locally (in my case, 2). We pick double the number of CPU cores since the jobs are relatively small: it’s possible the network will bottleneck us if the number of jobs is too small and we have to wait for many round trips for small batches of work.

Configuring the server

After installing distcc, you need to edit /etc/default/distcc.

Add your subnet to ALLOWEDNETS. Mine looks like:

ALLOWEDNETS="192.168.0.0/24"

Change LISTENER to your IP address. That is, the IP address the client will use to acces the server. For me that’s:

LISTENER="192.168.0.83"

And set ZEROCONF to "false", just in case it messes something up.

Now, enable and start the service:

# systemctl enable distcc
# systemctl start distcc

There is one small thing left to do. The platform on my Gentoo installation is x86_64-pc-linux-gnu, so when it sends out a job, it expects the compiler to be x86_64-pc-linux-gnu-gcc. This is not actually the case on my Debian installation, where the compiler is called x86_64-linux-gnu-gcc. Note that the pc is missing. This makes no difference - it really is the same platform. But unless we fix this, distcc will fail.

The solution (maybe a bit of a hack) is to just symlink all the platform-specific stuff (ld, nm, gcov, gcc, g++ etc). Here’s how I did that:

$ ls | grep x86_64-linux | while read a; do sudo ln -sf $a ${a//linux/pc-linux}; done

This certainly works, maybe there’s a better way to do it, though.

Since we are working in a single LAN, I assume you don’t firewall traffic between computers in your LAN (if you’re its only user, for example). If you do, you want to open port 3632 on your servers.

The moment of truth

Now, try to emerge a random package using pump and distcc as follows:

# pump emerge firefox

This should transparently just use distcc and send some jobs over to one or more of your servers.

Take a look at /var/log/distcc.log on one of your servers, it should be filled with lines like this:

distccd[14465] (dcc_job_summary) client: 192.168.0.82:36466 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:169ms x86_64-pc-linux-gnu-g++ main.cpp

My build time is 1/30 of what it was before, it’s incredible. Of course, what I save in time, I lose in electricity. So maybe next time I’ll just use a binary distro. But then, of course, I wouldn’t be able to make use of USE flags to stop 20 scripting languages getting pulled in when I install sudo.

Happy compiling!