Playing around with distcc
2017-03-20
Today, I decided to install Gentoo on a spare machine I had lying
around, since I was bored. Obviously, the first issue I ran into was
that emerge x11-base/xorg-server was taking a really long time to
run since the Xorg server is a pretty bloated program. Then emerge
firefox was taking forever too.
One solution (for Firefox anyway) was to use the provided binary
packages, this meant firefox-bin for Firefox. But this means I
abandon all of the nice features (for me, that means USE flags) that
Gentoo offers. If I'm going to download a load of binaries that I
can't customize, why not just install Debian?
So the solution is to speed up compilation. That means putting more
CPU cores to work. But my poor old ThinkPad only has 2 cores! This is
where distcc comes in.
What is distcc?
It's a way of distributing compilation jobs to build servers which are supposed to have huge beefy CPUs. This saves a lot of time.
To explain how to set it up, I only need quote the distcc man page:
1 For each machine, download distcc, unpack, and install.
2 On each of the servers, run distccd --daemon with --allow
options to restrict access.
3 Put the names of the servers in your environment:
$ export DISTCC_HOSTS='localhost red green blue'
4 Build!
$ make -j8 CC=distcc
This is an OK explanation of the general process, but the specifics can get a bit more fiddly than that.
My setup
Because I didn't want to put too much effort into standardising the
OSs of my machines, my pile of shitty laptops build cluster runs a
whole host of different Linux distros. There's Debian, Slackware,
Ubuntu, Arch. I really wanted to use my OpenBSD server in here, but
sadly distcc doesn't abstract away the build server OS that much: the
files it sends just get compiled by the server's compiler then sent
back - nothing too fancy.
My instructions will focus on Debian, since my Xeon desktop and dual-Opteron server both run Debian - all the other machines probably contribute less than 10% of the total CPU power.
Installing and configuring on Gentoo should be basically the same if you're using systemd.
Installing
The package and ebuild are both named distcc, so just use apt or
emerge as appropriate. Also install distcc-pump, for an extra
speed boost: it offloads even more of the build process to the server
(take a look at
this).
Configuring the client
There are a few things that cause distcc to fail.
On the client (where the jobs get sent from), in /etc/distcc/hosts,
there may be a line reading +zeroconf. If you don't use Zeroconf,
just delete this line. It will likely cause distcc to fail to
distribute jobs even though it's supposed to realise you're not using
Zeroconf. Type the IP addresses or hostnames (if they will resolve
correctly) here, separated by spaces. For each IP, add ,cpp,lzo to
the end, so that pump mode will work correctly and compression will be
used to send files.
So an example config might look like:
192.168.0.32,cpp,lzo big-server,cpp,lzo my-desktop,cpp,lzo
Then go to /etc/portage/make.conf and add distcc distcc-pump
to the FEATURES variable (or create it if it doesn't exist).
Change MAKE_OPTS to -jN -lM where N is double the total number
of CPU cores available and M is however many jobs you want to run
locally (in my case, 2). We pick double the number of CPU cores since
the jobs are relatively small: it's possible the network will
bottleneck us if the number of jobs is too small and we have to wait
for many round trips for small batches of work.
Configuring the server
After installing distcc, you need to edit /etc/default/distcc.
Add your subnet to ALLOWEDNETS. Mine looks like:
ALLOWEDNETS="192.168.0.0/24"
Change LISTENER to your IP address. That is, the IP address the
client will use to acces the server. For me that's:
LISTENER="192.168.0.83"
And set ZEROCONF to "false", just in case it messes something up.
Now, enable and start the service:
$ systemctl enable distcc
$ systemctl start distcc
There is one small thing left to do. The platform on my Gentoo
installation is x86_64-pc-linux-gnu, so when it sends out a job, it
expects the compiler to be x86_64-pc-linux-gnu-gcc. This is not
actually the case on my Debian installation, where the compiler is
called x86_64-linux-gnu-gcc. Note that the pc is missing. This
makes no difference - it really is the same platform. But unless we
fix this, distcc will fail.
The solution (maybe a bit of a hack) is to just symlink all the platform-specific stuff (ld, nm, gcov, gcc, g++ etc). Here's how I did that:
$ ls | grep x86_64-linux | while read a; do sudo ln -sf $a ${a//linux/pc-linux}; done
This certainly works, maybe there's a better way to do it, though.
Since we are working in a single LAN, I assume you don't firewall traffic between computers in your LAN (if you're its only user, for example). If you do, you want to open port 3632 on your servers.
The moment of truth
Now, try to emerge a random package using pump and distcc as
follows:
$ pump emerge firefox
This should transparently just use distcc and send some jobs over to
one or more of your servers.
Take a look at /var/log/distcc.log on one of your servers, it should
be filled with lines like this:
distccd[14465] (dcc_job_summary) client: 192.168.0.82:36466 COMPILE_OK exit:0 sig:0 core:0 ret:0 time:169ms x86_64-pc-linux-gnu-g++ main.cpp
My build time is 1/30 of what it was before, it's incredible. Of
course, what I save in time, I lose in electricity. So maybe next time
I'll just use a binary distro. But then, of course, I wouldn't be able
to make use of USE flags to stop 20 scripting languages getting pulled
in when I install sudo.
Happy compiling!