Humanizing the Build Process with SCons

Rave Software Development

Nearly everyone who has built software on Unix systems has had to deal with a tool called GNU Make.

A lot of people like GNU Make at first; I know I did. Makefiles–the scripts that you write to define how to build your software–define things in terms of declarative rules, as opposed to imperative commands. Essentially, you write down what files depend on what, how to build each file, and Make takes care of building the software as quickly as possible. For instance, here’s a really simple Makefile:

  HelloWorld : HelloWorld.c
      gcc HelloWorld.c -o HelloWorld

In plain English, the first line of this file says “the program HelloWorld depends on a file called HelloWorld.c.” The second line contains the shell command that Make runs if it decides that the HelloWorld program needs to be rebuilt. One of Make’s nicest features is that it is smart enough to rebuild only the necessary files–so if you change one file in a large source tree, Make only rebuilds that one file and the files that depended on it.

It is a pretty neat concept, but after using it for a while, I realized that building even moderately complex programs required massive, obfuscated Makefiles. There are little “shortcuts” that Makefiles can use that tell them how to, say, turn any C file into a program, but these shortcuts are really difficult to remember, and even harder to read. After a while, I dreaded writing them, and would sometimes spend hours tweaking them in an effort to make them as readable and modular as possible–an endeavor which I usually failed at.

It’s sad when reading a build file is harder than reading the source code it’s building. But that’s usually the case with Makefiles.

Recently, though, our treasurer Andrew pointed me to a new build tool called SCons. Like Make, SCons uses a declarative rule-based system to figure out what it needs to build and when, but it adds a layer of object-orientation into the mix that makes the programmer’s job a whole lot more humane. Here’s the equivalent of the above Makefile in SCons syntax:

  Program( source = "HelloWorld.c" )

That’s pretty much it, and to anyone who’s ever programmed in a procedural or object-oriented language, it should look pretty straightforward.

In fact, it’s Python.

That’s one of the other great things about SCons: because your build files are written in an eminently readable, full-fledged programming language, they become just another part of your source code. You can create classes and functions. You can refactor. You can apply the exact same coding standards that you use in Python files to SCons files. You can even perform code reviews on them.

But aside from that, SCons abstracts out a whole lot of the nitty-gritty details involved in building software, as is plainly evident from the two examples above. This makes build files a lot easier to understand, maintain, and extend. And not only that, but in abstracting out the platform-specific details of the build process, the build files become easily portable to different operating systems and build environments, which can’t be said for the Makefile above.

So, it goes without saying that we’re using SCons here at Humanized for all our software-building needs. It keeps simple things simple while still allowing us to customize the heck out of it when we need to, and it’s easy to learn–thanks in no small part to its extensive documentation. And it just works.

I never thought writing build files could be this pleasant.

Addendum: I recently gave a presentation on SCons at the April Chicago Python User Group (ChiPy) meeting, and have put my presentation slides online if you’d like to use them. My presentation design notes are written as comments at the top of the HTML source code.

I couldn’t disagree more. I work in a medium-sized software shop where we use scons to build a few million lines of C++ code (and lots of other assorted junk) on several platforms.

I am honestly kinda fond of scons, but it is not easy to use or to understand. Our build system has ballooned to ten thousand lines of Python code.

scons is smarter than most of us, myself included. Trying to figure out what it is thinking when things go wrong in a large project is practically impossible.

Today, I had a build failing in between build steps because scons was attempting to stat() a file that didn’t exist. The file was not even supposed to be part of the build anymore. scons’s error message was useless. (It turned out scons was tripping over a broken symlink.)

Anyone looking for a build system for their busniess should pass on scons, at least until it reaches 1.0. Lone geniuses looking for a really cool alternative build tool to kick around at home should absolutely test their wits against it. Guaranteed fun.

Thanks for the comments, Jason. The projects we’re using SCons for aren’t as big as yours, so it’s good to know how SCons is viewed on some more large-scale software. Another issue we’ve had a bit of a problem with has been the amount of time it takes SCons to process all its build files, which I imagine you’ve run into as well.

We’ve also run into other problems with SCons: it couldn’t detect our particular compiler, for instance, so we submitted a patch that fixed the issue. So far, though, we’ve found that fixing SCons is a better (and more productive) alternative than trying to switch to any of the other solutions out there–for instance, while your build files are ten thousand lines of Python code, I wonder how many lines of makefiles they’d be, or how complex equivalent MSVS project files would be.

On that note, we’ve also found refactoring our SCons files to be invaluable in reducing the size of our build files. For instance, we’ve subclassed the default Environment class to do a lot of common operations for us, and we’ve created our own Builders and Scanners that make things much easier to maintain. I might go so far as to say that our build files have actually been getting smaller over the past few weeks. I guess I’d say that what I like most about SCons is that it allows us to keep our build files as manageable as our own source code.

But, that said, we’ll keep the roadblocks you mentioned in mind, and proceed with caution.

Check out Ruby rake for another take on the same problem.

A totally manual build process, i.e. tracking dependency manually, is absolutely inhuman. Often in source code there is enough information to build automatically; what actually do any decent IDE.

See:

• Clump — say goodbye to Makefiles
• make - Wikipedia
• Topological sorting - Wikipedia