From Preparation

Build Systems and Reproducibility

A test fails on a colleague’s machine and passes on mine. We are on the same commit. We spend an afternoon on it and the answer turns out to be that they have OpenSSL 3.0 installed and I have 1.1.1, because they set up their laptop four months after I set up mine, and the build picked up whichever one was sitting in /usr/lib.

Nothing in the repository records that difference. The commit is the same, the test command is the same, and the two of us got different software out the other end. What we ran was not “the code at that commit.” It was the code at that commit plus a few hundred things that happened to be installed on each laptop, and only one of those two ingredients is written down anywhere.

Scale that up. Fifty engineers, a continuous integration fleet, a machine on a manufacturing line, and a build that takes forty minutes. Now the question “why did this binary do that” has no answer, because nobody can reconstruct the machine it was built on.

1. What a build system is doing

A build system takes source files and produces artifacts: binaries, container images, libraries. A shell script does that too. The difference is that a build system knows the dependency graph, meaning which outputs depend on which inputs.

Knowing the graph buys two things a script cannot have.

Incrementality. If a file did not change, and nothing it depends on changed, the thing built from it does not need to be rebuilt. A script has no way to know this, so it rebuilds everything or it guesses with timestamps and gets it wrong.
Parallelism. Two parts of the graph that do not depend on each other can be built at the same time. The system can work this out from the graph rather than being told.

Both of these are only correct if the system genuinely knows every input. If some input is invisible to it, the cache will hand you a stale artifact and the parallel build will race. That is the problem hermeticity solves.

2. Hermeticity

Bazel’s definition, and I want to read it carefully because every word in it is doing work:

When given the same input source code and product configuration, a hermetic build system always returns the same output by isolating the build from changes to the host system.

Three parts.

“The same input source code and product configuration.” The inputs are declared. Not discovered, not inherited from the environment. Declared in the build files.

“Always returns the same output.” Same inputs, same bytes out. Not “a working build,” not “an equivalent binary.” The same output.

“By isolating the build from changes to the host system.” This is the mechanism. The build is made insensitive to what is installed on the machine running it.

The word that captures the consequence is hermetic, meaning sealed. A hermetic build cannot see the host’s OpenSSL, so the host’s OpenSSL cannot change the answer.

2.1 Treating tools as source

The move that makes this work is counterintuitive the first time you see it. The compiler is an input to the build, exactly like a .c file is. So the compiler has to be declared and version-pinned like one.

Bazel puts it as: hermetic build systems “treat tools as source code. They download copies of tools and manage their storage and use inside managed file trees.”

So the build does not use a compiler, it uses the compiler, a specific one it fetched and controls. If I upgrade my system GCC, the build output does not move, because the build was never using my system GCC.

The same reasoning extends to anything the build might reach for: the linker, Python, a code generator, a curl call to some server. Bazel’s phrasing is that hermetic builds avoid relying on “services external to the build environment.” A build step that downloads from a URL at build time is not hermetic, because that URL can serve different bytes tomorrow.

2.2 Sandboxing, and why declaring inputs is not enough

Declaring your inputs is a promise. Sandboxing is enforcement.

Under sandboxing, each build step runs with a filesystem view containing only its declared inputs. If a step secretly reads /usr/include/openssl/ssl.h without declaring it, the file is not there, and the step fails. The undeclared dependency becomes a build error rather than a silent correctness bug that shows up months later as a stale cache entry.

This is the part people find painful, and the pain is the point. Every failure of this kind is a dependency that was real and invisible. The build system is not creating the problem, it is surfacing one that already existed.

2.3 What hermeticity buys

Bazel’s four:

Speed. “The output of an action can be cached, and the action need not be run again unless inputs change.” Once you can identify inputs exactly, you can cache on them safely, including across machines. My CI can reuse an artifact your laptop built, because the identity of an artifact is the identity of its inputs and not the machine that ran it.
Parallel execution. The full graph is known, so the system can schedule it.
Multiple builds on one machine. Two projects wanting different toolchain versions do not conflict, because neither is using the machine’s toolchain.
Reproducibility. “Hermetic builds are good for troubleshooting because you know the exact conditions that produced the build.” This is the one the opening story is about.

3. How Bazel identifies an input

Sameness of inputs is checked with hashes. Bazel hashes the content of every input to an action, and that hash is the cache key.

Content rather than timestamp, and the difference matters. A timestamp says when a file was written. A hash says what is in it. Touch a file without editing it and the timestamp moves while the hash does not, so a hash-keyed cache correctly does nothing. Two engineers who independently write the same file get the same hash and share the cache entry, which a timestamp scheme cannot do.

4. Nix, and the same problem from the other end

Nix answers the same question with a different unit. Bazel’s unit is the build action inside a project. Nix’s unit is the derivation, which the manual describes as a single build task, and its scope is the whole package: the compiler itself, OpenSSL, Python, all of it.

A derivation has three required attributes:

system, the target platform, such as x86_64-linux.
name, a symbolic identifier.
builder, the program that runs to do the build.

Everything else passed to it becomes an environment variable for the builder, and those extras can themselves be other derivations. That is how the graph forms: a derivation for your program takes derivations for its libraries as inputs, and those take their own.

4.1 The output path is the hash

Here is the mechanism worth carrying away. A Nix build output lands at a path that is, in the manual’s words, “a concatenation of the cryptographic hash of all build inputs, the name attribute and the output name.”

So a package does not live at /usr/lib/libssl.so. It lives at something like /nix/store/<hash>-openssl-3.0.8/, where the hash covers every input that produced it: the source, the compiler, the flags, the dependencies, recursively.

Two consequences fall straight out.

Different inputs cannot collide. OpenSSL 1.1.1 and 3.0 hash differently, so they occupy different paths and coexist. The opening story’s failure is not possible, because there is no single /usr/lib for two versions to fight over.
Identical inputs land at the identical path, on any machine. Which is what makes a shared binary cache safe. If a build server already produced /nix/store/<hash>-openssl-3.0.8/, my machine can download it instead of compiling, and it is not trusting the server’s good intentions, it is relying on the path naming the exact inputs.

Nix also scans build outputs for references to input paths, looking for the hash portions, and registers what it finds as runtime dependencies. So the dependency list is derived from what the artifact actually references rather than from what someone wrote in a manifest.

5. Why both

They are not competitors, and the division of labour is clean once you see the two scopes.

Nix produces the environment. The exact compiler, the exact system libraries, the exact toolchain, pinned by hash.
Bazel builds your code inside it, with fine-grained per-target caching and parallelism across a large repository.

Bazel is very good at “rebuild only the seventeen targets affected by this change” and is not in the business of pinning your libc. Nix is very good at “this is precisely the world the build runs in” and does not give you target-level incrementality inside a million-line repository. Used together, Nix answers what the machine is and Bazel answers what to rebuild.

6. What it costs

Worth being honest about, because the cost is what makes this a real decision.

The build files become a real artifact. Bazel wants dependencies declared per target. When you add an import, you also edit a build file. Tooling can generate a lot of this, and it is still a tax on every change.

Third-party code has to be brought inside. A dependency that expects to ./configure against whatever the system has needs work to make hermetic. This is where most of the initial effort goes.

The failure mode moves earlier and gets louder. A missing dependency used to be a bug that showed up in production. Now it is a build failure in front of an engineer who did not think they were touching that. That trade is correct and it does not feel correct at 6pm.

Reproducible is not automatically bit-identical. Hermeticity removes the host as a variable. It does not stop a compiler embedding a timestamp, or a build step iterating a hash map in nondeterministic order, or an archive recording file ordering that varies. Those are separate sources of nondeterminism inside the build itself, and they get fixed one at a time. A hermetic build is a precondition for a bit-identical one, not a guarantee of it.

The payoff against all of that is a single question becoming answerable: given this artifact, what exactly produced it? At a company where an artifact might end up on an implanted device or a manufacturing-line machine, that question is not an engineering convenience.