Rebuilding software five years later, how hard can it be? It can’t bethat hard, especially when you pride yourself on having a tool thatcan travel intimeand that does a good job at ensuring reproduciblebuilds, right?

In hindsight, we can tell you: it’s more challenging than itseems. Users attempting to travel 5 years back with guix time-machineare (or were) unavoidably going to hit bumps on the road—a realproblem because that’s one of the use cases Guix aims to support well,in particular in a reproducibleresearch context.

In this post, we look at some of the challenges we face while travelingback, how we are overcoming them, and open issues.

The visionFirst of all, one clarification: Guix aims to support time travel, butwe’re talking of a time scale measured in years, not in decades. Weknow all too well that this is already very ambitious—it’s somethingthat probably nobody except Nix and Guix are eventrying. More importantly, software deployment at the scale of decadescalls for very different, more radical techniques; it’s the work ofarchivists.

Concretely, Guix 1.0.0 was released in2019 andour goal is to allow users to travel as far back as 1.0.0 and redeploysoftware from there, as in this example:

$ guix time-machine -q --commit=v1.0.0 -- \ environment --ad-hoc python2 -- python> guile: warning: failed to install localePython 2.7.15 (default, Jan 1 1970, 00:00:01) [GCC 5.5.0] on linux2Type "help", "copyright", "credits" or "license" for more information.>>> (The command above uses guix environment, the predecessor of guix shell,which didn’t exist back then.)It’s only 5 years ago but it’s pretty much remote history on the scaleof software evolution—in this case, that history comprises majorchanges in Guixitself andin Guile.How well does such a command work? Well, it depends.

The project has two build farms; bordeaux.guix.gnu.org has beenkeeping substitutes (pre-built binaries) of everything it built sinceroughly 2021, while ci.guix.gnu.org keeps substitutes for roughly twoyears, but there is currently no guarantee on the durationsubstitutes may be retained.Time traveling to a period where substitutes are available isfine: you end up downloading lots of binaries, but that’s OK, you ratherquickly have your software environment at hand.

Bumps on the build roadThings get more complicated when targeting a period in time for whichsubstitutes are no longer available, as was the case for v1.0.0 above.(And really, we should assume that substitutes won’t remain availableforever: fellow NixOS hackers recently had to seriously considertrimming their 20-year-long history ofsubstitutesbecause the costs are not sustainable.)

Apart from the long build times, the first problem that arises in theabsence of substitutes is source code unavailability. I’ll spare youthe details for this post—that problem alone would deserve a book.Suffice to say that we’re lucky that we started working on integratingGuix with SoftwareHeritageyears ago, and that there has been great progress over the last coupleof years to get closer to full package source codearchival (more precisely: 94% ofthe source code of packages available in Guix in January 2024 isarchived, versus 72% of the packages available in May 2019).

So what happens when you run the time-machine command above? Itbrings you to May 2019, a time for which none of the official buildfarms had substitutes until a few days ago. Ideally, thanks toisolated buildenvironments,you’d build things for hours or days, and in the end all those binarieswill be here just as they were 5 years ago. In practice though, thereare several problems that isolation as currently implemented does notaddress.

Among those, the most frequent problem is time traps: software buildprocesses that fail after a certain date (these are also referred to as“time bombs” but we’ve had enough of these and would rather call for aceasefire). This plagues a handful of packages out of almost 30,000 butunfortunately we’re talking about packages deep in the dependency graph.Here are some examples:

• OpenSSL unit tests failafter a certain date because some of the X.509 certificates they usehave expired.
• GnuTLS had similar issues;newer versions rely ondatefudge tofake the date while running the tests and thus avoid that problemaltogether.
• Python 2.7, found in Guix 1.0.0, also had thatproblem with its TLS-relatedtests.
• OpenJDK would fail to build at somepoint with this interestingmessage: Error: time is more than 10 years from present: 1388527200000 (the build system would consider that its data aboutcurrencies is likely outdated after 10 years).
• Libgit2, a dependency of Guix, had (has?) a time-dependenttests.
• MariaDB tests started failing in2019.

Someone traveling to v1.0.0 will hit several of these, preventingguix time-machine from completing. A serious bummer, especially tothose who’ve come to Guix from the perspective of making their researchworkflowreproducible.

Time traps are the main road block, but there’s more! In rare cases,there’s software influenced by kernel details not controlled by thebuild daemon:

• Tests of the hwloc hardware locality library would fail whenrunning on a Btrfs file system.

In a handful of cases, but important ones, builds might fail whenperformed on certain CPUs. We’re aware of at least two cases:

• Python 3.9 to 3.11 would set a signal handler stack too small foruse on Intel Sapphire Rapids XeonCPUs (it’s morecomplicated than this but the end result is: it will no longer buildon modern hardware).
• Firefox would reportedly crash on Raptor Lake CPUs running an buggyversion of theirfirmware.

Neither time traps nor those obscure hardware-related issues can beavoided with the isolation mechanism currently used by the build daemon.This harms time traveling when substitutes are unavailable. Giving upis not in the ethos of this project though.

Where to go from here?There are really two open questions here:

1. How can we tell which packages needs to be “fixed”, and how:building at a specific date, on a specific CPU?
2. How can keep those aspects of the build environment (time, CPUvariant) under control?

Let’s start with #2. Before looking for a solution, it’s worthremembering where we come from. The build daemon runs build processeswith a separate root filesystem, underdedicated user IDs, and in separate Linuxnamespaces,thereby minimizing interference with the rest of the system and ensuringa well-defined buildenvironment.This technique wasimplementedby Eelco Dolstra for Nix in 2007 (with namespace support addedin2012),at a time where the word container had to do with boats and before“Docker” became the name of a software tool. In short, the approachconsists in controlling the build environment in every detail (it’s atodds with the strategy that consists in achieving reproducible buildsin spite of high build environmentvariability).That these are mere processes with a bunch of bind mounts makes thisapproach inexpensive and appealing.

Realizing we’d also want to control the build environment’s date,we naturally turn to Linux namespaces to address that—Dolstra, Löh, andPierron already suggested something along these lines in the conclusionof their 2010 Journal of Functional Programmingpaper. Turns outthere is now a timenamespace.Unfortunately it’s limited to CLOCK_MONOTONIC and CLOCK_BOOTTIMEclocks; the manual page states:

Note that time namespaces do not virtualize the CLOCK_REALTIMEclock. Virtualization of this clock was avoided for reasons ofcomplexity and overhead within the kernel.

I hear you say: What aboutdatefudge andlibfaketime?These rely on the LD_PRELOAD environment variable to trick the dynamiclinker into pre-loading a library that provides symbols such asgettimeofday and clock_gettime. This is a fine approach in somecases, but it’s too fragile and too intrusive when targeting arbitrarybuild processes.

That leaves us with essentially one viable option: virtual machines(VMs). The full-system QEMU lets you specify the initial real-timeclock of the VM with the -rtc flag, which is exactly what we need(“user-land” QEMU such as qemu-x86_64 does not support it). And ofcourse, it lets you specify the CPU model to emulate.

News from the pastNow, the question is: where does the VM fit? The author consideredwriting a packagetransformationthat would change a package such that it’s built in a well-defined VM.However, that wouldn’t really help: this option didn’t exist in pastrevisions, and it would lead to a different build anyway from theperspective of the daemon—a differentderivation.

The best strategy appeared to beoffloading:the build daemon can offload builds to different machines over SSH, wejust need to let it send builds to a suitably-configured VM. To dothat, we can reuse some of the machinery initially developed forchildhurdsthat takes care of setting up offloading to the VM: creating substitutesigning keys and SSH keys, exchanging secret key material between thehost and the guest, and so on.

The end result is a service for Guix Systemusersthat can be configured in a few lines:

(use-modules (gnu services virtualization))(operating-system ;; … (services (append (list (service virtual-build-machine-service-type)) %base-services))) The default setting above provides a 4-core VM whose initial date isJanuary 2020, emulating a Skylake CPU from that time—the right setup forsomeone willing to reproduce old binaries. You can check theconfiguration like this:

$ sudo herd configuration build-vmCPU: Skylake-Clientnumber of CPU cores: 4memory size: 2048 MiBinitial date: Wed Jan 01 00:00:00Z 2020 To enable offloading to that VM, one has to explicitly start it, likeso:

$ sudo herd start build-vm From there on, every native build is offloaded to the VM. The key partis that with almost no configuration, you get everything set up to buildpackages “in the past”. It’s a Guix System only solution; if you runGuix on another distro, you can set up a similar build VM but you’llhave to go through the cumbersome process that is all taken care ofautomatically here.

Of course it’s possible to choose different configuration parameters:

(service virtual-build-machine-service-type (virtual-build-machine (date (make-date 0 0 00 00 01 10 2017 0)) ;further back in time (cpu "Westmere") (cpu-count 16) (memory-size (* 8 1024)) (auto-start? #t))) With a build VM with its date set to January 2020, we have been able torebuild Guix and its dependencies along with a bunch of packages such asemacs-minimal from v1.0.0, overcoming all the time traps and otherchallenges described earlier. As a side effect, substitutesare now available from ci.guix.gnu.org so you can even try this athome without having to rebuild the world:

$ guix time-machine -q --commit=v1.0.0 -- build emacs-minimal --dry-runguile: warning: failed to install localesubstitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%38.5 MB would be downloaded: /gnu/store/53dnj0gmy5qxa4cbqpzq0fl2gcg55jpk-emacs-minimal-26.2 For the fun of it, we went as far as v0.16.0, released in December2018:

guix time-machine -q --commit=v0.16.0 -- \ environment --ad-hoc vim -- vim --version This is the furthest we can go sincechannelsand the underlying mechanisms that make time travel possible did notexist before that date.

There’s one “interesting” case we stumbled upon in that process: inOpenSSL 1.1.1g (released April 2020 and packaged in December2020),some of the test certificates are not valid before April 2020, so thebuild VM needs to have its clock set to May 2020 or thereabouts.Booting the build VM with a different date can be done withoutreconfiguring the system:

$ sudo herd stop build-vm$ sudo herd start build-vm -- -rtc base=2020-05-01T00:00:00 The -rtc … flags are passed straight to QEMU, which is handy whenexploring workarounds…

The time-travel continuous integrationjobset has been set up tocheck that we can, at any time, travel back to one of the past releases.This at least ensures that Guix itself and its dependencies havesubstitutes available at ci.guix.gnu.org.

Reproducible research workflows reproducedIncidentally, this effort rebuilding 5-year-old packages has allowed usto fix embarrassing problems. Software that accompanies research papersthat followed our reproducibilityguidelinescould no longer be deployed, at least not without this clock twiddlingeffort:

• codeof [Re] Storage Tradeoffs in a Collaborative Backup Service forMobile Devices, submittedas part of the ReScience Ten Years ReproducibilityChallenge in June 2020,and which is precisely about showcasing reproducible deployment withGuix;
• codeof the 2022 Nature Scientific Data article entitled Towardpractical transparent verifiable and long-term reproducible researchusing Guix, whichrelied on an April 2020 revision of Guix to deploy (Simon Tournierwho co-authored the paper reportedearlieron a failed attempt showing just how challenging it was).

It’s good news that we can now re-deploy these 5-year-old softwareenvironments with minimum hassle; it’s bad news that holding thispromise took extra effort.

The ability to reproduce the environment of software that accompaniesresearch work should not be considered a mundanity or an exercise that’s“overkill”.The ability to rerun, inspect, and modify software are the naturalextension of the scientific method. Without a companion reproduciblesoftware environment, research papers are merely the advertisement ofscholarship, to paraphrase Jon Claerbout.

The futureThe astute reader surely noticed that we didn’t answer question #1above:

How can we tell which packages needs to be “fixed”, and how: buildingat a specific date, on a specific CPU?

It’s a fact that Guix so far lacks information about the date, kernel,or CPU model that should be used to build a given package.Derivationspurposefully lack that information on the grounds that it cannot beenforced in user land and is rarely necessary—which is true, but“rarely” is not the same as “never”, as we saw. Should we create acatalog of date, CPU, and/or kernel annotations for packages found inpast revisions? Should we define, for the long-term, anall-encompassing derivation format? If we did and effectively requiredvirtual build machines, what would that mean from abootstrappingstandpoint?

Here’s another option: build packages in VMs running in the year 2100,say, and on a baseline CPU. We don’t need to require all users to setup a virtual build machine—that would be impractical. It may be enoughto set up the project build farms so they build everything that way.This would allow us to catch time traps and year 2038bugs before they bite.

Before we can do that, the virtual-build-machine service needs to beoptimized. Right now, offloading to build VMs is as heavyweight asoffloading to a separate physical build machine: data is transferredback and forth over SSH over TCP/IP. The first step will be to run SSHover a paravirtualized transport instead such as AF_VSOCKsockets.Another avenue would be to make /gnu/store in the guest VM an overlayover the host store so that inputs do not need to be transferred andcopied.

Until then, happy software (re)deployment!

AcknowledgmentsThanks to Simon Tournier for insightful comments on a previous versionof this post.