Software development is a social process. What might be a “bug” forsomeone might well be a “feature” for someone else. The Guix projectrediscovered it the hard way when, after “fixing a bug” that had beenpresent in Guix System for years, it was confronted with an uproar in itsuser base.

In this post we look at why developers considered the initial behavior a“bug”, why users on the contrary had come to rely on it, and whydevelopers remained blind to it. A patch to reinstate the initialbehavior is being reviewed. Thispost is also an opportunity for us Guix developers to extend ourapologies to our users whose workflow was disrupted.

The crux of the matter

Anyone who’s used Guix System in the past has seen this message on theconsole during the boot process:

error in finalization thread: Success

The following picture shows a typical boot screen (with additionalmessages in the same vein):

If you have never seen it before, it may look surprising to you. GuixSystem users lived with it literally for years; the message became ahint that the boot process was, indeed, successful.

A few months ago, a contributor sought to satisfy their curiosity byfinding the origin of the message. It did look like a spurious errormessage, after all, and perhaps the right course of action would be toaddress the problem at its root—or so they thought.

As it turns out, the message originated inGuile—checkout the Guilemanualif you’re curious about finalization. Investigation revealed twothings: first, that this perror call in Guile was presumably reportingthe wrong error code—this wasfixed.

The second error—the core of the problem—lied in Guix System itself.Remember that, in its quest of memory safety™, statelessness, and fun,Guix System does it all in Guile Scheme—well, except for the kernel (fornow). As soon as Linux has booted, Guix System spawns Guile to run bootcode that’s in its initial RAMdisk(“initrd”). Right before executingshepherd, its service manager, asPID 1, the initrd code would carelessly close all the file descriptorsabove 2 to make sure they do not leak into PID 1. The problem—youguessed it—is that one of them was the now-famous file descriptor of thefinalization thread’s pipe; the finalization thread would quickly noticeand boom!

error in finalization thread: Success

Our intrepid developers thought: “hey, we found it! Let’s fix it!”. Andso theydid.

Breaking user workflows

This could have been the end of the story, but there’s more to it thansoftware. As Xkcd famously captured, this wasbound to break someone’s workflow. Indeed, had developers paid moreattention to what users had to say, they would have known that thestatus quo was preferable.

For some time now, users had shown that they held the error/successmessage deep in their heart. The message was seen on the blackboard atthe Ten Years of Guix celebration, as amotto, as a rallying cry, spontaneously put on display:

What’s more, a fellow NixOS hacker and Guix enthusiast, beguiled by thispowerful message, designedstickers and brought them to FOSDEM in February 2023:

The sticker design builds upon the “test pilot” graphics made by LuisFelipe for the 1.3.0release.The test pilot has a bug on its helmet. In a way, the drawing and errormessage both represent, metaphorically, a core tenet of Guix as aproject; just like Haskell is avoiding success at all costs, Guixseems trapped in an error/success quantum state.

Had it gone too far? Was calling it a “bug” the demonstration of thearrogance of developers detached from the reality of the community?

Fixing our mistakes

Those who installed Guix System starting from version1.4.0 havebeen missing out on the error/success boot message. The patchsubmitted today finally reinstatesthat message. The review process will determine whether consensus is toenable it by default—as part of%base-service—orwhether to make it optional—after all, we also need to accommodate theneeds of new users who never saw this message. This will allow usersto restore their workflow, while also ensuring that those freshlyprinted stickers remain relevant.

This incident had broader consequences in the project. It led some tosuggest that we, finally, set up a request-for-comment (RFC) kind ofprocess that would give all the community a say on important topics—aprocess most large free software projects have developed in one form oranother. Such a process could have prevented this incident: instead ofarrogantly labeling it as a “bug”, developers would have proposed an RFCto remove the message; the discussion period, most likely, would havemade it clear that removal was not a desirable outcome and we would allhave moved on.

This incident made many users uncomfortable, but we are glad that it isnow being addressed. The lessons learned will be beneficial to theproject for the years to come.

Credits

Testpilotby Luis Felipe distributed under the terms ofCC-BY-SA 4.0;sticker design distributed underCC-BY-SA 4.0 aswell. Blackboard picture by Julien Lepiller underCC0; stickerpictures underCC0.

Many thanks to the anonymous sticker provider!

About GNU Guix

GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.