Guix is a handy tool for developers; guix shell,in particular, gives a standalone development environment for yourpackage, no matter what language(s) it’s written in. To benefit fromit, you have to initially write a package definition and have it eitherin Guix proper, in a channel, or directly upstream as a guix.scm file.This last option is appealing: all developers have to do to get set upis clone the project's repository and run guix shell, with noarguments—we looked at the rationale for guix shell in an earlierarticle.
Development needs go beyond development environments though. How candevelopers perform continuous integration of their code in Guix buildenvironments? How can they deliver their code straight to adventuroususers? This post describes a set of files developers can addto their repository to set up Guix-based developmentenvironments, continuous integration, and continuous delivery—all atonce.
How do we go about “Guixifying” a repository? The first step, as we’veseen, will be to add a guix.scm at the root of the repository inquestion. We’ll take Guileas an example in this post: it’s written in Scheme (mostly) and C, andhas a number of dependencies—a C compilation tool chain, C libraries,Autoconf and its friends, LaTeX, and so on. The resulting guix.scmlooks like the usual packagedefinition,just without the define-public bit:
;; The ‘guix.scm’ file for Guile, for use by ‘guix shell’.(use-modules (guix) (guix build-system gnu) ((guix licenses) #:prefix license:) (gnu packages autotools) (gnu packages base) (gnu packages bash) (gnu packages bdw-gc) (gnu packages compression) (gnu packages flex) (gnu packages gdb) (gnu packages gettext) (gnu packages gperf) (gnu packages libffi) (gnu packages libunistring) (gnu packages linux) (gnu packages pkg-config) (gnu packages readline) (gnu packages tex) (gnu packages texinfo) (gnu packages version-control))(package (name "guile") (version "3.0.99-git") ;funky version number (source #f) ;no source (build-system gnu-build-system) (native-inputs (append (list autoconf automake libtool gnu-gettext flex texinfo texlive-base ;for "make pdf" texlive-epsf gperf git gdb strace readline lzip pkg-config) ;; When cross-compiling, a native version of Guile itself is ;; needed. (if (%current-target-system) (list this-package) '()))) (inputs (list libffi bash-minimal)) (propagated-inputs (list libunistring libgc)) (native-search-paths (list (search-path-specification (variable "GUILE\_LOAD\_PATH") (files '("share/guile/site/3.0"))) (search-path-specification (variable "GUILE\_LOAD\_COMPILED\_PATH") (files '("lib/guile/3.0/site-ccache"))))) (synopsis "Scheme implementation intended especially for extensions") (description "Guile is the GNU Ubiquitous Intelligent Language for Extensions,and it's actually a full-blown Scheme implementation!") (home-page "https://www.gnu.org/software/guile/") (license license:lgpl3+))Quite a bit of boilerplate, but now someone who’d like to hack on Guilejust needs to run:
guix shellThat gives them a shell containing all the dependencies of Guile: thoselisted above, but also implicit dependencies such as the GCC toolchain, GNU Make, sed, grep, and so on. The chef’s recommendation:
guix shell --container --link-profileThat gives a shell in an isolated container, and all the dependenciesshow up in $HOME/.guix-profile, which plays well with caches such asconfig.cacheand absolute file names recorded in generated Makefiles and the likes.The fact that the shell runs in a container brings peace of mind:nothing but the current directory and Guile’s dependencies is visibleinside the container; nothing from the system can possibly interferewith your development.
Now that we have a package definition, why not also take advantage of itso we can build Guile with Guix? We had left the source field empty,because guix shell above only cares about the inputs of ourpackage—so it can set up the development environment—not about thepackage itself.
To build the package with Guix, we’ll need to fill out the sourcefield, along these lines:
(use-modules (guix) (guix git-download) ;for ‘git-predicate’ …)(define vcs-file? ;; Return true if the given file is under version control. (or (git-predicate (current-source-directory)) (const #t))) ;not in a Git checkout(package (name "guile") (version "3.0.99-git") ;funky version number (source (local-file "." "guile-checkout" #:recursive? #t #:select? vcs-file?)) …)Here’s what we changed:
(guix git-download) to our set of imported modules, so wecan use its git-predicate procedure.vcs-file? as a procedure that returns true when passeda file that is under version control. For good measure, we add afallback case for when we’re not in a Git checkout: always returntrue.source to alocal-file—arecursive copy of the current directory ("."), limited to filesunder version control (the #:select? bit).From there on, our guix.scm file serves a second purpose: it lets usbuild the software with Guix. The whole point of building with Guix isthat it’s a “clean” build—you can be sure nothing from your working treeor system interferes with the build result—and it lets you test avariety of things. First, you can do a plain native build:
guix build -f guix.scmBut you can also build for another system (possibly after setting upoffloadingor transparentemulation):
guix build -f guix.scm -s aarch64-linux -s riscv64-linux… or cross-compile:
guix build -f guix.scm --target=x86\_64-w64-mingw32You can also use package transformationoptionsto test package variants:
# What if we built with Clang instead of GCC?guix build -f guix.scm \ --with-c-toolchain=guile@3.0.99-git=clang-toolchain# What about that under-tested configure flag?guix build -f guix.scm \ --with-configure-flag=guile@3.0.99-git=--disable-networkingHandy!
We now have a Git repository containing (among other things) a packagedefinition. Can’t we turn it into achannel?After all, channels are designed to ship package definitions to users,and that’s exactly what we’re doing with our guix.scm.
Turns out we can indeed turn it into a channel, but with one caveat: wemust create a separate directory for the .scm file(s) of our channelso that guix pull doesn’t load unrelated .scm files whensomeone pulls the channel—and in Guile, there are lots of them! Sowe’ll start like this, keeping a top-level guix.scm symlink for thesake of guix shell:
mkdir -p .guix/modulesmv guix.scm .guix/modules/guile-package.scmln -s .guix/modules/guile-package.scm guix.scmTo make it usable as part of a channel, weneed to turn our guix.scm file into amodule:we do that by changing the use-modules form at the top to adefine-module form. We also need to actually export a packagevariable, with define-public, while still returning the package valueat the end of the file so we can still use guix shell and guix build -f guix.scm. The end result looks like this (not repeating things thathaven’t changed):
(define-module (guile-package) #:use-module (guix) #:use-module (guix git-download) ;for ‘git-predicate’ …)(define-public guile (package (name "guile") (version "3.0.99-git") ;funky version number …));; Return the package object define above at the end of the module.guileWe need one last thing: a .guix-channelfileso Guix knows where to look for package modules in our repository:
;; This file lets us present this repo as a Guix channel.(channel (version 0) (directory ".guix/modules")) ;look for package modules under .guix/modules/To recap, we now have these files:
.├── .guix-channel├── guix.scm → .guix/modules/guile-package.scm└── .guix └── modules └── guile-package.scmAnd that’s it: we have a channel! (We could do better and supportchannelauthenticationso users know they’re pulling genuine code. We’ll spare you the detailshere but it’s worth considering!) Users can pull from this channel byadding it to~/.config/guix/channels.scm,along these lines:
(append (list (channel (name 'guile) (url "https://git.savannah.gnu.org/git/guile.git") (branch "main"))) %default-channels)After running guix pull, we can see the new package:
$ guix describeGeneration 264 May 26 2023 16:00:35 (current) guile 36fd2b4 repository URL: https://git.savannah.gnu.org/git/guile.git branch: main commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc guix c5bc698 repository URL: https://git.savannah.gnu.org/git/guix.git commit: c5bc698e8922d78ed85989985cc2ceb034de2f23$ guix package -A ^guile$guile 3.0.99-git out,debug guile-package.scm:51:4guile 3.0.9 out,debug gnu/packages/guile.scm:317:2guile 2.2.7 out,debug gnu/packages/guile.scm:258:2guile 2.2.4 out,debug gnu/packages/guile.scm:304:2guile 2.0.14 out,debug gnu/packages/guile.scm:148:2guile 1.8.8 out gnu/packages/guile.scm:77:2$ guix build guile@3.0.99-git[…]/gnu/store/axnzbl89yz7ld78bmx72vpqp802dwsar-guile-3.0.99-git-debug/gnu/store/r34gsij7f0glg2fbakcmmk0zn4v62s5w-guile-3.0.99-gitThat’s how, as a developer, you get your software delivered directly intothe hands of users! No intermediaries, yet no loss of transparency andprovenance tracking.
With that in place, it also becomes trivial for anyone to create Dockerimages, Deb/RPM packages, or a plain tarball with guix pack:
# How about a Docker image of our Guile snapshot?guix pack -f docker -S /bin=bin guile@3.0.99-git# And a relocatable RPM?guix pack -f rpm -R -S /bin=bin guile@3.0.99-gitWe now have an actual channel, but it contains only one package. Whilewe’re at it, we can define packagevariantsin our guile-package.scm file, variants that we want to be able totest as Guile developers—similar to what we did above withtransformation options. We can add them like so:
;; This is the ‘.guix/modules/guile-package.scm’ file.(define-module (guile-package) …)(define-public guile …)(define (package-with-configure-flags p flags) "Return P with FLAGS as addition 'configure' flags." (package/inherit p (arguments (substitute-keyword-arguments (package-arguments p) ((#:configure-flags original-flags #~(list)) #~(append #$original-flags #$flags))))))(define-public guile-without-threads (package (inherit (package-with-configure-flags guile #~(list "--without-threads"))) (name "guile-without-threads")))(define-public guile-without-networking (package (inherit (package-with-configure-flags guile #~(list "--disable-networking"))) (name "guile-without-networking")));; Return the package object defined above at the end of the module.guileWe can build these variants as regular packages once we’ve pulled thechannel. Alternatively, from a checkout of Guile, we can run a command likethis one from the top level:
guix build -L $PWD/.guix/modules guile-without-threadsThis channel becomes even more interesting once we set up continuousintegration (CI).There are several ways to do that.
You can use one of the mainstream continuous integration tools, such asGitLab-CI. To do that, you need to make sure you run jobs in a Dockerimage or virtual machine that has Guix installed. If we were to do thatin the case of Guile, we’d have a job that runs a shell command likethis one:
guix build -L $PWD/.guix/modules guile@3.0.99-gitDoing this works great and has the advantage of being easy to achieve onyour favorite CI platform.
That said, you’ll really get the most of it by usingCuirass, a CI tool designed for andtightly integrated with Guix. Using it is more work than using a hostedCI tool because you first need to set it up, but that setup phase isgreatly simplified if you use its Guix Systemservice.Going back to our example, we give Cuirass a spec file that goes likethis:
;; Cuirass spec file to build all the packages of the ‘guile’ channel.(list (specification (name "guile") (build '(channels guile)) (channels (append (list (channel (name 'guile) (url "https://git.savannah.gnu.org/git/guile.git") (branch "main"))) %default-channels))))It differs from what you’d do with other CI tools in two important ways:
guile and guix.Indeed, our own guile package depends on many packages provided bythe guix channel—GCC, the GNU libc, libffi, and so on. Changes topackages from the guix channel can potentially influence ourguile build and this is something we’d like to see as soon aspossible as Guile developers.guile channel transparently get pre-builtbinaries!From a developer’s viewpoint, the end result is this statuspage listing evaluations: eachevaluation is a combination of commits of the guix and guilechannels providing a number of jobs—one job per package defined inguile-package.scm times the number of target architectures.
As for substitutes, they come for free! As an example, since ourguile jobset is built on ci.guix.gnu.org, which runs guix publishin addition to Cuirass, one automatically gets substitutes for guilebuilds from ci.guix.gnu.org; no additional work is needed for that.
The Cuirass spec above is convenient: it builds every package in ourchannel, which includes a few variants. However, this might beinsufficiently expressive in some cases: one might want specificcross-compilation jobs, transformations, Docker images, RPM/Debpackages, or even system tests.
To achieve that, you can write amanifest.The one we have for Guile has entries for the package variants wedefined above, as well as additional variants and cross builds:
;; This is ‘.guix/manifest.scm’.(use-modules (guix) (guix profiles) (guile-package)) ;import our own package module(define* (package->manifest-entry* package system #:key target) "Return a manifest entry for PACKAGE on SYSTEM, optionally cross-compiled toTARGET." (manifest-entry (inherit (package->manifest-entry package)) (name (string-append (package-name package) "." system (if target (string-append "." target) ""))) (item (with-parameters ((%current-system system) (%current-target-system target)) package))))(define native-builds (manifest (append (map (lambda (system) (package->manifest-entry* guile system)) '("x86\_64-linux" "i686-linux" "aarch64-linux" "armhf-linux" "powerpc64le-linux")) (map (lambda (guile) (package->manifest-entry* guile "x86\_64-linux")) (cons (package (inherit (package-with-c-toolchain guile `(("clang-toolchain" ,(specification->package "clang-toolchain"))))) (name "guile-clang")) (list guile-without-threads guile-without-networking guile-debug guile-strict-typing))))))(define cross-builds (manifest (map (lambda (target) (package->manifest-entry* guile "x86\_64-linux" #:target target)) '("i586-pc-gnu" "aarch64-linux-gnu" "riscv64-linux-gnu" "i686-w64-mingw32" "x86\_64-linux-gnu"))))(concatenate-manifests (list native-builds cross-builds))We won’t go into the details of this manifest; suffice to say that itprovides additional flexibility. We now need to tell Cuirass to buildthis manifest, which is done with a spec slightly different from theprevious one:
;; Cuirass spec file to build all the packages of the ‘guile’ channel.(list (specification (name "guile") (build '(manifest ".guix/manifest.scm")) (channels (append (list (channel (name 'guile) (url "https://git.savannah.gnu.org/git/guile.git") (branch "main"))) %default-channels))))We changed the (build …) part of the spec to '(manifest ".guix/manifest.scm") so that it would pick our manifest, and that’sit!
We picked Guile as the running example in this post and you can see theresult here:
.guix-channel;.guix/modules/guile-package.scmwith the top-level guix.scm symlink;.guix/manifest.scm.These days, repositories are commonly peppered with dot files forvarious tools: .envrc, .gitlab-ci.yml, .github/workflows,Dockerfile, .buildpacks, Aptfile, requirements.txt, and whatnot.It may sound like we’re proposing a bunch of additional files, but infact those files are expressive enough to supersede most or all ofthose listed above.
With a couple of files, we get support for:
guix shell);guix build);guix pack).At the Guix headquarters, we’re quite happy about the result. We’vebeen building a unified tool set for reproducible software deployment;this is an illustration of how you as a developer can benefitfrom it!
Thanks to Attila Lendvai, Brian Cully, and Ricardo Wurmus for providingfeedback on an earlier draft of this post.
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.