OverviewIt’s been an absolutely manic week in the Linux security community as the newsand reaction to the recent announcement of a backdoor in the xz-utils projectwas announced late last week, so we dive deep into this issue and discuss how itimpacts Ubuntu and give some insights for what this means for the open sourceand Linux communities in the future.

This week in Ubuntu Security Updates20 unique CVEs addressed

[USN-6718-2] curl vulnerability* 1 CVEs addressed in Xenial ESM (16.04 ESM), Bionic ESM (18.04 ESM) + CVE-2024-2398

[USN-6719-1] util-linux vulnerability* 1 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS), Mantic (23.10) + CVE-2024-28085

[USN-6686-5] Linux kernel (Intel IoTG) vulnerabilities* 9 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS) + CVE-2024-0607 + CVE-2024-0340 + CVE-2023-6121 + CVE-2023-51782 + CVE-2023-51779 + CVE-2023-46862 + CVE-2023-46343 + CVE-2023-4134 + CVE-2023-22995

[USN-6715-1] unixODBC vulnerability* 1 CVEs addressed in Xenial ESM (16.04 ESM), Bionic ESM (18.04 ESM), Focal (20.04 LTS), Jammy (22.04 LTS), Mantic (23.10) + CVE-2024-1013

[USN-6704-4] Linux kernel (Intel IoTG) vulnerabilities* 5 CVEs addressed in Focal (20.04 LTS), Jammy (22.04 LTS) + CVE-2024-24855 + CVE-2024-1086 + CVE-2024-1085 + CVE-2023-32247 + CVE-2023-23000

[USN-6707-4] Linux kernel (Azure) vulnerabilities* 4 CVEs addressed in Jammy (22.04 LTS) + CVE-2024-26599 + CVE-2024-26597 + CVE-2024-1086 + CVE-2024-1085

[USN-6720-1] Cacti vulnerability* 1 CVEs addressed in Jammy (22.04 LTS) + CVE-2023-39361

Goings on in Ubuntu Security Communityxz-utils backdoor and Ubuntu* https://www.openwall.com/lists/oss-security/2024/03/29/4 * Late last week, 28th / 29th March backdoor in liblzma from xz-utils wasdisclosed to the open source community via oss-security mailing list - thiswas in the recent 5.6.0/5.6.1 releases from late Feb/early March this year * initially the impact was not entirely clear - assumed initially that it mayimpact only xz-utils and so only in handling of xz compressed data / files -but even with that assumption that perhaps it could then infect anything thatgot compressed/decompressed * within a few hours though became clear that the primary target was notxz-utils/liblzma itself but openssh - and the affect was to provide a backdoorinto openssh that would allow the attacker to get remote access to any machinerunning ssh server with this backdoor liblzma installed * To paint a picture - this was all unfolding on late Thursday / Friday of theEaster break, so lots of folks were either EOD or on leave etc - and trying tograpple with a threat that we knew could possibly impact the impedending 24.04LTS release * Good news: * TL;DR for Ubuntu, this version was only ever in the -proposed pocket for thecurrently in-development 24.04 release - not in any other Ubuntu versions -and was removed as soon as we became aware, so unless you are running thedevel release AND you had manually opted to install this version from the-proposed pocket, you would not be affected - very lucky * https://discourse.ubuntu.com/t/xz-liblzma-security-update/43714 * What do we know about this? A lot - there has been significant investigation(and speculation) since it was announced, both at the social side of thingsand the technical aspects of the backdoor itself * For the purpose of the podcast, we won’t go too deep into either but will tryand cover the salient details * Regarding the inclusion of the backdoor itself - looks to have been a verylong and patient campaign by the attacker, who slowly gained the trust of theupstream project over the last 2 years and likely pressured the maintainer viasock-puppet accounts to then get themselves added as an additional maintainer * Since then they seemed to be quite a good maintainer themselves - diligentlyadding new features and bug fixes etc over the past 2 years, but then suddenlyintroduced the backdoor into the most recent 2 releases * The method of introducing the backdoor was also interesting, in that itrequired 2 parts - the binary containing the backdoor and the code to get thiscompiled into the liblzma library at build time + The binary was committed into the upstream git repo disguised as an xzarchive itself used as part of the test suite + The code to inject this into the build was NOT part of the git repo, butinstead was just in the tarball prepared by the maintainer for the officialrelease + And it used many levels of obfuscation to hide this backdoor within thatfake test xz archive * So the attacker was not just patient but also very technically skilled - andnot just multiple levels of obfuscation in the build process but the backdoorcode itself contained many elements to try and make it harder to recognise andreverse engineer, presumably to allow it to hide in plain sight + although as we will see, this runtime obfuscation within the backdoor binarywas what gave it away eventually * It’s often said that one of the advantages of Open Source is the hugecommunity, which is summarised as Linus’ Law - with enough eyeballs all bugsare shallow - but sadly this wasn’t proven out in this case * Backdoor was notfound by anyone doing review of the changes upstream or by the various distroslike Debian / Fedora / OpenSUSE / Arch or even Ubuntu when incorporating thisnew version into their repos - but instead was found by Andres Freund, one ofthe maintainers of PostgreSQL, when they were looking to benchmark some newchanges scheduled for the next PostgreSQL release * Luckily decided to use Debian unstable for this, and Debian had incorporatedthis new version into unstable a few weeks ago, and wanted to get theperformance noise floor of the system as low as possible before doingbenchmarking of PostgreSQL - noticed large transient CPU spikes in sshd andthen eventually weird memory errors in sshd due to bugs in the initial versionof the backdoor * After a lot of painstaking work was able to determine that liblzma was theculprit and appeared to contain some very strange code to hook into theauthentication process of sshd when it was launched via systemd * Was able to trace that back to the aforementioned manually prepared tarballsof xz-utils on Github * The CPU spikes Andres observed were due to the use of things like a trie tolookup symbol names at runtime, rather than directly encoding them in theexploit binary, presumably to try and make reverse engineering of the binaryharder (since you can’t just run strings on it and get any real sensible output) * Some excellent writeups have been done regarding both the technical aspects ofthe backdoor itself, as well as the process taken by the attacker toincorporate this into the xz-utils project - both from a community point ofview and a technical point of view * From a technical point of view, the impact of this backdoor was to allow anattacker to get pre-authentication remote-code execution in sshd via aspecific private key when connecting to the server - NOBUS + Hooked into the RSA certificate validation process in sshd, looking for aparticular matching private key from the client - and if found would thenproceed to execute arbitrary commands specified by the client withoutrequiring usual authentication + By using this mechanism, nobody but the attacker can use the backdoor sincethey don’t have the matching private key - so the impact of the backdoor issomewhat limited + BUT the fact they targeted such a widely used and deployed package across ahuge number of distros, means they essentially wanted a backdoor into anyLinux machine in the future that only they could use * Interesting to try and speculate who the attacker could be (nation state?) andwhat their intended purpose was especially given this wide reaching goal ofgetting this into all the major Linux distros - but it would be justspeculation * So rather than speculate, for the purpose of this podcast episode, interestingto look at the timeline as it concerned Ubuntu * The publishing history of the package is all visible in Launchpad * Packages in Ubuntu get inherited from Debian who also publish history * Upstream published the first backdoored version 5.6.0 via GitHub tarball on 24th Feb 2024 * Debian incorporated this into unstable on 26th Feb * On 27th Feb, the Ubuntu Archive Auto-Sync bot copied this version into noble-proposed * Due to the ongoing time_t transition, sat it noble-proposed for the next month * Security team heard whispers of the possible backdoor just hours before it waspublicly disclosed, and as soon as we heard of the possible backdoor, andrealised that it only affected the version in-development noble-proposed wequickly notified the Archive Admin team who then deleted it fromnoble-proposed on 29th March, neutralising the main threat * Most important thing to know for Ubuntu, we have taken a very conservativeapproach - not only have we removed this version from noble-proposed as soonas we became aware, we are then rebuilding every binary package that got builtsince that compromised version was in noble-proposed originally - out of anabundance of caution - we don’t have any information that says this backdoorwas doing anything other than what the various writeups have found so far, BUTwe also can’t be certain that it didn’t have other functionality either - sobeing very cautious and rebuilding everything that was itself built since 27thFeb * As such, delayed the development of 24.04 so beta release is slipping by one week * One of the most interesting parts is the sheer luck that this was found - notby security researchers or maintainers but by a developer from Microsoft whohappened to be looking for the right things at the right time and decided tobe curious * Also for Ubuntu, luck that the time_t transition in Debian/Ubuntu caused manypackages to be stuck in noble-proposed and not in the release pocket, elsemany Ubuntu users and developers would have been impacted if this had migratedto the noble release pocket * Also interesting that the attacker appears to have had quite a good grasp onOSS development practices and was quite persistent in trying to get thisincorporated into distros - even urging for this new version to be synced toUbuntu so that it would land in the upcoming noble release as recently asThursday last week, just hours before the public disclosure + Not only could they do all the original social engineering work upstream,and technical work to develop and hide the backdoor, but could theninterface with distros via their established practices to try and get themto incorporate their new backdoored version faster than they may haveotherwise * Huge amount of work has been done to detail both the timeline of the attack aswell as the technical details of both the code used to incorporate thebackdoored code into the final liblzma binary during the build process, aswell as the details of the backdoor itself and how it operates at runtime * In the end highlights both the challenges and strengths of OSS - lots of OSSprojects have long dependency chains - in this case openssh when integratedwith libsystemd which in turn used liblzma - and it is unclear who themaintainers and authors are or what procedures are in place for vetting andtransferring ownership of OSS projects - all present significant challengesfor OSS * However, significant strength of OSS is the visibility and ability for anyoneto get involved, which is what we saw in the aftermath - despite all theadvanced obfuscation techniques employed was able to be analysed in a matterof days by the community working together - and to analyse it in a huge amountof depth and in such an open way that it leaves little room for questioningthe validity of the assessment - anyone can double check the work and come tothe same conclusions * This isn’t the first software supply chain attack and likely isn’t even thefirst against an OSS project but it is a wake-up call to the OSS and Linuxecosystem

Get in contact* security@ubuntu.com * #ubuntu-security on the Libera.Chat IRC network * ubuntu-hardened mailing list * Security section on discourse.ubuntu.com * @ubuntusecurity@fosstodon.org, @ubuntu_sec on twitter