What We Know About The XZ Utils Backdoor Supply Chain Attack

What We Know About The XZ Utils Supply Chain Attack

TL;DR

This Friday, an engineer discovered a backdoor that was implanted in xz utils, an open-source data compression library available on almost all flavors of Linux and Unix-like operating systems. There are strong indications that the people behind the attack likely spent years on it. And they were very close to seeing the backdoor merged into Debian and Red Hat Linux, the two most wide spread distributions until it was spotted by the engineer. Some Linux flavors did distribute it for a brief period, so keep reading to see if you’re affected.

FAQ

What is xz-utils?

xz-utils is a library that provides data compression on almost all Unix-like operating systems. The library is used in critical systems for compressing and decompressing data, including in SSH connections.

How was the xz-utils backdoor discovered?

Andres Freund discovered the backdoor while troubleshooting performance issues a Debian system was having with SSH, the most common remote login protocol. To be specific, SSH logins consumed to much CPU, while also generating errors with valgrind, a utility for memory monitoring.

Eventually, he discovered the issues were caused by the updates made to xz-utils.

What does the backdoor do?

The malicious code added to version 5.6.0 and 5.6.1 allow Remote Code Execution when the lzma compression and decompression methods are used in the context of SSH.

The backdoor is triggered only when the attacker connects to a SSH server with a specific key. The command is passed via the connecting public key to the Unix system call, so it will execute with the same privileges as the SSH daemon, which is usually root.

What systems are affected?

Research at time of writing, indicates the backdoor targeted Debian and Red Hat based distributions.

Some non-enterprise Linux distributions were affected and released advisories:

• Fedora Rawhide - advisory
• Fedora 41 - advisory
• Debian testing, unstable and experimental v5.5.1alpha-0.1 to 5.6.1-1 - advisory
• openSUSE Tumbleweed and MicroOS - advisory
• Kali Linux - advisory

Is macOS affected by the xz backdoor?

MacOS seems to not be affected. However, Homebrew had hosted the tampered library versions 5.6.0 and 5.6.1, so you might have it installed. Homebrew marked those versions as untrusted and downgraded the latest available package to version 5.4.6. If brew upgrade was executed recently, you might have the tampered version installed or cached locally.

Check what version you have:

xz --version

To uninstall xz-utils run:

brew update
brew cleanup xz --prune=0

Then reinstall:

brew reinstall xz

If you encounter issues, see the homebrew discussion here.

The backdoor checks if it’s running with glibc on amd64/x86 so it wouldn’t be included in macOS because it uses BSD libc (and newer macs use the ARM platform).

xz Backdoor Technical Details

This might be the best executed supply chain attack discovered and discussed publicly.

The backdoor has several components:

• crafted test files in tests/ folder
• a build script build-to-host.m4 that unpacks malicious code and modifies the build process
• IFUNC glibc functionality is used to perform runtime hooking of OpenSSH’s authentication methods.

The build script does some checks before tampering with the build, making it suitable for specific platforms:

• verifies if the build target is amd64/x86_64
• and if the target uses linux-gnu

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1)

Additionally, it checks what is the used toolchain:

if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
exit 0
fi
if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
exit 0
fi
LDv=$LD" -v"
if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
exit 0

And if it’s a Debian/Red Hat package

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

So the code is not actually backdoored, but the resulting tarballs are, which made it hard to detect.

When running:

• when verifying a SSH public key, if it matches a certain fingerprint, the contents is decrypted using a pre-shared key and the content is passed to the system call (hence resulting in RCE), and skips verification.
• if the fingerprint doesn’t match, it falls back to the normal public key verification.
• the backdoor triggers only when the xz library is loaded by a /usr/bin/sshd process.

Social engineering aspects

Allegedly, sock puppet accounts were used to put pressure and convince Linux distributions to include the latest version of the xz library into their packages. They also put pressure to change the library maintainer to a more active one.

The threat actor demonstrated great levels of research, development, security tradecraft and patience to get this far. The GitHub changes were done over multiple years (first commit being made two years ago) and were carefully planned. For example, the actor introduced code that made it incompatible with OSS Fuzzer and succeeded in getting the xz library excluded from the fuzzer scanning last year.

Events Timeline

• 2021 - JiaT75 (Jia Tan), the backdoor author, creates GitHub account and starts making commits. A suspicious commit (now fixed) is in the libarchive library where the safe_fprint function was replaced with an unsafe variant.
• 2022, April - Jia Tan submits a patch. A new account, Jigar Kumar starts pressuring for this patch to be merged. Not a lot after, he begins pressuring Lasse Collin (the xz maintainer) to add another maintainer to xz.
• 3 days later, Jia Tan makes the first commit to xz and slowly begins being a regular contributor to xz.
• Another account - Dennis Ens - starts participating in the pressures.
• 2023, January - Jia Tan merges the first commit, seemingly obtaining a trusted status.
• 2023, March - the primary contact email in Google’s OSS Fuzzer is changed from Lasse’s to Jia Tan.
• 2023, June - The first backdoor related code is committed. Lasse Collin was assigned as the author, but it was Jia Tan who committed it and it was originally written by Hans Jansen. The same people will also apply pressure to merge the library in Debian.
• 2023, July - A Pull Request was opened to disable IFUNC for fuzzing builds to the the changes they introduced previously. Jia Tan opens a ticket drawing more attention to IFUNC.
• 2024, February - A PR in OSS Fuzzer changes the URL of the project to one owned by Jia Tan.
• 2024, March - A commit is made by Jia Tan with the final stages required to execute the backdoor.

What to do next?

This backdoor will be probably discussed for months due to the complexity, patience and social engineering aspects involved. Check your systems, support open source and stay informed as things might change as researchers dive deeper into the exploit.

Resources:

• Disclosure
• Link