Is this only happened with SSH, or other network facing services using liblzma too?
The malicious code attempts to hook in to libcrypto, so potentially other services that use libcrypto could be affected too. I don’t think extensive research has been done on this yet.
SSH doesn’t even use liblzma. It’s pulling in the malicious code via libsystemd, which does use liblzma.
Edit: “crypto” meaning cryptography of course, not cryptocurrency.
Also, even aside from the attack code here having unknown implications, the attacker made extensive commits to liblzma over quite a period of time, and added a lot of binary test files to the xz repo that were similar to the one that hid the exploit code here. He also was signing releases for some time prior to this, and could have released a signed tarball that differed from the git repository, as he did here. The 0.6.0 and 0.6.1 releases were contained to this backdoor aimed at sshd, but it’s not impossible that he could have added vulnerabilities prior to this. Xz is used during the Debian packaging process, so code he could change is active during some kind of sensitive points on a lot of systems.
It is entirely possible that this is the first vulnerability that the attacker added, and that all the prior work was to build trust. But…it’s not impossible that there were prior attacks.
Holly shit
And you know what? Doing updates once a week saved me from updating to this version :)
I upgraded to 5.6.0-1 on the 28th Februar already. Over a month ago. On a server. That’s the first time Arch testing has fucked me so hard lol.
You probably are fairly safe. Yeah, okay, from a purely-technical standpoint, your server was wide-open to the Internet. But unless some third party managed to identify and leverage the backdoor in the window between you deploying it and it being fixed, only the (probably state-backed) group who are doing this would have been able to make use of it. They probably aren’t going to risk exposing their backdoor by exploiting it on your system unless they believe that you have something that would be really valuable to them.
Maybe if you’re a critical open-source developer, grabbing your signing keys or other credentials might be useful, given that they seem to be focused on supply-chain attacks, but for most people, they probably just aren’t worth the risk. Only takes them hitting some system with an intrusion-detection system that picks up on the breakin, them leaving behind traces, and some determined person tracking down what happened, and they’ve destroyed their exploit.
Having arch has benefits because of more up to date packages but ofc as it happened to you, it introduces more risks
Reading the comments here https://news.ycombinator.com/item?id=39865810 it appears that libarchive may be tainted as well.
ELI5 what does this mean for the average Linux user? I run a few Ubuntu 22.04 systems (yeah yeah, I know, canonical schmanonical) - but they aren’t bleeding edge, so they shouldn’t exhibit this vulnerability, right?
In this case I think that’s just Fedora and Debian Sid users or so.
The backdoor only activates during DEB or RPM builds, and was quickly discovered so only rolling release distros using either package format were affected.
Not regular Fedora, though, it was only in Fedora Rawhide and Fedora 41, so very very early, bleeding edge distributions. Nothing that a regular Fedora user would be using.
https://access.redhat.com/security/cve/cve-2024-3094
E: and Fedora 40 beta which some regular users could conceivably be using
https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users
It mostly affects/targets the build systems of binary distros - infecting their build machines with this would result in complete compromise of released distro down the line.
apt info xz-utils
Your version is old as balls. Even if you were on Mantic, it would still be old as balls.
deleted by creator
This also affects Fedora 40 and Fedora Rawhide - https://discussion.fedoraproject.org/t/warning-malicious-code-in-current-pre-release-testing-versions-variants-f40-and-rawhide-affected-users-of-f40-rawhide-need-to-respond/110683
The person who found the backdoor : https://mastodon.social/@AndresFreundTec/112180083704606941
If you’re using
xz
version 5.6.0 or 5.6.1, please upgrade asap, especially if you’re using a rolling-release distro like Arch or its derivatives. Arch has rolled out the patched version a few hours ago.Dang, Arch never sleeps, does it? That’s a 24/7 incident response squad level of support.
Gentoo just reverted back to the last tar signed by another author than the one seeming responsible for the backdoor. The person has been on the project for years, so one should keep up to date and possibly revert even further back than just from 5.6.*. Gentoo just reverted to 5.4.2.
Just updated on void and saw the same thing
Backdoor only gets inserted when building RPM or DEB. So while updating frequently is a good idea, it won’t change anything for Arch users today.
No, read the link you posted:
Arch does not directly link openssh to liblzma, and thus this attack vector is not possible. You can confirm this by issuing the following command:
ldd "$(command -v sshd)"
However, out of an abundance of caution, we advise users to remove the malicious code from their system by upgrading either way.
I think that was a precaution. The malicious build script ran during the build, but the backdoor itself was most likely not included in the resuling package as it checked for specific packaging systems.
when building RPM or DEB.
Which ones? Everything I run seems to be clear.
https://access.redhat.com/security/cve/CVE-2024-3094
Products / Services Components State Enterprise Linux 6 xz Not affected Enterprise Linux 7 xz Not affected Enterprise Linux 8 xz Not affected Enterprise Linux 9 xz Not affected (and thus all the bug-for-bug clones)
I think it needs to be
- rolling release (because it was caught so quickly that it hasn’t made its way into any cadence based distro yet)
- using the upstream Makefile task to build a RPM or DEB (because the compromised build script directly checks for that and therefore doesn’t trigger for a destdir build like Gentoo’s or Arch’s)
- using the upstream provided tarball as opposed to the one GitHub provides, or a git clone (because only that contains the compromised Makefile, running autotools yourself is safe)
Points 1 and 2 mean that only rolling release RPM and DEB distros like Debian Sid and Fedora are candidates. I didn’t check if they use the Makefile and the compromised tarballs.
Fedora 41, Fedora Rawhide, Debian Sid are the currently known affected ones AFAIK.
Those getting the most recent software versions, so nothing that should be running in a server.
Some no-name came and without any problems asked to become a maintainer in a project used in almost any distro, took it over, put a backdoor in there and no one had any questions? In this case, everything turned out thanks to pure chance. Noname screwed up his backdoor, which attracted the attention of a guy from Microsoft, and out of boredom, he dug up what was what. And if I hadn’t messed up, or that guy from Microsoft decided to go drink beer instead of poking around in the xz code, then no one would have discovered anything. It’s scary to imagine how many of these nonames are sitting in all these thousands of open source projects, waiting in the wings to roll out a malicious patch.
That’s the scary thing. It looks like this narrowly missed getting into Debian and RH. Downstream downstream that is… everything.
I will laugh out loud if the “fixed” binary contains a second backdoor, but one of better quality. It’s reminiscent of a poorly hidden small joint, which is naturally found, and then bargaining, apologizing and making amends begin. Although now it is generally not clear where the code is more proven.
Since the actual operation of the liblzma SSH backdoor payload is still unknown, there’s a protocol for securing your impacted systems:
• Consider all data, including key material and secrets on the impacted system as compromised. Expand the impact to other systems, as needed (for example: if a local SSH key is used to access a remote system then the remote system must be considered impacted as well, within the scope the key provides).
• Wipe the impacted host and reinstall it from scratch. Use known good install that does not contain the malicious payload. Generate new keys and passwords. Do not reuse any from the impacted systems.
• Restore configuration and data from backups, but from before the time the malicious liblzma package was installed. However, be careful not to allow potentially leaked credentials or keys to have access to the newly installed system (for example via $HOME/.ssh/authorized_keys).
This handles the systems themselves. Unfortunately any passwords and other credentials stored, accessed or processed with the impacted systems must be considered compromised as well. Change passwords on web sites and other services as needed. Consider the fact that the attacker may have accessed the services and added ways to restore access via for example email address or phone number in their control. Check all information stored on the services for correctness.
This is a lot of work, certainly much more than just upgrading the liblzma package. This is the price you have to pay to stay safe. Just upgrading your liblzma package and hoping for the best is always an option, too. It’s up to you to decide if this is a risk worth taking.
This recovery protocol might change somewhat once the actual operation of the payload is figured out. There might be situations where the impact could be more limited.
As an example: If it turns out that the payload is fully contained and only allows unauthorized remote access via the tampered sshd, and the host is not directly accessible from the internet (the SSH port is not open to internet) this would mean that it might be possible to clean up the system locally without full reinstall.
However, do note that the information stored on the system might have still been leaked to outside world. For example leaked ssh keys without a passphrase could still afford the attacker access to remote systems.
This is a long con, and honestly the only people at fault are the bad actors themselves. Assuming Jia Tan’s GitHub identity and pgp key weren’t compromised by someone else, this backdoor appears to be the culmination of three years of work.
t y for sharing.
#showerthoughts The problem is in upstream and has only entered Debian Sid/unstable. Does this mean that for example bleeding edge Arch (btw) sshd users are compromised already ?
Looks like the 5.6.1-2 release on Arch moved from using the published GitHub releases to just using the git repository directly, which as I understand avoids the exploit (because the obfuscated script to inject the exploit is only present in the packaged tarballs and not the git repo itself)
They also believe we (Arch users) are unaffected because this backdoor targeted Debian and Redhat type packaging specifically and also relied on a certain SSH configuration Arch doesn’t use. To be honest while it’s nice to know we’re unaffected, it’s not at all comforting that had the exploiter targeted Arch they would have succeeded. Just yesterday I was talking to someone about how much I love rolling release distros and now I’m feeling insecure about it.
More details here: https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/issues/2
Someone always has to be the guinea pig.
That being said, maybe there’s an argument for distros that do rolling releases to have an “intentionally delayed rolling release” that just trails the regular rolling release by a fixed amount of time to provide more time for guinea pigs to run into things. If you want rolling, but can live with the delay, just use that.
OpenSuse Slowroll does pretty much that, a slightly delayed rolling release.
Arch has pushed the patched
xz
just a few hours ago: https://archlinux.org/news/the-xz-package-has-been-backdoored/Thanks a bunch.
Homebrew rolled back the release after finding out
Here’s a link to the PR for anyone who’s interested
Arch is on 5.6.1 as of now: https://archlinux.org/packages/core/x86_64/xz/
We at Nixpkgs have barely evaded having it go to a channel used by users and we don’t seem to be affected by the backdoor.
Arch had a patch rolled out yesterday [1][2][3] that switches to the git repo. On top of that the logic in the runtime shim and build script modifier was orchestrated to target Debian and RPM build systems and environments [4].
[2] https://gitlab.archlinux.org/archlinux/packaging/packages/xz/-/issues/2
[3] https://security.archlinux.org/CVE-2024-3094
[4] https://www.openwall.com/lists/oss-security/2024/03/29/4
Since you didn’t build a RPM or DEB package however, your didn’t compile in the backdoor.
Damn, it is actually scary that they managed to pull this off. The backdoor came from the second-largest contributor to xz too, not some random drive-by.
It would be nice if we could press formal charges
Assuming that it’s just that person, that it’s their actual name and that they’re in the US…
Do you have a source for this?
I don’t have a source but I think it is safe to say given the large corporations and government institutions that rely on XZ utils. I’m sure Microsoft, Amazon, redhat ect are in talks with the federal government about this
Source: they made it up
They’ve been contributing to xz for two years, and commited various “test” binary files.
Either that or the attacker was very good at choosing their puppet…
Well the account is focused on one particular project which makes sense if you expect to get burned at some point and don’t want all your other exploits to be detected. It looks like there was a second sock puppet account involved in the original attack vector support code.
We should certainly audit other projects for similar changes from other psudoanonymous accounts.
Yeah, and the 700 commits should be reverted… just in case we missed something.
That is a should be standard procedure