Over the next two months, all SecureDrops will see their underlying Ubuntu operating system upgraded from version 20.04 (code-named “Focal”) to 24.04 (“Noble”). This is the first time we’ve performed an in-place upgrade. This blog post explains the technical background and process that led to this. Key information regarding the upgrade for SecureDrop administrators and journalists will be made available in our documentation.
Background
Since its first version, SecureDrop has run on the Ubuntu Server operating system. Ubuntu provides security fixes and updates for each long-term support release for five years; SecureDrop currently uses Ubuntu 20.04 “Focal.” As of May 2025, that version will no longer be supported, so all servers must migrate to Ubuntu 24.04 “Noble,” which will be supported until May 2029.
Historically this upgrade has involved taking a backup of your SecureDrop, reinstalling the servers using the newer Ubuntu version, reinstalling SecureDrop itself, and then restoring the backup. From a development perspective, this was reliable, as it reused the same robust logic as with routine OS upgrades.
However, this put the burden on SecureDrop administrators to go through a manual process. For this round of upgrades, we wanted to see if we could spare administrators the extra work and automate the upgrade process.
Here be dragons
Strictly speaking, Ubuntu does not support upgrading from 20.04 to 24.04 — you need to first upgrade from 20.04 to 22.04, then to 24.04. Doing two upgrades instead of one would be even more unappealing.
This requirement is due in part to the underlying mechanics of package migrations. If a package “foo” wants to move its data folder from /var/data/foo1 to /var/data/foo2, the 22.04 version of the package will contain code to make the move. The 24.04 version of the package doesn’t need to keep that migration code around, since all users should’ve already migrated. So if you jump from 20.04 to 24.04 directly, you’ll miss the migration.
In addition, Ubuntu also has its own updater; you run the do-release-upgrade command, which can apply other fixes that aren’t contained in packages.
One advantage we have is that SecureDrops are relatively homogeneous. They are all installed in the same way and run the same set of software. We actively discourage administrators from making unnecessary modifications, since untracked modifications could impede further updates or even compromise the security of the system.
Would an automated upgrade be feasible? We decided to try it. We simply changed /etc/apt/sources.list to replace all instances of “focal” with “noble”, and then ran sudo apt-get update && sudo apt-get upgrade. Unsurprisingly, it didn’t exactly work on the first attempt.
Getting over the wall
One of the biggest changes we encountered was around the software firewall. After the first upgrade test, the firewall was down because it had been entirely uninstalled! The default Ubuntu firewall, ufw, now explicitly conflicted with our preferred firewall, iptables-persistent. We solved this by uninstalling ufw before the upgrade.
Once that was resolved, we now had a firewall, but it didn’t contain any rules. Under the hood, iptables switched to using Linux’s newer nftables subsystem, so we needed to enable those modules in our kernel builds.
Even then, our firewall rules were being lost in the upgrade. We believe this is caused by the files being moved from the iptables-persistent package to the netfilter-persistent one, which likely would not have happened if we didn’t jump long-term support releases. But we didn’t fully investigate the root cause as we figured out a simple workaround: back up the firewall rules before the upgrade and then restore them afterward.
The remaining issues we stumbled upon were generally simpler to resolve, but still too many to describe them all here. If you’re interested, you can look through all the pull requests tagged “noble.”
Look before you leap
While we could verify that a server set up today was upgradable, that doesn’t guarantee that existing servers are in the same condition. By design, SecureDrop developers have zero access to running servers, so we can’t (for example) SSH in to check if there is enough free disk space.
As part of last year’s SecureDrop 2.11.0 release, we shipped a script that automatically checks a number of things to ensure the instance is ready for an upgrade. It verifies that pre-upgrade migrations have completed successfully, there is enough free disk space, no failed systemd units, and more.
Of course, we don’t want to fall into the TOCTOU trap, so the check runs daily and right before the upgrade happens.
Creating upgrade.rs
The upgrade script is implemented in Rust instead of Python for two main reasons. First, Python itself is upgraded as part of the OS upgrade, so any instability could leave our script unable to run. In contrast, our Rust script is a statically compiled binary that only links to libc (and if libc is broken, we have much bigger problems). The second reason is that Rust has much more explicit error handling.
The script is organized as a giant state machine. Based on which state was just completed, it knows what to run next. The state is saved on disk each time it’s updated, allowing for the script to pick up right where it left off after reboots or other interruptions.
Each state has a corresponding function with documentation (example) to make it easy to understand what is happening.
Like all of our other processes, the script is run as a systemd service, triggered by a timer. Combined with the fact it was shipped as a Debian package, we ran into an odd scenario in which the package containing the script was upgraded by apt, so it instructed systemd to restart all of its services, which in turn killed the script and apt mid-upgrade.
We adopted a similar solution to what unattended-upgrades does: 1) spawn apt in a separate process group, and 2) set KillMode=process in the systemd service so apt will stay alive. We also set RefuseManualStop, just in case people try to manually stop the upgrade unit.
Coordination
We want to make sure we’re available to support our users through any major upgrades or milestones, so we opted for a phased rollout.
First, we’re giving administrators three weeks from the SecureDrop 2.12.0 release, to manually initiate the upgrade. This has a few advantages in that we can better suppress unnecessary notifications, but more importantly, administrators can plan out when exactly the upgrade will happen and allocate time just in case something goes wrong.
If administrators don’t do a manual upgrade during that window, our automatic update process will kick in. When upgrading to SecureDrop 2.12.0, each instance will randomly generate a number from one to five, and save it as that server’s bucket. On our end, we’ll ship a package with a JSON file that instructs the upgrade script which buckets should be upgraded at that time.
To move through the different buckets, we’ll ship new versions of the package with an updated JSON file. This allows us to upgrade roughly only 20% of servers at a time, without needing to actively select or decide which servers get upgraded.
The application server does publicly expose which operating system it’s running (example API response), so we can observe the progress of those upgrades. However, the monitor server intentionally is not publicly accessible, so we won’t have visibility into monitor server upgrades.
Final thoughts
Ubuntu 24.04 “Noble” will be the fifth version of Ubuntu that SecureDrop has used and we anticipate it will be the last. As research and development continues on the future SecureDrop Protocol, we expect it will be built on top of a different operating system.
Thank you to all of the contributors to Ubuntu, who made it a stable platform we could build on top of for over a decade.