Security Advisory

Security Advisory: Misconfigured package repository servers and developer infrastructure

April 22, 2024

An internal review discovered that some servers maintained by Freedom of the Press Foundation (FPF) and used to support SecureDrop were installed in an insecure manner. We have found no evidence of compromise, but the servers have been reinstalled from scratch and relevant credentials rotated as a precautionary measure. It’s important to note that even if there was a compromise, it would not have been possible to serve malicious updates to SecureDrop instances or workstations.

No action is needed by most users; if you have taken steps to hide the physical and network location of your servers, please review the ”Possible user action needed” section below.

The servers in question are:

How did it happen?

Our servers use Debian and most software is installed from Debian packages. However, in this case, the github-webhook PyPI package had been installed using pip via Ansible, as root. By default, pip verifies the TLS connection to pypi.org, but does no other additional integrity checks such as verifying package checksums.

For a successful compromise, an attacker would have needed to either 1) man-in-the-middle our connection to PyPI or 2) upload malicious code to PyPI by hijacking a maintainer account or PyPI itself. In both cases, it would also have needed to happen in the narrow window during which the servers were being freshly installed (pip was only used at install time, not to upgrade).

We audited the installed code on the servers, as well as the code uploaded to PyPI, against the GitHub repository and found them all to match and contain no malicious code. (Of course, this does not 100% rule out a compromise, as a well-resourced attacker could have covered their tracks.)

What could an attacker do?

First, let’s start with what an attacker couldn’t have done: deliver malicious updates to SecureDrop instances or workstations. The SecureDrop signing key is stored offline with individual maintainers and not available on those servers. Without access to the signing key, an attacker could only have created malicious packages with invalid signatures, which would have been loudly rejected by SecureDrop servers and workstations. We have seen no evidence of any attempt to deliver malicious updates.

If an attacker had gained root access on the package repository servers, they would have been able to view HTTP access logs containing the IP addresses of the public internet egress for SecureDrop instances. See the “Possible user action needed” section below for more details on the implications of this.

An attacker would have been able to prevent instances from receiving SecureDrop application updates. We monitor known SecureDrop instances and have seen no evidence of blocked updates, and we have received no suspicious reports of blocked updates from instance operators.

The PGP signing key used by developers for testing automated nightlies and RC packages is stored on apt-test and yum-test repository servers and would have been accessible to an attacker who managed to compromise them. This key is only used by developers for testing automated nightlies and RC packages.

The ws-ci-runner server is used to orchestrate CI for SecureDrop Workstation that runs on Qubes OS. It contained a classic GitHub personal access token (PAT), which enabled push access (among other things) to a number of SecureDrop Git repositories.

On all servers an attacker would have been able to access TLS certificates (relatively short-lived since they’re issued by Let’s Encrypt).

How we’re addressing it

Within 48 hours of discovery, we fully rebuilt all affected package repository servers using a vendored copy of the github-webhook code. Webhook secrets (used as part of delivering the webhook payloads) were also rotated. All TLS certificates on the old servers were revoked.

The signing key for the apt-test and yum-test repositories was also rotated.

The CI server for SecureDrop Workstation was also rebuilt in a similar way, and was issued with a narrowly scoped GitHub PAT for the only API endpoint that’s needed. The previous classic PAT was revoked.

We received logs from GitHub regarding the use of the classic PAT over the past six months, which revealed no malicious usage, leaving a five-month window unaccounted for. Most commits to the SecureDrop project are PGP-signed by their authors, so we re-reviewed all unsigned commits in that timeframe to ensure no tampering occurred.

Longer-term, we are planning to simplify our package repository infrastructure to reduce both attack surface and the potential for misconfiguration. We will also request an independent assessment of supply chain security within the scope of the next SecureDrop audit.

Possible user action needed

As mentioned earlier, the package repository server’s logs contain the IP addresses of SecureDrop instances. This is not a cause for concern for the vast majority of instances, as they are self-hosted by news organizations and the IPs are not considered particularly secret.

However, instance operators taking additional steps to hide the physical and network location of their servers may consider updating their setup accordingly (keeping in mind that we have seen no indication that these logs were compromised).

Timeline

  • June-August 2022: package repository servers begin using pip-installed code
  • April 2023: “classic” GitHub PAT issued for use in SecureDrop Workstation CI
  • April 3, 2024: internal review surfaces insecure installation, classic GitHub PAT is revoked
  • April 4, 2024: apt-qa.freedom.press, apt-test.freedom.press, yum-qa.securedrop.org, yum-test.securedrop.org servers reinstalled
  • April 5, 2024: apt.freedom.press and yum.securedrop.org reinstalled
  • April 11, 2024: ws-ci-runner.securedrop.org reinstalled
  • April 16, 2024: auditing of unsigned commits completed
  • April 22, 2024: advisory published

Questions and comments

If you have questions or comments regarding this Security Advisory, please contact us:

  • Via our Support Portal, if you are a member (membership is available to SecureDrop administrators on request);
  • Via securedrop@freedom.press (PGP encrypted) for sensitive security issues (please use judiciously);
  • Via our community forums.

We also encourage you to file nonsensitive issues you encounter in our GitHub repository (issue report form).

Return to News