Skip to content

Stop trusting compromised eMudhra roots in CI images - #218

Merged
jandro996 merged 3 commits into
DataDog:masterfrom
eliottness:eliottness/emudhra-ca-roots
Oct 27, 2026
Merged

jandro996 merged 3 commits into
DataDog:masterfrom
eliottness:eliottness/emudhra-ca-roots

Conversation

@eliottness

@eliottness eliottness commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Stops CI, test and build images from trusting two compromised roots: emSign ECC Root CA - G3 and AAA Certificate Services (incident IR-62343). The APPSEC-71273 ledger lists 6 impacted image references for this repository. Images that CI never builds or pulls are left alone.

Changed Dockerfiles:

  • ci/alpine-stock8/Dockerfile: base moved from alpine:3.19 to alpine:3.24.3. Installs p11-kit-trust, points the libnssckbi.so* NSS modules at the p11-kit trust module, and removes the NSS test copy libnssckbi-testlib.so*.
  • ci/alpine-temurin21/Dockerfile and ci/alpine-temurin25/Dockerfile: base moved from alpine:3.19 to alpine:3.24.3, with the same p11-kit change. After copying the JDK, any cacerts entry whose subject is either compromised root is deleted, and the build fails if one is still there. The current eclipse-temurin:21-jdk-alpine / :25-jdk-alpine tags (jdk-21.0.12.1+1, jdk-25.0.4.1+1) still ship emSign ECC G3; Temurin 21.0.13+9 and 25.0.5+8 ship neither root, so the step becomes a no-op guard once the tags move.
  • ci/alpine/Dockerfile: stays on alpine:3.11, on purpose: the musl native binaries built here must run on old musl. It gets no fixed ca-certificates package, so both compromised roots are removed in place: disabled in /etc/ca-certificates.conf, their .crt files deleted, update-ca-certificates run, and /etc/ssl/cert.pem linked to the bundle. openjdk8 pulls in nss, whose builtin roots still include both, so after installing it the libnssckbi.so* modules are pointed at p11-kit-trust and libnssckbi-testlib.so* is removed.

Motivation

IR-62343 / Jira APPSEC-71286 (parent APPSEC-71273). These are the roots behind the compromised eMudhra intermediates, so every CI image that trusts them keeps the exposure.

Description of the Change

The new image reference alpine:3.24.3 was pulled and scanned with the eMudhra scanner, which checks every file in the image for the roots' public keys; it returns no hits. Packages installed on top (nss via openjdk8) bring their own NSS builtin roots, which the p11-kit redirect and testlib removal handle. The changed Dockerfiles were not built locally; PR CI builds them (see Additional Notes).

Of the 6 impacted references: 2 are fixed here (alpine:3.19 and alpine:3.11), 0 are fixed at their in-repo image source (clean once the image job republishes), and 4 are follow-ups below.

Follow-ups (not in this PR):

  • centos:7: NSS 3.131.1 is unavailable for EOL CentOS 7, and the blocklist attempt still scanned positive in NSS and OS/Java stores, so the edit was restored at the stock8 target in actions.yml:520.
  • centos:6: NSS 3.131.1 is unavailable for EOL CentOS 6, the base scan hits AAA, and the full build fails with curl exit 139 under amd64 emulation on arm64 hosts, so the stock8 target in actions.yml:530 keeps its current base.
  • quay.io/pypa/manylinux2014_aarch64@sha256:d4cb8ad04184: no clean tag exists in this line, since the 2024-02-04 image and the newest 2026.10.22-0 both hit NSS, system bundles and vendored certifi, so actions.yml:259 keeps this glibc aarch64 baseline.
  • quay.io/pypa/manylinux2010_x86_64@sha256:3b5eb5ab9bc7: the 2022-08-05 tag still resolves to the impacted digest and the supported manylinux_2_34_x86_64@sha256:c8f6ffd1ef26 still hits OS/certifi stores, so actions.yml:251 keeps this glibc x86_64 baseline.

Additional Notes

  • The earlier description called alpine-stock8 "patched in place (no clean version to move to)". The diff moves it to alpine:3.24.3 like the other two Alpine JDK images, so this description follows the diff.
  • CI builds each changed Dockerfile: the Test matrix runs docker build -t <image> ci/<image> (actions.yml:606), and the Native_binaries_Stage_linux musl jobs run docker build ci/alpine (actions.yml:282).

Jira: APPSEC-71286 (parent APPSEC-71273)

Move the JDK test images from alpine 3.19 to alpine 3.24.3, route NSS through p11-kit, and drop the emSign root from the
copied Temurin JDKs while they still ship it. The musl native-binary image stays on alpine 3.11 for musl compatibility and
has both roots removed in place (IR-62343). CentOS 6/7 and manylinux images are follow-ups.

Jira: APPSEC-71286
@eliottness
eliottness marked this pull request as ready for review October 26, 2026 13:33
@eliottness
eliottness requested a review from a team as a code owner October 26, 2026 13:33
Comment thread ci/alpine/Dockerfile Resolved
Comment thread ci/alpine-temurin21/Dockerfile Resolved
openjdk8 depends on nss, and NSS compiles its builtin roots into
/usr/lib/libnssckbi.so* and the test copy /usr/lib/libnssckbi-testlib.so*.
Both still carry emSign ECC Root CA - G3 and AAA Certificate Services.

- ci/alpine (Alpine 3.11): install p11-kit-trust after openjdk8, point the
  libnssckbi.so* modules at /usr/lib/pkcs11/p11-kit-trust.so, and remove
  libnssckbi-testlib.so*.
- ci/alpine-stock8, ci/alpine-temurin21, ci/alpine-temurin25 (Alpine 3.24.3):
  also remove libnssckbi-testlib.so*. nss 3.124-r0 in v3.24 ships
  libnssckbi-testlib.so.124, which the existing libnssckbi.so* glob misses.

Jira: APPSEC-71286
The previous step deleted one hard-coded alias and silently skipped when
the alias did not match, which would leave the root trusted if Temurin
renamed the entry.

Now, in ci/alpine-temurin21 and ci/alpine-temurin25:
- find entries whose Owner is CN=emSign ECC Root CA - G3 or
  CN=AAA Certificate Services and delete them by alias;
- list the store again and fail the build if either subject is still there.

eclipse-temurin:21-jdk-alpine (jdk-21.0.12.1+1) and :25-jdk-alpine
(jdk-25.0.4.1+1) still ship emSign ECC G3. Temurin 21.0.13+9 and 25.0.5+8
(2026-10-22) ship neither root, so the step becomes a no-op guard once the
Docker Hub tags move.

Jira: APPSEC-71286
@eliottness

Copy link
Copy Markdown
Contributor Author

@jandro996 both points are addressed (69677cd, f08ad35); replies are on the threads. Could you take another look? I could not re-request review through the API from the fork account (missing permission).

@jandro996 jandro996 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick follow-up. I rebuilt the four changed images from f08ad35 locally and scanned them: no hits for either root in a raw filesystem grep, in any keystore (OS java/cacerts, JDK 8, Temurin 21/25, which went from 118 to 117 entries) or in the OS bundle / /etc/ssl/cert.pem, and libnssckbi.so* points to p11-kit-trust in all four. LGTM.

@eliottness

Copy link
Copy Markdown
Contributor Author

Thanks for the review @jandro996! The PR is approved and every check is green (merge state CLEAN), but I do not have write access to libddwaf-java. Could you merge it when you have a moment?

@jandro996
jandro996 merged commit 194fa4f into DataDog:master Oct 27, 2026
42 checks passed
@jandro996 jandro996 mentioned this pull request Oct 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants