Skip to content

client: exclude jdk15on BouncyCastle artifacts from the shaded jar - #14353

Closed
weizhouapache wants to merge 3 commits into
apache:mainfrom
weizhouapache:24-client-exclude-bc-jdk15on
Closed

weizhouapache wants to merge 3 commits into
apache:mainfrom
weizhouapache:24-client-exclude-bc-jdk15on

Conversation

@weizhouapache

Copy link
Copy Markdown
Member

Description

The client shade config only excluded the jdk18on BouncyCastle artifacts, which are shipped as separate jars in the management server lib folder. If a stale or transitive dependency pulls in bcprov/bcpkix/bctls-jdk15on, the old classes are bundled into the fat jar and can shadow bcprov-jdk18on depending on the classpath order of lib/*, failing with NoSuchFieldError (e.g. xmss_SHAKE128_512ph) when the root CA module starts.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

The client shade config only excluded the jdk18on BouncyCastle artifacts,
which are shipped as separate jars in the management server lib folder.
If a stale or transitive dependency pulls in bcprov/bcpkix/bctls-jdk15on,
the old classes are bundled into the fat jar and can shadow
bcprov-jdk18on depending on the classpath order of lib/*, failing with
NoSuchFieldError (e.g. xmss_SHAKE128_512ph) when the root CA module starts.
@weizhouapache

Copy link
Copy Markdown
Member Author

I faced the issue below on ubuntu 24/26

2026-10-07 16:52:24,463 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:[]) (logid:) configuring bean RootCAProvider.
2026-10-07 16:52:24,478 DEBUG [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Generating root CA public/private keys
2026-10-07 16:52:27,123 ERROR [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Failed to load user-provided CA certificate. Check that ca.plugin.root.ca.certificate is set and in PEM format. Overwriting with auto-generated certificate.
2026-10-07 16:52:27,123 DEBUG [o.a.c.c.p.RootCAProvider] (main:[]) (logid:) Generating root CA certificate
2026-10-07 16:52:27,192 ERROR [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Failed to load module [root-ca] due to: org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is java.lang.NoSuchFieldError: xmss_SHAKE128_512ph

Quick workaround on the management server: put the BouncyCastle jars first in CLASSPATH in /etc/default/cloudstack-management:
CLASSPATH="/usr/share/cloudstack-management/lib/bcprov-jdk18on-1.83.jar:/usr/share/cloudstack-management/lib/bcpkix-jdk18on-1.83.jar:/usr/share/cloudstack-management/lib/bctls-jdk18on-1.83.jar:/usr/share/cloudstack-management/lib/*:..."

The issue has been reported by @RosiKyu in the comment #14033 (comment)

This issue happens rarely, I did not face it before, but faced it 3 times yesterday. (related to #12794 )
cc @vishesh92

@codecov

codecov Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 19.91%. Comparing base (ed1db53) to head (0b340c5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #14353      +/-   ##
============================================
- Coverage     19.91%   19.91%   -0.01%     
+ Complexity    20200    20199       -1     
============================================
  Files          6373     6373              
  Lines        577230   577230              
  Branches      70696    70696              
============================================
- Hits         114958   114953       -5     
- Misses       449703   449711       +8     
+ Partials      12569    12566       -3     
Flag Coverage Δ
uitests 3.71% <ø> (ø)
unittests 21.18% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vishesh92
vishesh92 requested a lite review from Copilot October 8, 2026 07:46
@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The reviewed shading configuration change addresses the artifact conflict with no unresolved issues.

0 open findings

What changed in this PR

Updates the client shaded JAR configuration to prevent legacy BouncyCastle JDK15 artifacts from shadowing JDK18 artifacts.

Changes:

  • Excludes bcprov, bcpkix, bctls, and bcutil JDK15 artifacts from shading.
File Description
client/​pom.xml Adds BouncyCastle JDK15 artifact exclusions.

🧠 Review effort: Lite


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19466

@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan test ubuntu24 kvm-ubuntu24

@weizhouapache

Copy link
Copy Markdown
Member Author

Vishesh has found this is caused by netris-java-sdk update. good finding!
cc @Pearl1594 @nvazquez

@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan test ubuntu24 kvm-ubuntu24

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu24 mgmt + kvm-ubuntu24) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17098)

@weizhouapache

Copy link
Copy Markdown
Member Author

this PR does not fix the issue

root@pr14353-t17098-kvm-ubuntu24-mgmt1:~# grep root-ca /var/log/cloudstack/management/management-server.log 
2026-10-08 09:40:47,335 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy:           root-ca
2026-10-08 09:41:30,401 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Module Hierarchy:           root-ca
2026-10-08 09:43:58,069 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [root-ca] context.
2026-10-08 09:43:58,069 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to load module [root-ca] context.
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/root-ca/spring-root-ca-context.xml]
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/ca/spring-core-lifecycle-ca-context-inheritable.xml]
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/core/spring-core-lifecycle-core-context-inheritable.xml]
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/system/spring-core-system-context-inheritable.xml]
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/system/spring-framework-config-system-context-inheritable.xml]
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/system/spring-contrail-system-context-inheritable.xml]
2026-10-08 09:43:58,070 INFO  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Loading module context [root-ca] from URL [jar:file:/usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar!/META-INF/cloudstack/bootstrap/spring-bootstrap-context-inheritable.xml]
2026-10-08 09:44:03,371 ERROR [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Failed to load module [root-ca] due to: org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is java.lang.NoSuchFieldError: xmss_SHAKE128_512ph

and

root@pr14353-t17098-kvm-ubuntu24-mgmt1:~# unzip -l /usr/share/cloudstack-management/lib/cloudstack-24.0.0-shapeblue20439.jar |grep -c "org/bouncycastle"
6295

The netris-java-sdk 1.0.0 artifact was republished without a version bump
and now declares bcprov-jdk15on and bctls-jdk15on 1.70. These were shaded
into the management server jar next to the bcprov/bcpkix/bctls-jdk18on 1.83
jars in lib/, and the old classes could shadow the new ones depending on
the classpath order, failing with NoSuchFieldError (xmss_SHAKE128_512ph)
when the root CA module starts.

The SDK still gets BouncyCastle (BouncyCastleProvider and
BouncyCastleJsseProvider) from the managed jdk18on 1.83 artifacts.
@weizhouapache
weizhouapache deleted the 24-client-exclude-bc-jdk15on branch October 8, 2026 10:25
@weizhouapache
weizhouapache restored the 24-client-exclude-bc-jdk15on branch October 8, 2026 10:25
@weizhouapache weizhouapache reopened this Oct 8, 2026
@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19470

@weizhouapache

Copy link
Copy Markdown
Member Author

@blueorangutan test ubuntu24 kvm-ubuntu24

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu24 mgmt + kvm-ubuntu24) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17100)

@weizhouapache

Copy link
Copy Markdown
Member Author

this PR is not required when bump the netris-java-sdk to 1.1.0

closing

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-17102)

@nvazquez

nvazquez commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Hi @weizhouapache @vishesh92 please check #14365

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants