Repository navigation
Management server hangs when js.interpretation.enabled=true #12523
Description
problem
Enabling the global setting js.interpretation.enabled=true causes the CloudStack management server to hang during startup at the module loading phase. This prevents using the secondary storage selector feature (createSecondaryStorageSelector, updateSecondaryStorageSelector APIs).
versions
- 4.20 branch (4.20.3.0-SNAPSHOT) - AFFECTED - needs fix before 4.20.3.0 release
- 4.20.2.0 (released) - Not affected (commit not present, setting does not exist, APIs work unconditionally)
Introduced by commit 03a4b9f ("server,utils: improve js interpretation functionality") merged on Oct 16, 2025 into the 4.20 branch.
The commit:
- Added
js.interpretation.enabledas a hidden ConfigKey (default: false) - Made
CreateSecondaryStorageSelectorCmdandUpdateSecondaryStorageSelectorCmdconditional on this setting - Made significant changes to
JsInterpreter.java(security hardening, daemon workers, etc.)
When the setting is enabled, something in the initialization causes the MS server to hang.
The steps to reproduce the bug
- Deploy CloudStack 4.20.3.0
- Verify server is running:
curl -s "http://localhost:8080/client/api?command=listZones" | head -5
- Enable the js.interpretation.enabled setting:
mysql -u root -p cloud -e "UPDATE configuration SET value='true' WHERE name='js.interpretation.enabled';"
- Restart management server:
systemctl restart cloudstack-management
- Wait 3+ minutes and check server status:
curl -s "http://localhost:8080/client/api?command=listZones" | head -5
Expected Result
Server starts successfully and createSecondaryStorageSelector API is available.
Actual Result
Server returns 503 Service Unavailable. Log shows server stuck at module loading phase:
2026-01-26 19:53:01,937 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [redfish] context.
2026-01-26 19:53:01,937 WARN [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Application context not found for module definition [redfish]
Log timestamp remains static while current time progresses - server is hung.
Evidence
4.20 branch build (shapeblue18197) - Bug present:
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# rpm -qa | grep cloudstack
cloudstack-common-4.20.3.0-shapeblue18197.noarch
cloudstack-management-4.20.3.0-shapeblue18197.noarch
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# mysql -u root -p cloud -e "UPDATE configuration SET value='true' WHERE name='js.interpretation.enabled';"
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# systemctl restart cloudstack-management
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# sleep 180
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# curl -s "http://localhost:8080/client/api?command=listZones" | head -5
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 Service Unavailable</title>
</head>
4.20.2.0 released version (shapeblue17631) - Works correctly:
[root@ref-trl-10714-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# rpm -qa | grep cloudstack
cloudstack-common-4.20.2.0-shapeblue17631.noarch
cloudstack-management-4.20.2.0-shapeblue17631.noarch
mysql> SELECT name, value FROM configuration WHERE name LIKE '%interpret%';
Empty set (0.00 sec)
[root@ref-trl-10714-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list apis 2>/dev/null | grep -i "name.*selector"
"name": "createSecondaryStorageSelector",
"name": "updateSecondaryStorageSelector",
"name": "listSecondaryStorageSelectors",
"name": "removeSecondaryStorageSelector",
Impact
- Secondary storage selector feature is unusable on current 4.20 branch
- Quota tariff activation rules may also be affected (uses same JsInterpreter)
- Blocks testing of PR Consider secondary storage selectors during cold volume migration #10957 and any other functionality that depends on this feature
Related
- Commit: 03a4b9f
- Blocks PR: Consider secondary storage selectors during cold volume migration #10957
What to do about it?
No response
Activity
RosiKyu commentedon Jan 26, 2026
Update
The root cause has been identified. The
js.interpretation.enabledsetting requires an encrypted value in the database, not plain text.Workaround:
cat /etc/cloudstack/management/keyjava -classpath /usr/share/cloudstack-common/lib/cloudstack-utils.jar com.cloud.utils.crypt.EncryptionCLI -p <key> -i trueUPDATE configuration SET value='<encrypted>' WHERE name='js.interpretation.enabled';Remaining issues:
Could be closed as user error, but suggesting improvement to error handling.
DaanHoogland commentedon Jan 27, 2026
@winterhazel , any thoughts?
winterhazel commentedon Jan 28, 2026
@DaanHoogland regarding the startup failure, it is the behavior I find correct for this situation. There should also be a clear message in the logs informing that there was an issue while decrypting this configuration's value, so that operators know what to look into.
I have some thoughts about
js.interpretation.enabledthough. I do not have access to the discussion regarding the CVE that prompted the introduction of this setting to know why it was handled that way. However, I think that it should not have been made a hidden setting with an encrypted value, and should be enabled by default. The vulnerability was fixed as far as I am aware; also, the APIs that allow configuring scripts (host, Quota tariff, and secondary storage selector configuration) should only be accessible to people with access to the infrastructure. Hence, if a new vulnerability with the interpreter gets discovered and exploited, that is not an issue with the platform, but internal permission granting issues. Other features that come enabled by default may have vulnerabilities that we are not aware of yet, but that's not a reason for we to disable them by default. Having it as a hidden encrypted setting just makes it unnecessarily difficult for operators to use the features.DaanHoogland commentedon Jan 28, 2026
in short @winterhazel , you would agree with ^ this? Should be a simple PR. I’ll put it on my list
winterhazel commentedon Feb 2, 2026
@DaanHoogland yes, I agree with @RosiKyu's suggestions in 1 and 3.
2 not so much as I think that it should not be hidden at all.
shwstppr commentedon Feb 2, 2026
@winterhazel @RosiKyu cc @DaanHoogland The idea behind keeping some settings hidden is to not allow root admin accounts to change them. Only a privileged user having access to DB would be able to change them.
Incidentally all hidden settings are encrypted currently which can be documented.
winterhazel commentedon Feb 4, 2026
@shwstppr ok, but my point is that
js.interpretation.enabledshould not be kept hidden. Is there a reason for restricting root admin accounts, which can already do anything they want with all resources in the environment as long as they receive API permission, from enabling a feature that only they will be able to use?shwstppr commentedon Feb 5, 2026
@winterhazel while in most cases ROOT admin will have access to the system as well (underlying server), but there can be cases when ROOT admin is just the CloudStack admin. In those cases, a non-hidden config can be changed by this CloudStack admin and the system can be under security risk as highlighted by the CVE for which this setting was introduced. There could be better means to alter such configs, but for no,w ACS provides only the hidden configs
winterhazel commentedon Feb 5, 2026
@shwstppr as I commented in #12523 (comment): the known vulnerabilities pertaining the JS were patched, weren't they? Other features that come enabled by default almost certainly have vulnerabilities that we are not aware of yet, but we cannot put them behind a hidden setting due to a hypothetical security issue. If we were consistent with how the JS interpretation was handled, then we would have to also disable by default and place behind a hidden setting features like direct download, volume/template upload and register, which had CVEs in the past, and all other functionalities that uses administrator/user input in a shell/python script.
The extraconfig feature could also be used to abuse the environment. Following your logic, root admins should not be able to see/change this configuration, yet the configurations related to this feature are not hidden.
shwstppr commentedon Feb 5, 2026
@winterhazel it was not my logic alone. It was a PMC decision. I was just explaining the case. If you're not happy with it please raise with it.
Also, you may create a change PR to make it a regular config and get it merged ✌️
GutoVeronezi commentedon Feb 5, 2026
@shwstppr @winterhazel
Reviewing the private discussion, I see that the fix started with the config as hidden, and it was mentioned that we could discuss it, but we never had the actual discussion about whether it should or should not be hidden (perhaps because of the urgency to release the fixes or lack of time/attention; water under the bridge). Now that we no longer have such urgency, it is a good opportunity to discuss whether it makes sense to change it or keep it as it is.
@winterhazel, if you will, create a PR with your proposal so we can move forward with the discussion.
3 remaining items
DaanHoogland commentedon Mar 3, 2026
@RosiKyu I read your comments and then the discussions following. What in you opinion is to be done (in order of priority)?
@winterhazel same question to you..
winterhazel commentedon Apr 10, 2026
@DaanHoogland I think that the startup failure when an encrypted setting has a decrypted value in the database is the correct behavior. This indicates to operators that there is something wrong that they need to check. We just need to improve the logging when this failure happens. Maybe something like:
We expect the value of setting '<name>' to be encrypted in the database with the Management Server's key, but we were unable to decrypt it using this key. This issue may happen when the value was manually changed in the database to a plain decrypted value. For reference on how to change the value of encrypted settings through the database, see <URL to the docuementation>.'github-actions commentedon Jul 2, 2026
🎯 Triage report
When
js.interpretation.enabledis set to a plain-texttruevalue in the database (rather than the required encrypted form), the management server hangs silently during startup with no actionable error message. The root cause is user configuration error — the setting is a hidden encrypted key that must be set viaEncryptionCLI— but the actual bug is that the server hangs indefinitely instead of failing fast with a clear diagnostic. Community consensus is to improve error handling for misconfigured encrypted settings.📊 Assessment
type:bugcomponent:management-serverSeverity:Minortype:bug,component:management-server,Severity:Minor💡 Notes and suggestions
Workaround (documented in comments):
cat /etc/cloudstack/management/keyjava -classpath /usr/share/cloudstack-common/lib/cloudstack-utils.jar com.cloud.utils.crypt.EncryptionCLI -p <key> -i trueFix suggestion: In the configuration loading/decryption code, catch
EncryptionException/ decryption failure and emit a log message such as:Then fail fast with a clear startup error rather than hanging.
Related: PR #12605 proposes making
js.interpretation.enableda regular (non-hidden) configuration.Add this agentic workflows to your repo
To install this agentic workflow, run
scottsignal commentedon Sep 9, 2026
This makes it really painful to enable js.interpretation.enabled. There are legitimate reasons to use this flag.