You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
improvement: externalize timeout for hypervisor/agent commands #8506
As discussed in #8502 (comment) it could be useful to externalize the wait timeouts for hypervisor/agent commands like ModifyStoragePoolCommand.
The downside of introducing new global configurations for each of these commands is having too many configurations so maybe a generic solution can be implemented.
I did not have time to elaborate on the proposal until now, but here is the draft.
For some edge cases, we have "wait"/"timeout" hardcoded values or values that come from global settings to set the commands timeout, like for GetAutoScaleMetricsCommand and for MigrateCommand. Currently, we have 385 classes that extend com.cloud.agent.api.Command (I skipped classes that extend com.cloud.agent.api.Answer); therefore, externalizing global settings for each command to provide flexibility is not feasible; however, we can reach flexibility by other means.
Currently, if we do not set a wait to the command, AgentManagerImpl will fallback to the wait global setting:
I propose to create a table of command_timeout in which we would store the classpath of the command and its wait/timeout. Then, instead of fall backing directly to the wait global setting, we would check if there is an entry of the command in this table. If it exists, then we use the entry's value; otherwise, we fall back to the wait global setting. Thus, we can change the commands wait/timeout on demand and in a flexible way.
The first step of the proposal would be creating the table and changing AgentManagerImpl to check the table. We could already create some entries for hardcoded values, like for the GetAutoScaleMetricsCommand.
Then, as a second step, we could create APIs to allow operators to list, persist, and remove the wait/timeouts. We could also create an annotation to describe the commands and create an API to allow operators to list the annotated commands; thus, making it clear to them which processes they can manage the timeout.
As a third and final step, we could extend this feature to be managed via GUI.
I will create a POC of this proposal; meanwhile, let me know your thoughts about the proposal.
Note: this proposal cover the common cases; eventually we will discuss the edge cases.
Following there is the code I used to retrieve the classes that extend com.cloud.agent.api.Command:
changed the title [-]improvement: externalize timeout for ModifyStoragePoolCommand and other hypervisor/agent commands[/-][+]improvement: externalize timeout for hypervisor/agent commands[/+]on Feb 6, 2024
Activity
shwstppr commentedon Feb 2, 2024
Moving to 4.19.1.0 milestone for possible work on a generic solution for externalizing command timeouts
GutoVeronezi commentedon Feb 2, 2024
Hey @DaanHoogland @vishesh92
I did not have time to elaborate on the proposal until now, but here is the draft.
For some edge cases, we have "wait"/"timeout" hardcoded values or values that come from global settings to set the commands timeout, like for
GetAutoScaleMetricsCommandand forMigrateCommand. Currently, we have 385 classes that extendcom.cloud.agent.api.Command(I skipped classes that extendcom.cloud.agent.api.Answer); therefore, externalizing global settings for each command to provide flexibility is not feasible; however, we can reach flexibility by other means.Currently, if we do not set a
waitto the command,AgentManagerImplwill fallback to thewaitglobal setting:cloudstack/engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java
Lines 441 to 443 in af8a582
I propose to create a table of
command_timeoutin which we would store the classpath of the command and its wait/timeout. Then, instead of fall backing directly to thewaitglobal setting, we would check if there is an entry of the command in this table. If it exists, then we use the entry's value; otherwise, we fall back to thewaitglobal setting. Thus, we can change the commands wait/timeout on demand and in a flexible way.The first step of the proposal would be creating the table and changing
AgentManagerImplto check the table. We could already create some entries for hardcoded values, like for theGetAutoScaleMetricsCommand.Then, as a second step, we could create APIs to allow operators to list, persist, and remove the wait/timeouts. We could also create an annotation to describe the commands and create an API to allow operators to list the annotated commands; thus, making it clear to them which processes they can manage the timeout.
As a third and final step, we could extend this feature to be managed via GUI.
I will create a POC of this proposal; meanwhile, let me know your thoughts about the proposal.
Note: this proposal cover the common cases; eventually we will discuss the edge cases.
Following there is the code I used to retrieve the classes that extend
com.cloud.agent.api.Command:To execute it, run
java -cp <path-to-cloud-client.jar> <path-to-file-with-the-above-code>.Following there are the classes retrieved by the script:
Classes that extend com.cloud.agent.api.Command
GutoVeronezi commentedon Feb 3, 2024
@DaanHoogland @vishesh92 I've created PR #8605 to address the POC I'd mentioned .
[-]improvement: externalize timeout for ModifyStoragePoolCommand and other hypervisor/agent commands[/-][+]improvement: externalize timeout for hypervisor/agent commands[/+]shwstppr commentedon Jul 8, 2025
Fixed in #9659