Repository navigation
cannot add new host to cluster #8310
Description
- Bug Report
Cloudstack-agent
version 4.18.1.0
Im using shapeblue repository.
servers are running latest version of Ubuntu.
Im trying to add a new host but it fails, on the management server there is not error or failure.
in the agent.log I got the following.
2023-12-05 16:03:32,884 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-12-05 16:03:32,888 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.1.0
2023-12-05 16:03:32,890 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-12-05 16:03:32,942 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2023-12-05 16:03:32,942 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2023-12-05 16:03:32,994 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2023-12-05 16:03:32,995 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2023-12-05 16:03:33,094 INFO [cloud.agent.Agent] (main:null) (logid:) id is
2023-12-05 16:03:33,108 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2023-12-05 16:03:33,165 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks
2023-12-05 16:03:33,170 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2023-12-05 16:03:33,299 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1076)
at com.cloud.agent.Agent.(Agent.java:190)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:452)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
at com.cloud.agent.AgentShell.start(AgentShell.java:511)
at com.cloud.agent.AgentShell.main(AgentShell.java:541)
EXPECTED RESULTS
the host to be added to the cluster.
ACTUAL RESULTS
in the management GUI when im trying to add the host its not giving an error nor does it give a confirmation.
Activity
boring-cyborg commentedon Dec 5, 2023
Thanks for opening your first issue here! Be sure to follow the issue template!
rajujith commentedon Dec 6, 2023
@shulemmosko have you verified all the dependencies for KVM are installed and libvirtd is running? Which distribution and version you are using?
shulemmosko commentedon Dec 6, 2023
@rajujith
Yes, as far as I know, all the dependencies are running.
Ubuntu Focal fully up-to-date
and using the latest release of Cloudstack 4.18.1.0
The following line of the log looks to me as the issue, but I'm not sure.
[kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks
Thank you!
shulemmosko commentedon Dec 7, 2023
So for now i ended up downgrading the agent to 4.17.2.0
so I'm running the management on 4.18.1.0 and agents on 4.17.2.0
this combination is working fine now.
rajujith commentedon Dec 7, 2023
@shulemmosko mismatching versions of the management server and agent could give you other issues.
shulemmosko commentedon Dec 7, 2023
@rajujith the issue is that i cant downgrade the management without restoring the previous database that was running under 4.17
weizhouapache commentedon Dec 8, 2023
@shulemmosko
this is just a warning message, no issues found caused by it.
weizhouapache commentedon Dec 8, 2023
@shulemmosko
it looks libvirt does not work
can you check if
virsh listcommand works ?shulemmosko commentedon Dec 8, 2023
yes virsh list works fine, and im actually running vm's on it.
shulemmosko commentedon Dec 8, 2023
@weizhouapache @rajujith
i just tried upgrading one host again to 4.18 and it does not connect. this time I'm not getting the libvirt error
2023-12-08 14:27:20,667 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2023-12-08 14:27:20,671 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.18.1.0
2023-12-08 14:27:20,673 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2023-12-08 14:27:20,724 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2023-12-08 14:27:20,725 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2023-12-08 14:27:20,775 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2023-12-08 14:27:20,776 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2023-12-08 14:27:20,871 INFO [cloud.agent.Agent] (main:null) (logid:) id is 1220
2023-12-08 14:27:20,885 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2023-12-08 14:27:20,940 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) Failed to find passphrase for keystore: cloud.jks
2023-12-08 14:27:20,944 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2023-12-08 14:27:21,211 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) IO uring driver for Qemu: disabled
2023-12-08 14:27:21,296 INFO [kvm.storage.KVMStoragePoolManager] (main:null) (logid:) adding storage adaptor for com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor
2023-12-08 14:27:21,300 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent:
com.cloud.utils.exception.CloudRuntimeException: java.lang.reflect.InvocationTargetException
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.(KVMStoragePoolManager.java:119)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1151)
at com.cloud.agent.Agent.(Agent.java:190)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:452)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:431)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:415)
at com.cloud.agent.AgentShell.start(AgentShell.java:511)
at com.cloud.agent.AgentShell.main(AgentShell.java:541)
only the "failed to find passphrase for keystore"
shulemmosko commentedon Dec 8, 2023
@weizhouapache @rajujith
The Passphrase is in the agent.properties file and the cloud.jks is in /etc/cloudstack/agent/
weizhouapache commentedon Dec 8, 2023
@shulemmosko
this is not the root cause, you can ignore it.
it has been fixed in #7923 in upcoming 4.19, but there is no issue caused by it.
shulemmosko commentedon Dec 8, 2023
I see, so how can i troubleshoot the connection between the agent and management?
this is how it looks in the UI.
Thanks!
weizhouapache commentedon Dec 8, 2023
@shulemmosko
can you list the files below
shulemmosko commentedon Dec 8, 2023
@weizhouapache @rajujith
18 remaining items
weizhouapache commentedon Feb 6, 2024
@rp-
there are some users facing the issue below
which seems to be caused by the change in #6682
there is no issue before this change.
the workaround is (1) remove linstor jar ; or (2) create symblink for /usr/bin/hostname
@rp- can you please have a look ?
rp- commentedon Feb 7, 2024
I'm not sure if there was a specific reason to provide the full path here.
But it also seems all modern distributions have
hostnameat/usr/bin/hostnamenow.Anyway I can change it back to
"hostname"weizhouapache commentedon Feb 7, 2024
@rp- thanks
I agree with that most OS have the file, but this issue has been reported by users 2-3 times already.
it looks the file is missing in some version/release of
hostnamepackage.yadvr commentedon Feb 8, 2024
@shulemmosko were you able to consider and try some of the suggestions?
linstor: use relative hostname path
linstor: use relative hostname path (#8633)
weizhouapache commentedon Feb 9, 2024
fixed by #8633
linstor: use relative hostname path (apache#8633)
linstor: use relative hostname path (apache#8633)
linstor: use relative hostname path (apache#8633)