Repository navigation
The cloudstack-agent can not work 4.18.1.0 #8223
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
- 4.18.1.0
CLOUDSTACK VERSION
- Cloudstack management (fresh install)
- OS: Ubuntu 22.04
- Cloudstack-management: 4.18.1.0
- Cloudstack agent (fresh install)
- OS: Ubuntu 22.04
- Cloudstack-agent: 4.18.1.0
SUMMARY
After installing cloudstack-agent it won't start.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Stopped CloudStack Agent.
░░ Subject: A stop job for unit cloudstack-agent.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit cloudstack-agent.service has finished.
░░
░░ The job identifier is 28655 and the job result is done.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.244s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Started CloudStack Agent.
░░ Subject: A start job for unit cloudstack-agent.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit cloudstack-agent.service has finished successfully.
░░
░░ The job identifier is 28655.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell).
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN Please initialize the log4j system properly.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO [cloud.agent.AgentShell] (main:) (logid:) Agent started
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO [cloud.agent.AgentShell] (main:) (logid:) Implementation Version is 4.18.1.0
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO [cloud.agent.AgentShell] (main:) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: Unable to find the guid
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service has successfully entered the 'dead' state.
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.239s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.
Error log from cloudstack-management server
2023-11-13 00:34:31,480 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:823)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy200.discoverHosts(Unknown Source)
at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException: can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
... 60 more
2023-11-13 00:34:31,482 INFO [c.c.a.ApiServer] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
Activity
boring-cyborg commentedon Nov 12, 2023
Thanks for opening your first issue here! Be sure to follow the issue template!
rajujith commentedon Nov 13, 2023
@clustspace could you verify that the KVM host has the required dependencies installed like libvirt ,KVM etc? If thats good, is this KVM added to Cloudstack? try adding it.
clustspace commentedon Nov 13, 2023
libvirt-daemon-system is already the newest version (8.0.0-1ubuntu7.7).weizhouapache commentedon Nov 13, 2023
@clustspace
this could be caused by some configurations (e.g. network) on the kvm host.
can you upload
clustspace commentedon Nov 13, 2023
@weizhouapache there is no setup.log on kvm host
Log from management-server.log in first post
weizhouapache commentedon Nov 13, 2023
@clustspace
can you enable DEBUG log level on management server and retry to add the host, it will give more details ?
sed -i "s/INFO/DEBUG/g" /etc/cloudstack/management/log4j-cloud.xml
You need to restart management server to make it effective.
clustspace commentedon Nov 13, 2023
@weizhouapache same error
First of all, we can't start cloudstack-agent on host server
tampler commentedon Nov 13, 2023
Having exactly the same error on the same setup.
ACS 4.18.1.0 on Ubuntu 22.04.3.For me this happens for a non-root user (
admin). Here's my setup for avisudolist:Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.0.20 at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178) at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321) at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)PS: Seems like an
agentmiscofing on my side. That's what I see in the agent log:Caused by: java.io.IOException: SSL Handshake failed while connecting to host: acs port: 8250 at com.cloud.utils.nio.NioClient.init(NioClient.java:67) at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)tampler commentedon Nov 13, 2023
Looks like #6716 is back
weizhouapache commentedon Nov 13, 2023
@tampler
It looks they are different.
Although the error logs show that there are exception "Failed to setup keystore", the root cause seems not to be related to keystore, but others like network configurations, privileges, etc.
It would be good to grant the non-root user stronger privileges, if you do not use root when add a host.
tampler commentedon Nov 19, 2023
There are few caveats in installing KVM on Ubuntu 22.04. Those are not yet covered in the project documentation.
One is an ability for an
agenthost to register itself within themgmthost, which is done somewhat counter intuitively. To resolve your issue, do this:libvirtconfigs, following the docagenthost:sudo nano /etc/cloudstack/agent/agent.propertiesand set proper values for UUID, host and bridgescloudstackgeneratedssh keyto theauthorized_keyson amgmthost:libvirtandagentand make sure they started properly:mgmtservice UI config. When you get to adding the host, specify forrootuser and click theSSH Public Key Accessto login with the autogenerated key.NOTE: You'll probably face another errors, which you would report here or open another issue 😏😏😏
tampler commentedon Nov 21, 2023
That's a config issue for Ubuntu 22.04. Related to #8218
weizhouapache commentedon Dec 1, 2023
yes @tampler
If you do not want to use root password, ssh key is a good alternative. it just works.
If you use a non-root user,
/etc/sudoersshould be configured. I suspect the following configuration is correct for non-root users.https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#install-and-configure-the-agent
1 remaining item
weizhouapache commentedon Dec 1, 2023
yes, please @tampler
it looks like a valid improvement .
yadvr commentedon Feb 8, 2024
@clustspace were you able to follow some suggestions and resolve your issue?
clustspace commentedon Feb 8, 2024
No, without results....we install 4.17 version
weizhouapache commentedon Feb 9, 2024
I wonder if the issue is same as #8310 and related to #8633
@clustspace can you check if
/usr/bin/hostnameexists ?yadvr commentedon Apr 30, 2024
@clustspace can you try again with 4.18.2.0 or 4.19.0.1 release ? Here's some notes for install on Ubuntu https://rohityadav.cloud/blog/cloudstack-kvm/
mfreeman451 commentedon Jan 1, 2025
having the same issue with non-root user on 24.04, fix for me was to re-configure my hosts to allow root login and adopt hosts that way.