com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:  #6716

Activity

1. boring-cyborg commented on Sep 8, 2022

   

   boring-cyborgBot

   on Sep 8, 2022

   More actions

   Thanks for opening your first issue here! Be sure to follow the issue template!
2. yadvr commented on Sep 8, 2022

   

   yadvr

   on Sep 8, 2022

   Member

   More actions

   @bkrajendra have you allowed ssh server on the KVM host to login as root user and password (password based auth)? You can also disable ca framework all together by setting the auth strictness global setting to false and try adding the host again (or restart the cloudstack-agent if it was setup already).
3. bkrajendra commented on Sep 8, 2022

   

   bkrajendra

   on Sep 8, 2022

   Author

   More actions

   @rohityadavcloud thank you for the quick reply.
   Yes, I have access to the Host from the management server. I have tried access from the terminal and it works with password-based auth.

   I'll try auth strictness global setting and test again and update here.
4. bkrajendra commented on Sep 9, 2022

   

   bkrajendra

   on Sep 9, 2022

   Last edited by bkrajendra

   Author

   More actions

   @rohityadavcloud
   I tried the following things:

   1. set ca.plugin.root.auth.strictness=false restarted management server, restarted agent - still same error
   2. uninstalled agent on a host, removed /etc/cloudstack/agent folder - now I got properties not found error and after that again same error related to failed to setup Keystore.
      Should I clean and install the host and try again?
5. Hudratronium commented on Sep 9, 2022

   

   Hudratronium

   on Sep 9, 2022

   More actions

   @bkrajendra
   I am correct that your user for adding the host has this config in sudoers?
   cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent

   Just out of curiosity:
   What's happening when you connect via ssh to the host while using the 'cloudstack' user and run
   sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
6. bkrajendra commented on Sep 10, 2022

   

   bkrajendra

   on Sep 10, 2022

   Author

   More actions

   @Hudratronium
   After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal
   Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

   About what you said: (in case of Ubuntu 20.04)

   I am using the root user for all operations. I don't have cloudstack user.
   root ALL=(ALL) ALL
   Second, when I try keystore-setup from ssh, this is what I get:

   sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
   cat: : No such file or directory
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   
7. Lueton commented on Sep 14, 2022

   

   Lueton

   on Sep 14, 2022

   Last edited by Lueton

   More actions

   @Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

   About what you said: (in case of Ubuntu 20.04)

   I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:

   sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
   cat: : No such file or directory
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   

   same issue for me, no matter if via cloudstack, ssh or local
8. mlsorensen commented on Sep 14, 2022

   

   mlsorensen

   on Sep 14, 2022

   Last edited by mlsorensen

   Contributor

   More actions

   @Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

   About what you said: (in case of Ubuntu 20.04)

   I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:

   sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
   cat: : No such file or directory
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   chmod: missing operand after ‘600’
   Try 'chmod --help' for more information.
   

   These errors when manually running keystore-setup are likely due to not providing all of the extra flags necessary to run keystore-setup script. They point to empty variables. You can possibly try something like this to see if it works on the Ubuntu system, or perhaps exposes a missing utility or path:

   /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/cloud.jks mykeystorepassword 365 /etc/cloudstack/agent/cloud.csr
   

   As far as systemvm issues, that's separate debugging. Could be that the system image is not provisioned or it is up but the internal agent isn't reachable. Would need to find out if the VM is running and get on console, or if it isn't running find at what step it failed.

   It may not strictly be necessary to use system VMs if you opt for "direct download" VM templates and don't need VNC console, but it's worth trying to find out what is wrong.

   👍React with 👍1Reacted by gizmocoder
9. yadvr commented on Sep 14, 2022

   

   yadvr

   on Sep 14, 2022

   Member

   More actions

   @bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

   In case you still hit issues, let me know. I'm happy to update/edit the QIG and notes on my blog. cc @wido @weizhouapache any pointers if you can see, as you're one of the major Ubuntu users.
10. Lueton commented on Sep 14, 2022

    

    Lueton

    on Sep 14, 2022

    More actions

    @bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

    In case you still hit issues, let me know. I'm happy to update/edit the QIG and notes on my blog. cc @wido @weizhouapache any pointers if you can see, as you're one of the major Ubuntu users.

    I installed the general Ubuntu Server 20.04.
    I'll try your installation guide as soon as possible, thanks for sharing!
11. Hudratronium commented on Sep 14, 2022

    

    Hudratronium

    on Sep 14, 2022

    More actions

    @bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

    We were using Ubuntu Server 20.04.0 - which should be completely supported as far as i am aware.
12. weizhouapache commented on Sep 15, 2022

    

    weizhouapache

    on Sep 15, 2022

    Member

    More actions

    @Hudratronium
    yes, ubuntu 20.04 is offically fully supported in cloudstack.

    Questions

    • was the host added in other cloudstack platform before ?
    • Did you clean the host before you add it to the new platform ?
13. Lueton commented on Sep 16, 2022

    

    Lueton

    on Sep 16, 2022

    More actions

    I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

    However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.
    

    Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.
14. weizhouapache commented on Sep 16, 2022

    

    weizhouapache

    on Sep 16, 2022

    Member

    More actions

    I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

    However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.

    Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.

    @Lueton
    if you add the host using user/password, this error can be ignored.
15. Lueton commented on Sep 16, 2022

    

    Lueton

    on Sep 16, 2022

    More actions

    @weizhouapache thats good to know, i was very confused!

    this is my /agent/setup.log after trying to add this machine as a new host, maybe you can identify the issue.

    root@delta:~# cat /var/log/cloudstack/agent/setup.log
    DEBUG:root:execute:uname -r
    DEBUG:root:execute:uname -m
    DEBUG:root:execute:hostname -f
    DEBUG:root:execute:kvm-ok
    DEBUG:root:execute:awk '/MemTotal/ { printf "%.3f \n", $2/1024 }' /proc/meminfo
    DEBUG:root:execute:ip a | grep "^\w" | grep -iv "^lo" | wc -l
    DEBUG:root:execute:service apparmor status
    DEBUG:root:execute:apparmor_status |grep libvirt
    DEBUG:root:Failed to execute:
    DEBUG:root:execute:/bin/systemctl mask libvirtd.socket
    DEBUG:root:execute:/bin/systemctl mask libvirtd-ro.socket
    DEBUG:root:execute:/bin/systemctl mask libvirtd-admin.socket
    DEBUG:root:execute:/bin/systemctl mask libvirtd-tls.socket
    DEBUG:root:execute:/bin/systemctl mask libvirtd-tcp.socket
    DEBUG:root:execute:systemctl restart libvirtd
    DEBUG:root:execute:ufw allow 22
    DEBUG:root:execute:ufw allow 1798
    DEBUG:root:execute:ufw allow 16509
    DEBUG:root:execute:ufw allow 16514
    DEBUG:root:execute:ufw allow proto tcp from any to any port 5900:6100
    DEBUG:root:execute:ufw allow proto tcp from any to any port 49152:49216
    DEBUG:root:execute:sudo /usr/sbin/service ufw status
    DEBUG:root:execute:sudo /usr/sbin/service ufw stop
    DEBUG:root:execute:sudo /usr/sbin/service ufw status
    DEBUG:root:Failed to execute:● ufw.service - Uncomplicated firewall
         Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
         Active: inactive (dead) since Fri 2022-09-16 10:17:25 UTC; 23ms ago
           Docs: man:ufw(8)
        Process: 2354 ExecStop=/lib/ufw/ufw-init stop (code=exited, status=0/SUCCESS)
       Main PID: 396 (code=exited, status=0/SUCCESS)
    
    Sep 16 10:16:19 delta systemd[1]: Finished Uncomplicated firewall.
    Sep 16 10:17:25 delta systemd[1]: Stopping Uncomplicated firewall...
    Sep 16 10:17:25 delta systemd[1]: ufw.service: Succeeded.
    Sep 16 10:17:25 delta systemd[1]: Stopped Uncomplicated firewall.
    DEBUG:root:execute:sudo /usr/sbin/service ufw start
    DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
    DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
    DEBUG:root:execute:sleep 30
    DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
    DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
    DEBUG:root:Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent
    DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
    DEBUG:root:Failed to execute:● cloudstack-agent.service - CloudStack Agent
         Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
         Active: failed (Result: exit-code) since Fri 2022-09-16 10:17:28 UTC; 30s ago
           Docs: http://www.cloudstack.org/
       Main PID: 2216 (code=exited, status=143)
    
    Sep 16 10:17:23 delta systemd[1]: Started CloudStack Agent.
    Sep 16 10:17:23 delta java[2216]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
    Sep 16 10:17:23 delta java[2216]: SLF4J: Defaulting to no-operation (NOP) logger implementation
    Sep 16 10:17:23 delta java[2216]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
    Sep 16 10:17:27 delta systemd[1]: Stopping CloudStack Agent...
    Sep 16 10:17:28 delta systemd[1]: cloudstack-agent.service: Main process exited, code=exited, status=143/n/a
    Sep 16 10:17:28 delta systemd[1]: cloudstack-agent.service: Failed with result 'exit-code'.
    Sep 16 10:17:28 delta systemd[1]: Stopped CloudStack Agent.
    DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start
    
16. 9 remaining items

    Load more
17. weizhouapache

    added a commit that references this issue on Jan 13, 2023

    KVM: revert libvirtd config and retry if fail to add a host

    ...

    This issue will close once commit 18cffd1 is merged into the 'main' branch.

    18cffd1
18. weizhouapache

    linked a pull request that will close this issueKVM: revert libvirtd config and retry if fail to add a host #7090on Jan 13, 2023
19. DaanHoogland

    added this to the 4.18.0.0 milestone on Jan 13, 2023
20. weizhouapache commented on Jan 18, 2023

    

    weizhouapache

    on Jan 18, 2023

    Member

    More actions

    closed via #7090
21. DaanHoogland

    closed this as completedon Jan 18, 2023
22. weizhouapache

    moved this from 👀 Review to Wei's List in ACS Backlogon Jan 20, 2023
23. yadvr

    removed this from ACS Backlogon Mar 14, 2023
24. vishnuvs369 commented on Apr 12, 2024

    

    vishnuvs369

    on Apr 12, 2024

    Last edited by vishnuvs369

    More actions

    I want to add a kvm host on cloudstack...while adding it using CS ui getting some error...any one have an idea ??

    

    While checking cloudstack management server log...it showing

    2024-04-12 08:55:42,249 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    2024-04-12 08:55:43,301 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:
    sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
    sudo: a password is required
    
    2024-04-12 08:55:43,301 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.10.118 - Failed to setup keystore on the KVM host: 192.168.10.118
    2024-04-12 08:55:43,301 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.10.118 - Failed to setup keystore on the KVM host: 192.168.10.118
    com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.10.118
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationIn
    

    i tried to run the script getting below error...

    sudo /usr/share/cloudstack-common/scripts/util/keystore-setup

    Failed to generate CSR file, retrying after removing existing settings
    Reverting libvirtd to not listen on TLS
    Removing cloud.* files in /etc/cloudstack/agent
    Retrying to generate CSR file
    Failed to generate CSR file while retrying
    
25. weizhouapache commented on Apr 12, 2024

    

    weizhouapache

    on Apr 12, 2024

    Member

    More actions

    sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
    sudo: a password is required

    @vishnuvs369
    the error indicates that the user needs to be added to wheel group
26. vishnuvs369 commented on Apr 12, 2024

    

    vishnuvs369

    on Apr 12, 2024

    More actions

    @weizhouapache
    i have provided root as username and root user password for adding kvm host...now that error gone....now i'm getting different error.

    2024-04-12 09:34:01,056 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:Failed to generate CSR file, retrying after removing existing settings
    Reverting libvirtd to not listen on TLS
    Removing cloud.* files in /etc/cloudstack/agent
    Retrying to generate CSR file
    Failed to generate CSR file while retrying
    
    
    2024-04-12 09:34:01,056 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    2024-04-12 09:34:02,255 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-3f3b6330) (logid:f119a186) HA health check task is running...
    2024-04-12 09:34:02,885 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-23a200d4) (logid:6cdcc245) AutoScaling Monitor is running...
    2024-04-12 09:34:03,361 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:
    chmod: cannot access '/etc/cloudstack/agent/agent.properties': No such file or directory
    
    2024-04-12 09:34:03,388 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
    2024-04-12 09:34:04,431 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 09:34:04,431 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
    2024-04-12 09:34:05,254 DEBUG [c.c.a.ApiServlet] (qtp239372207-359:ctx-d1046213) (logid:b74dfb77) ===START===  192.168.10.111 -- GET  command=readyForShutdown&response=json
    2024-04-12 09:34:05,254 DEBUG [c.c.a.ApiServlet] (qtp239372207-359:ctx-d1046213) (logid:b74dfb77) Two factor authentication is already verified for the user 2, so skipping
    2024-04-12 09:34:05,261 DEBUG [c.c.a.ApiServer] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
    2024-04-12 09:34:05,263 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) Account [Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]] is Root Admin or Domain Admin, all APIs are allowed.
    2024-04-12 09:34:05,265 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [readyForShutdown] for user [User {"username":"admin","uuid":"8b9cdbeb-f746-11ee-9e73-000c29962f22"}.] as allowed.
    2024-04-12 09:34:05,266 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
    2024-04-12 09:34:05,267 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) API rate limiting is disabled. We will not use ApiRateLimitService.
    2024-04-12 09:34:05,269 DEBUG [c.c.a.ApiServlet] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) ===END===  192.168.10.111 -- GET  command=readyForShutdown&response=json
    2024-04-12 09:34:05,497 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 09:34:05,497 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
    2024-04-12 09:34:06,256 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-dcb4daff) (logid:1ac789dd) HA health check task is running...
    2024-04-12 09:34:06,562 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 09:34:06,562 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    2024-04-12 09:34:06,562 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand
    
    
27. weizhouapache commented on Apr 12, 2024

    

    weizhouapache

    on Apr 12, 2024

    Member

    More actions

    2024-04-12 09:34:01,056 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:Failed to generate CSR file, retrying after removing existing settings
    Reverting libvirtd to not listen on TLS
    Removing cloud.* files in /etc/cloudstack/agent
    Retrying to generate CSR file
    Failed to generate CSR file while retrying

    @vishnuvs369
    these certicicate errors are misleading
    please share the whole log on mgmt server and agent (agent.log, setup.log) when adding the host
28. vishnuvs369 commented on Apr 12, 2024

    

    vishnuvs369

    on Apr 12, 2024

    More actions

    @weizhouapache
    Below given full log of management server while adding an host..

    tail -f /var/log/cloudstack/management/management-server.log

    2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:Failed to generate CSR file, retrying after removing existing settings
    Reverting libvirtd to not listen on TLS
    Removing cloud.* files in /etc/cloudstack/agent
    Retrying to generate CSR file
    Failed to generate CSR file while retrying
    
    
    2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    2024-04-12 11:48:41,860 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:
    chmod: cannot access '/etc/cloudstack/agent/agent.properties': No such file or directory
    
    2024-04-12 11:48:41,886 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    2024-04-12 11:48:42,774 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-2784ff02) (logid:0eda3e49) HA health check task is running...
    2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) ===START===  192.168.10.111 -- GET  command=readyForShutdown&response=json
    2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) Two factor authentication is already verified for the user 2, so skipping
    2024-04-12 11:48:42,794 DEBUG [c.c.a.ApiServer] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
    2024-04-12 11:48:42,797 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Account [Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]] is Root Admin or Domain Admin, all APIs are allowed.
    2024-04-12 11:48:42,798 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [readyForShutdown] for user [User {"username":"admin","uuid":"8b9cdbeb-f746-11ee-9e73-000c29962f22"}.] as allowed.
    2024-04-12 11:48:42,799 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
    2024-04-12 11:48:42,800 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) API rate limiting is disabled. We will not use ApiRateLimitService.
    2024-04-12 11:48:42,802 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) ===END===  192.168.10.111 -- GET  command=readyForShutdown&response=json
    2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    2024-04-12 11:48:43,414 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-e1dfc0f1) (logid:c70e7745) VM Operation Thread Running
    2024-04-12 11:48:43,420 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) ClusterDRS.poll is being called at 2024-04-12 11:49:00 GMT
    2024-04-12 11:48:43,425 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) Removed 0 old drs migration plans
    2024-04-12 11:48:43,431 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) VM scheduler.poll is being called at 2024-04-12 11:49:00 GMT
    2024-04-12 11:48:43,434 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Got 0 scheduled jobs to be executed at 2024-04-12 11:49:00 GMT
    2024-04-12 11:48:43,436 INFO  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Cleaned up 0 VM scheduled job entries
    2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    2024-04-12 11:48:45,058 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 11:48:45,058 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    2024-04-12 11:48:45,058 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
    	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
    	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
    	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
    	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:516)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    2024-04-12 11:48:45,059 DEBUG [c.c.r.ResourceManagerImpl] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
    com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
    	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
    	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
    	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
    	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:516)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	... 60 more
    2024-04-12 11:48:45,060 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
    2024-04-12 11:48:45,060 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Exception: 
    com.cloud.exception.DiscoveryException: Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:834)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
    	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
    	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
    	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
    	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:516)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	... 59 more
    Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	... 60 more
    
    

    i could'nt see any log for setup...while checking on the path /var/log/cloudstack/agent/ is an empty directory

    
29. weizhouapache commented on Apr 12, 2024

    

    weizhouapache

    on Apr 12, 2024

    Member

    More actions

    @weizhouapache Below given full log of management server while adding an host..

    tail -f /var/log/cloudstack/management/management-server.log

    2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:Failed to generate CSR file, retrying after removing existing settings
    Reverting libvirtd to not listen on TLS
    Removing cloud.* files in /etc/cloudstack/agent
    Retrying to generate CSR file
    Failed to generate CSR file while retrying
    
    
    2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    2024-04-12 11:48:41,860 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
    SSH command output:
    chmod: cannot access '/etc/cloudstack/agent/agent.properties': No such file or directory
    
    2024-04-12 11:48:41,886 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    2024-04-12 11:48:42,774 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-2784ff02) (logid:0eda3e49) HA health check task is running...
    2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) ===START===  192.168.10.111 -- GET  command=readyForShutdown&response=json
    2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) Two factor authentication is already verified for the user 2, so skipping
    2024-04-12 11:48:42,794 DEBUG [c.c.a.ApiServer] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
    2024-04-12 11:48:42,797 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Account [Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]] is Root Admin or Domain Admin, all APIs are allowed.
    2024-04-12 11:48:42,798 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [readyForShutdown] for user [User {"username":"admin","uuid":"8b9cdbeb-f746-11ee-9e73-000c29962f22"}.] as allowed.
    2024-04-12 11:48:42,799 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
    2024-04-12 11:48:42,800 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) API rate limiting is disabled. We will not use ApiRateLimitService.
    2024-04-12 11:48:42,802 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) ===END===  192.168.10.111 -- GET  command=readyForShutdown&response=json
    2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    2024-04-12 11:48:43,414 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-e1dfc0f1) (logid:c70e7745) VM Operation Thread Running
    2024-04-12 11:48:43,420 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) ClusterDRS.poll is being called at 2024-04-12 11:49:00 GMT
    2024-04-12 11:48:43,425 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) Removed 0 old drs migration plans
    2024-04-12 11:48:43,431 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) VM scheduler.poll is being called at 2024-04-12 11:49:00 GMT
    2024-04-12 11:48:43,434 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Got 0 scheduled jobs to be executed at 2024-04-12 11:49:00 GMT
    2024-04-12 11:48:43,436 INFO  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Cleaned up 0 VM scheduled job entries
    2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    2024-04-12 11:48:45,058 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
    SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!
    
    
    2024-04-12 11:48:45,058 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    2024-04-12 11:48:45,058 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
    	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
    	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
    	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
    	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:516)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    2024-04-12 11:48:45,059 DEBUG [c.c.r.ResourceManagerImpl] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
    com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
    	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
    	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
    	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
    	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:516)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	... 60 more
    2024-04-12 11:48:45,060 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
    2024-04-12 11:48:45,060 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Exception: 
    com.cloud.exception.DiscoveryException: Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:834)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
    	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
    	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
    	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
    	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
    	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
    	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
    	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
    	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
    	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
    	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
    	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
    	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
    	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
    	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
    	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    	at org.eclipse.jetty.server.Server.handle(Server.java:516)
    	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
    	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
    	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
    	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
    	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
    	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
    	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
    	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
    	... 59 more
    Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
    	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
    	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
    	... 60 more
    

    i could'nt see any log for setup...while checking on the path /var/log/cloudstack/agent/ is an empty directory

    

    can you upload the logs right after command=addHost ?
30. niyam-choudhari commented on Sep 25, 2024

    

    niyam-choudhari

    on Sep 25, 2024

    More actions

    Ensure keytool is installed and can run with sudo.
31. deajan commented on Apr 15, 2025

    

    deajan

    on Apr 15, 2025

    Contributor

    More actions

    Also see #10703 which has a PR for the keystore-setup script