Cannot start cloudstack-agent after upgrade from 4.16.1 to 4.17.1

ISSUE TYPE

Bug Report

COMPONENT NAME

cloudstack-agent

CLOUDSTACK VERSION

4.17.1

CONFIGURATION

kvm
SharedMountPoint (CephFS)

OS / ENVIRONMENT

CentOS Linux release 7.9.2009 (Core) 3.10.0-1160.45.1.el7.x86_64
libvirt-4.5.0-36.el7_9.5.x86_64
qemu-kvm-ev-2.12.0-44.1.el7_8.1.x86_64
qemu-img-ev-2.12.0-44.1.el7_8.1.x86_64
qemu-kvm-common-ev-2.12.0-44.1.el7_8.1.x86_64
libvirt-daemon-driver-qemu-4.5.0-36.el7_9.5.x86_64

SUMMARY

we have 10 cloudstack node are same configuration, however 2 of 10 node was failed to start cloudstack-agent after ran yum update cloudstack-agent to cloudstack-common-4.17.1.0-1.el7.x86_64, cloudstack-agent-4.17.1.0-1.el7.x86_64,
cloudstack-agent fail to start with reporting Unable to start agent: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor

Resolve by

force remove node from cloudstack-management
yum remove cloudstack-agent
rm -rf /etc/cloudstack/agent/
yum install cloudstack-agent
re-add same node from cloudstack-management

Although this is a workaround from this situation, but it involve quite much cleanup task when it occurs . we dont ever configure any scaleIO storage before.

STEPS TO REPRODUCE

systemctl

EXPECTED RESULTS

cloudstack-agent start without problem after upgrade.

ACTUAL RESULTS

-- Logs begin at Sun 2022-08-28 00:13:44 HKT. --
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: at com.cloud.agent.Agent.<init>(Agent.java:189)
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:451)
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:420)
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:404)
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: at com.cloud.agent.AgentShell.start(AgentShell.java:510)
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: at com.cloud.agent.AgentShell.main(AgentShell.java:545)
Oct 01 02:31:46 compute-20.lab.example.com java[2743357]: Unable to start agent: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
Oct 01 02:31:46 compute-20.lab.example.com systemd[1]: cloudstack-agent.service: main process exited, code=exited, status=67/n/a
Oct 01 02:31:46 compute-20.lab.example.com systemd[1]: Unit cloudstack-agent.service entered failed state.
Oct 01 02:31:46 compute-20.lab.example.com systemd[1]: cloudstack-agent.service failed.
Oct 01 02:31:56 compute-20.lab.example.com systemd[1]: cloudstack-agent.service holdoff time over, scheduling restart.
Oct 01 02:31:56 compute-20.lab.example.com systemd[1]: Stopped CloudStack Agent.
Oct 01 02:31:56 compute-20.lab.example.com systemd[1]: Started CloudStack Agent.
Oct 01 02:31:56 compute-20.lab.example.com java[2743451]: log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell).
Oct 01 02:31:56 compute-20.lab.example.com java[2743451]: log4j:WARN Please initialize the log4j system properly.
Oct 01 02:31:56 compute-20.lab.example.com java[2743451]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Agent started
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Implementation Version is 4.17.1.0
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.AgentShell] (main:) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: vm.rng.rate.period
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: vm.rng.enable
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: resource
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: vm.rng.model
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: guest.network.device
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: local.storage.uuid
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: router.aggregation.command.each.timeout
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: port
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: iscsi.session.cleanup.enabled
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: host
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: pod
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: guid
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: vm.rng.path
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: LibvirtComputingResource.id
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: cluster
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: public.network.device
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: private.network.device
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: zone
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: vm.rng.rate.bytes
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: domr.scripts.dir
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: keystore.passphrase
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: guest.cpu.mode
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: vm.migrate.wait
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: workers
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Found property: hypervisor.type
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Defaulting to using properties file for storage
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Defaulting to the constant time backoff algorithm
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.utils.LogUtils] (main:) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Using default Java settings for IPv6 preference for agent connection
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.agent.AgentShell] (main:) (logid:) Checking to see if agent.pid exists.
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.utils.ProcessUtil] (main:) (logid:) Executing: bash -c echo $PPID
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.utils.ProcessUtil] (main:) (logid:) Executing while with timeout : 120000
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.utils.ProcessUtil] (main:) (logid:) Execution is successful.
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [cloud.agent.Agent] (main:) (logid:) id is 75
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.resource.ServerResourceBase] (main:) (logid:) Retrieving network interface: cloudbr0
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.resource.ServerResourceBase] (main:) (logid:) Retrieving network interface: cloudbr0
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.resource.ServerResourceBase] (main:) (logid:) Retrieving network interface: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [cloud.resource.ServerResourceBase] (main:) (logid:) Retrieving network interface: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtComputingResource] (main:) (logid:) uefi.properties file found at /etc/cloudstack/agent/uefi.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtComputingResource] (main:) (logid:) guest.nvram.template.legacy = /usr/share/OVMF/OVMF_VARS.fd
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtComputingResource] (main:) (logid:) guest.loader.legacy = /usr/share/OVMF/OVMF_CODE.secboot.fd
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtComputingResource] (main:) (logid:) guest.nvram.template.secure = /usr/share/OVMF/OVMF_VARS.secboot.fd
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtComputingResource] (main:) (logid:) guest.loader.secure =/usr/share/OVMF/OVMF_CODE.secboot.fd
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtComputingResource] (main:) (logid:) guest.nvram.path = /var/lib/libvirt/qemu/nvram/
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [resource.virtualnetwork.VirtualRoutingResource] (main:) (logid:) The router.aggregation.command.each.timeout in seconds is set to 600
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/cloudstack-agent/lib/scripts/vm/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/cloudstack-agent/lib/scripts/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/cloudstack-agent/lib/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/cloudstack-agent/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for update_host_passwd.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/update_host_passwd.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-agent/lib/scripts/vm/network/vnet/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-agent/lib/scripts/vm/network/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-agent/lib/scripts/vm/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-agent/lib/scripts/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-agent/lib/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-agent/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for modifyvlan.sh in /usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvm/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-agent/lib/scripts/vm/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-agent/lib/scripts/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-agent/lib/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-agent/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for versions.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/versions.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvm/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-agent/lib/scripts/vm/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-agent/lib/scripts/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-agent/lib/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-agent/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for patch.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/vm/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/cloudstack-agent/lib/scripts/storage/qcow2/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/cloudstack-agent/lib/scripts/storage/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/cloudstack-agent/lib/scripts/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/cloudstack-agent/lib/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/cloudstack-agent/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createvm.sh in /usr/share/cloudstack-common/scripts/storage/qcow2/createvm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/scripts/storage/qcow2/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/scripts/storage/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/scripts/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-common/scripts/storage/qcow2/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/cloudstack-agent/lib/scripts/storage/qcow2/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/cloudstack-agent/lib/scripts/storage/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/cloudstack-agent/lib/scripts/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/cloudstack-agent/lib/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/cloudstack-agent/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for resizevolume.sh in /usr/share/cloudstack-common/scripts/storage/qcow2/resizevolume.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvm/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-agent/lib/scripts/vm/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-agent/lib/scripts/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-agent/lib/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-agent/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for kvmvmactivity.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmvmactivity.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/cloudstack-agent/lib/scripts/storage/qcow2/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/cloudstack-agent/lib/scripts/storage/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/cloudstack-agent/lib/scripts/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/cloudstack-agent/lib/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/cloudstack-agent/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for createtmplt.sh in /usr/share/cloudstack-common/scripts/storage/qcow2/createtmplt.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-agent/lib/scripts/vm/network/vnet/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-agent/lib/scripts/vm/network/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-agent/lib/scripts/vm/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-agent/lib/scripts/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-agent/lib/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-agent/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-common/scripts/vm/network/vnet/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for security_group.py in /usr/share/cloudstack-common/scripts/vm/network/security_group.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-agent/lib/scripts/vm/network/vnet/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-agent/lib/scripts/vm/network/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-agent/lib/scripts/vm/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-agent/lib/scripts/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-agent/lib/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-agent/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovstunnel.py in /usr/share/cloudstack-common/scripts/vm/network/vnet/ovstunnel.py
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/cloudstack-agent/lib/scripts/network/domr/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/cloudstack-agent/lib/scripts/network/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/cloudstack-agent/lib/scripts/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/cloudstack-agent/lib/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/cloudstack-agent/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for router_proxy.sh in /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-agent/lib/scripts/vm/network/vnet/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-agent/lib/scripts/vm/network/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-agent/lib/scripts/vm/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-agent/lib/scripts/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-agent/lib/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-agent/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-common/scripts/vm/network/vnet/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-dhcp-host.sh in /usr/share/cloudstack-common/scripts/vm/network/ovs-pvlan-kvm-dhcp-host.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-agent/lib/scripts/vm/network/vnet/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-agent/lib/scripts/vm/network/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-agent/lib/scripts/vm/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-agent/lib/scripts/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-agent/lib/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-agent/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-common/scripts/vm/network/vnet/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for ovs-pvlan-kvm-vm.sh in /usr/share/cloudstack-common/scripts/vm/network/ovs-pvlan-kvm-vm.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvm/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-agent/lib/scripts/vm/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-agent/lib/scripts/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-agent/lib/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-agent/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for pingtest.sh in /usr/share/cloudstack-common/scripts/vm/pingtest.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [kvm.resource.LibvirtConnection] (main:) (logid:) Looking for libvirtd connection at: qemu:///system
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: INFO  [kvm.resource.LibvirtConnection] (main:) (logid:) No existing libvirtd connection found. Opening a new one
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [kvm.resource.LibvirtConnection] (main:) (logid:) Successfully connected to libvirt at: qemu:///system
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [kvm.resource.LibvirtComputingResource] (main:) (logid:) Found libvirt default network, destroying it and setting autostart to false
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [kvm.resource.LibvirtCapXMLParser] (main:) (logid:) Found /usr/libexec/qemu-kvm as a suiteable emulator
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [kvm.resource.LibvirtCapXMLParser] (main:) (logid:) Found /usr/libexec/qemu-kvm as a suiteable emulator
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in the classpath
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) System resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Classpath resource: null
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/scripts/storage/qcow2/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/scripts/storage/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/scripts/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/lib/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-agent/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Searching in environment.properties
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: DEBUG [utils.script.Script] (main:) (logid:) Looking for managesnapshot.sh in /usr/share/cloudstack-common/scripts/storage/qcow2/managesnapshot.sh
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: ERROR [cloud.agent.AgentShell] (main:) (logid:) Unable to start agent:
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: com.cloud.utils.exception.CloudRuntimeException: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.<init>(KVMStoragePoolManager.java:116)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1056)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.Agent.<init>(Agent.java:189)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:451)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:420)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:404)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.start(AgentShell.java:510)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.main(AgentShell.java:545)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: Unable to start agent: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
Oct 01 02:31:57 compute-20.lab.example.com systemd[1]: cloudstack-agent.service: main process exited, code=exited, status=67/n/a
Oct 01 02:31:57 compute-20.lab.example.com systemd[1]: Unit cloudstack-agent.service entered failed state.
Oct 01 02:31:57 compute-20.lab.example.com systemd[1]: cloudstack-agent.service failed.

The problem seems to be that a StorageAdaptor is being attempted to be created which doesn´t have a default constructor, by the looks of the stacktrace.

@DaanHoogland

The problem seems to be that a StorageAdaptor is being attempted to be created which doesn´t have a default constructor, by the looks of the stacktrace.

@DaanHoogland I think it might not be caused by java code. because @levindecaro has fixed the issue without any code changes
maybe a packaging or installation or permission issue ?

@DaanHoogland

The problem seems to be that a StorageAdaptor is being attempted to be created which doesn´t have a default constructor, by the looks of the stacktrace.

@DaanHoogland I think it might not be caused by java code. because @levindecaro has fixed the issue without any code changes
maybe a packaging or installation or permission issue ?

Yes, but the stored configuration had to be thrown away.
As there is a workaround I wouldn't regard it critical. Not for smaller environments anyway.

@weizhouapache

@weizhouapache there may be a direct cause in terms of permissions but:

Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: ERROR [cloud.agent.AgentShell] (main:) (logid:) Unable to start agent:
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: com.cloud.utils.exception.CloudRuntimeException: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.<init>(KVMStoragePoolManager.java:116)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1056)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.Agent.<init>(Agent.java:189)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:451)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:420)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:404)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.start(AgentShell.java:510)
Oct 01 02:31:57 compute-20.lab.example.com java[2743451]: at com.cloud.agent.AgentShell.main(AgentShell.java:545)

this snippit shows that a default constructor is used in the code and it does not exist in the code. That is a bug in java code for sure.

@levindecaro

@levindecaro is there any way you could reproduce this and test a solution against it? i.e. #6797
I want to move this forward but have a lack of clue atm

@DaanHoogland

@DaanHoogland I'll try to validate it soon by reinstalling the agent with this revised version, I'm not sure it is 100% reproducible, but i'll report the result if got anything new. thank you

@DaanHoogland

@DaanHoogland

Just tried to build 4.17.2 with this #6797, but it still fail after agent upgrade

Dec 22 22:40:08 compute-20.example.com java[1991061]: com.cloud.utils.exception.CloudRuntimeException: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.<init>(KVMStoragePoolManager.java:116)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1056)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.agent.Agent.<init>(Agent.java:189)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:451)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:420)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:404)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.agent.AgentShell.start(AgentShell.java:510)
Dec 22 22:40:08 compute-20.example.com java[1991061]: at com.cloud.agent.AgentShell.main(AgentShell.java:545)
Dec 22 22:40:08 compute-20.example.com java[1991061]: Unable to start agent: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
Dec 22 22:40:08 compute-20.example.com libvirtd[1965359]: 2022-12-22 14:40:08.496+0000: 1965359: error : virNetSocketReadWire:1806 : End of file while reading data: Input/output error

@levindecaro

@levindecaro
Strange. We never see this issue in any testing environments with different OSes.
Can you share the list of files in /usr/share/cloudstack-agent/lib ?
which os and version, jdk version do you use ?

[root@compute-20 ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

[root@compute-20 ~]# java -version
openjdk version "11.0.11" 2021-04-20 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.11+9-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.11+9-LTS, mixed mode, sharing)

[root@compute-20 ~]# ls -l /usr/share/cloudstack-agent/lib
total 91352
-rw-r--r-- 1 root root   69409 Dec 22 17:57 activation-1.1.1.jar
-rw-r--r-- 1 root root   17536 Dec 22 17:57 annotations-13.0.jar
-rw-r--r-- 1 root root   90662 Dec 22 17:57 antisamy-1.5.3.jar
-rw-r--r-- 1 root root   14768 Dec 22 17:57 aopalliance-repackaged-2.5.0-b42.jar
-rw-r--r-- 1 root root  448794 Dec 22 17:57 apache-log4j-extras-1.2.17.jar
-rw-r--r-- 1 root root 1930381 Dec 22 17:57 aspectjweaver-1.8.13.jar
-rw-r--r-- 1 root root 1018510 Dec 22 17:57 aws-java-sdk-core-1.11.940.jar
-rw-r--r-- 1 root root  594212 Dec 22 17:57 aws-java-sdk-kms-1.11.940.jar
-rw-r--r-- 1 root root 1230346 Dec 22 17:57 aws-java-sdk-s3-1.11.940.jar
-rw-r--r-- 1 root root    8433 Dec 22 17:57 batik-constants-1.14.jar
-rw-r--r-- 1 root root  330318 Dec 22 17:57 batik-css-1.14.jar
-rw-r--r-- 1 root root   11466 Dec 22 17:57 batik-i18n-1.14.jar
-rw-r--r-- 1 root root    6663 Dec 22 17:57 batik-shared-resources-1.14.jar
-rw-r--r-- 1 root root  127477 Dec 22 17:57 batik-util-1.14.jar
-rw-r--r-- 1 root root  963713 Dec 22 17:57 bcpkix-jdk15on-1.70.jar
-rw-r--r-- 1 root root 5867298 Dec 22 17:57 bcprov-jdk15on-1.70.jar
-rw-r--r-- 1 root root 1457398 Dec 22 17:57 bctls-jdk15on-1.70.jar
-rw-r--r-- 1 root root  482530 Dec 22 17:57 bcutil-jdk15on-1.70.jar
-rw-r--r-- 1 root root  143992 Dec 22 17:57 bsh-core-2.0b4.jar
-rw-r--r-- 1 root root  415348 Dec 22 17:57 cglib-nodep-3.3.0.jar
-rw-r--r-- 1 root root  214381 Dec 22 17:57 checker-qual-3.5.0.jar
-rw-r--r-- 1 root root   97055 Dec 22 17:57 cloud-agent-4.17.2.0.jar
-rw-r--r-- 1 root root 2290269 Dec 22 17:57 cloud-api-4.17.2.0.jar
-rw-r--r-- 1 root root  576521 Dec 22 17:57 cloud-core-4.17.2.0.jar
-rw-r--r-- 1 root root  103668 Dec 22 17:57 cloud-engine-api-4.17.2.0.jar
-rw-r--r-- 1 root root   76537 Dec 22 17:57 cloud-engine-components-api-4.17.2.0.jar
-rw-r--r-- 1 root root 1873790 Dec 22 17:57 cloud-engine-schema-4.17.2.0.jar
-rw-r--r-- 1 root root  167883 Dec 22 17:57 cloud-engine-storage-4.17.2.0.jar
-rw-r--r-- 1 root root   17189 Dec 22 17:57 cloud-engine-storage-configdrive-4.17.2.0.jar
-rw-r--r-- 1 root root   64512 Dec 22 17:57 cloud-engine-storage-image-4.17.2.0.jar
-rw-r--r-- 1 root root   89243 Dec 22 17:57 cloud-engine-storage-volume-4.17.2.0.jar
-rw-r--r-- 1 root root    6665 Dec 22 17:57 cloud-framework-agent-lb-4.17.2.0.jar
-rw-r--r-- 1 root root    7622 Dec 22 17:57 cloud-framework-ca-4.17.2.0.jar
-rw-r--r-- 1 root root   78031 Dec 22 17:57 cloud-framework-cluster-4.17.2.0.jar
-rw-r--r-- 1 root root   29919 Dec 22 17:57 cloud-framework-config-4.17.2.0.jar
-rw-r--r-- 1 root root  169293 Dec 22 17:57 cloud-framework-db-4.17.2.0.jar
-rw-r--r-- 1 root root   10418 Dec 22 17:57 cloud-framework-direct-download-4.17.2.0.jar
-rw-r--r-- 1 root root   12441 Dec 22 17:57 cloud-framework-events-4.17.2.0.jar
-rw-r--r-- 1 root root   91967 Dec 22 17:57 cloud-framework-ipc-4.17.2.0.jar
-rw-r--r-- 1 root root  120836 Dec 22 17:57 cloud-framework-jobs-4.17.2.0.jar
-rw-r--r-- 1 root root   18092 Dec 22 17:57 cloud-framework-managed-context-4.17.2.0.jar
-rw-r--r-- 1 root root   12346 Dec 22 17:57 cloud-framework-rest-4.17.2.0.jar
-rw-r--r-- 1 root root   24812 Dec 22 17:57 cloud-framework-security-4.17.2.0.jar
-rwxr-xr-x 1 root root  525409 Oct 23  2021 cloud-plugin-hypervisor-kvm-4.15.0.0.jar
-rw-r--r-- 1 root root  593523 Dec 22 17:57 cloud-plugin-hypervisor-kvm-4.17.2.0.jar
-rw-r--r-- 1 root root   84494 Dec 22 17:57 cloud-plugin-network-ovs-4.17.2.0.jar
-rw-r--r-- 1 root root   82833 Dec 22 17:57 cloud-plugin-storage-volume-scaleio-4.17.2.0.jar
-rw-r--r-- 1 root root  151621 Dec 22 17:57 cloud-plugin-storage-volume-storpool-4.17.2.0.jar
-rw-r--r-- 1 root root  123023 Dec 22 17:57 cloud-secondary-storage-4.17.2.0.jar
-rw-r--r-- 1 root root 3435303 Dec 22 17:57 cloud-server-4.17.2.0.jar
-rw-r--r-- 1 root root  403787 Dec 22 17:57 cloud-utils-4.17.2.0.jar
-rw-r--r-- 1 root root  233859 Dec 22 17:57 commons-beanutils-1.9.2.jar
-rw-r--r-- 1 root root  353793 Dec 22 17:57 commons-codec-1.15.jar
-rw-r--r-- 1 root root  575389 Dec 22 17:57 commons-collections-3.2.1.jar
-rw-r--r-- 1 root root 1018316 Dec 22 17:57 commons-compress-1.21.jar
-rw-r--r-- 1 root root  362679 Dec 22 17:57 commons-configuration-1.10.jar
-rw-r--r-- 1 root root   25287 Dec 22 17:57 commons-daemon-1.2.3.jar
-rw-r--r-- 1 root root  207774 Dec 22 17:57 commons-dbcp2-2.8.0.jar
-rw-r--r-- 1 root root  146108 Dec 22 17:57 commons-digester-1.8.1.jar
-rw-r--r-- 1 root root   54495 Dec 22 17:57 commons-email-1.5.jar
-rw-r--r-- 1 root root   72446 Dec 22 17:57 commons-fileupload-1.4.jar
-rw-r--r-- 1 root root  305001 Dec 22 17:57 commons-httpclient-3.1.jar
-rw-r--r-- 1 root root  285424 Dec 22 17:57 commons-io-2.8.0.jar
-rw-r--r-- 1 root root  284220 Dec 22 17:57 commons-lang-2.6.jar
-rw-r--r-- 1 root root  577742 Dec 22 17:57 commons-lang3-3.11.jar
-rw-r--r-- 1 root root   61829 Dec 22 17:57 commons-logging-1.2.jar
-rw-r--r-- 1 root root 2213560 Dec 22 17:57 commons-math3-3.6.1.jar
-rw-r--r-- 1 root root  307003 Dec 22 17:57 commons-net-3.7.2.jar
-rw-r--r-- 1 root root  134858 Dec 22 17:57 commons-pool2-2.9.0.jar
-rw-r--r-- 1 root root  186077 Dec 22 17:57 commons-validator-1.6.jar
-rw-r--r-- 1 root root    5340 Dec 22 17:57 converter-moshi-2.9.0.jar
-rw-r--r-- 1 root root 1409265 Dec 22 17:57 cxf-core-3.2.14.jar
-rw-r--r-- 1 root root  688071 Dec 22 17:57 cxf-rt-frontend-jaxrs-3.2.14.jar
-rw-r--r-- 1 root root  376299 Dec 22 17:57 cxf-rt-transports-http-3.2.14.jar
-rw-r--r-- 1 root root   63292 Dec 22 17:57 eddsa-0.3.0.jar
-rw-r--r-- 1 root root 1347501 Dec 22 17:57 ehcache-core-2.6.11.jar
-rw-r--r-- 1 root root   13879 Dec 22 17:57 error_prone_annotations-2.3.4.jar
-rw-r--r-- 1 root root  395859 Dec 22 17:57 esapi-2.1.0.1.jar
-rw-r--r-- 1 root root    4617 Dec 22 17:57 failureaccess-1.0.1.jar
-rw-r--r-- 1 root root 7202725 Dec 22 17:57 groovy-all-2.4.17.jar
-rw-r--r-- 1 root root  173608 Dec 22 17:57 gson-1.7.2.jar
-rw-r--r-- 1 root root 2862361 Dec 22 17:57 guava-30.1-jre.jar
-rw-r--r-- 1 root root  748221 Dec 22 17:57 guava-testlib-18.0.jar
-rw-r--r-- 1 root root  186763 Dec 22 17:57 hk2-api-2.5.0-b42.jar
-rw-r--r-- 1 root root  189454 Dec 22 17:57 hk2-locator-2.5.0-b42.jar
-rw-r--r-- 1 root root  135317 Dec 22 17:57 hk2-utils-2.5.0-b42.jar
-rw-r--r-- 1 root root  780321 Dec 22 17:57 httpclient-4.5.13.jar
-rw-r--r-- 1 root root  328436 Dec 22 17:57 httpcore-4.4.14.jar
-rw-r--r-- 1 root root  161382 Dec 22 17:57 influxdb-java-2.21.jar
-rw-r--r-- 1 root root  102220 Dec 22 17:57 ini4j-0.5.4.jar
-rw-r--r-- 1 root root  565410 Dec 22 17:57 ion-java-1.0.2.jar
-rw-r--r-- 1 root root    8781 Dec 22 17:57 j2objc-annotations-1.3.jar
-rw-r--r-- 1 root root   75704 Dec 22 17:57 jackson-annotations-2.12.5.jar
-rw-r--r-- 1 root root  365536 Dec 22 17:57 jackson-core-2.12.5.jar
-rw-r--r-- 1 root root 1515991 Dec 22 17:57 jackson-databind-2.12.5.jar
-rw-r--r-- 1 root root   48468 Dec 22 17:57 jackson-dataformat-cbor-2.6.7.jar
-rw-r--r-- 1 root root  120362 Dec 22 17:57 jackson-datatype-jsr310-2.12.5.jar
-rw-r--r-- 1 root root   35626 Dec 22 17:57 jackson-jaxrs-base-2.12.1.jar
-rw-r--r-- 1 root root   16185 Dec 22 17:57 jackson-jaxrs-json-provider-2.12.1.jar
-rw-r--r-- 1 root root   34589 Dec 22 17:57 jackson-module-jaxb-annotations-2.8.4.jar
-rw-r--r-- 1 root root  142255 Dec 22 17:57 jasypt-1.9.3.jar
-rw-r--r-- 1 root root   33304 Dec 22 17:57 java-ipv6-0.17.jar
-rw-r--r-- 1 root root  379481 Dec 22 17:57 java-linstor-0.3.0.jar
-rw-r--r-- 1 root root  782774 Dec 22 17:57 javassist-3.27.0-GA.jar
-rw-r--r-- 1 root root   26586 Dec 22 17:57 javax.annotation-api-1.3.2.jar
-rw-r--r-- 1 root root    2497 Dec 22 17:57 javax.inject-1.jar
-rw-r--r-- 1 root root    5951 Dec 22 17:57 javax.inject-2.5.0-b42.jar
-rw-r--r-- 1 root root  608084 Dec 22 17:57 javax.mail-1.5.6.jar
-rw-r--r-- 1 root root  203426 Dec 22 17:57 javax.persistence-2.2.1.jar
-rw-r--r-- 1 root root   95505 Dec 22 17:57 javax.servlet-api-4.0.1.jar
-rw-r--r-- 1 root root  127509 Dec 22 17:57 javax.ws.rs-api-2.1.jar
-rw-r--r-- 1 root root  125632 Dec 22 17:57 jaxb-api-2.3.0.jar
-rw-r--r-- 1 root root  255502 Dec 22 17:57 jaxb-core-2.3.0.jar
-rw-r--r-- 1 root root  963660 Dec 22 17:57 jaxb-impl-2.3.0.jar
-rw-r--r-- 1 root root   17075 Dec 22 17:57 jbcrypt-1.0.0.jar
-rw-r--r-- 1 root root  181496 Dec 22 17:57 jersey-client-2.26.jar
-rw-r--r-- 1 root root 1125788 Dec 22 17:57 jersey-common-2.26.jar
-rw-r--r-- 1 root root   69689 Dec 22 17:57 jersey-entity-filtering-2.26.jar
-rw-r--r-- 1 root root   62474 Dec 22 17:57 jersey-hk2-2.26.jar
-rw-r--r-- 1 root root   60934 Dec 22 17:57 jersey-media-json-jackson-2.26.jar
-rw-r--r-- 1 root root   66241 Dec 22 17:57 jersey-media-multipart-2.26.jar
-rw-r--r-- 1 root root   27587 Dec 22 17:57 jmespath-java-1.11.940.jar
-rw-r--r-- 1 root root 1506993 Dec 22 17:57 jna-5.5.0.jar
-rw-r--r-- 1 root root 2681931 Dec 22 17:57 jna-platform-5.5.0.jar
-rw-r--r-- 1 root root  644415 Dec 22 17:57 joda-time-2.10.9.jar
-rw-r--r-- 1 root root  282591 Dec 22 17:57 jsch-0.1.55.jar
-rw-r--r-- 1 root root   45944 Dec 22 17:57 json-20090211.jar
-rw-r--r-- 1 root root   33015 Dec 22 17:57 jsr305-1.3.9.jar
-rw-r--r-- 1 root root 1487085 Dec 22 17:57 kotlin-stdlib-1.4.10.jar
-rw-r--r-- 1 root root  191212 Dec 22 17:57 kotlin-stdlib-common-1.4.0.jar
-rw-r--r-- 1 root root    3985 Dec 22 17:57 kotlin-stdlib-jdk7-1.4.10.jar
-rw-r--r-- 1 root root   16230 Dec 22 17:57 kotlin-stdlib-jdk8-1.4.10.jar
-rw-r--r-- 1 root root  148040 Dec 22 17:57 libvirt-0.5.3.jar
-rwxr-xr-x 1 root root    5764 Dec 22 17:57 libvirtqemuhook
-rw-r--r-- 1 root root    2199 Dec 22 17:57 listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
-rw-r--r-- 1 root root   15680 Dec 22 17:57 logging-interceptor-4.9.0.jar
-rw-r--r-- 1 root root  519087 Dec 22 17:57 mail-1.5.0-b01.jar
-rw-r--r-- 1 root root   85448 Dec 22 17:57 metrics-core-3.0.2.jar
-rw-r--r-- 1 root root   30865 Dec 22 17:57 metrics-jvm-3.0.2.jar
-rw-r--r-- 1 root root   63977 Dec 22 17:57 mimepull-1.9.6.jar
-rw-r--r-- 1 root root  138602 Dec 22 17:57 moshi-1.8.0.jar
-rw-r--r-- 1 root root  118965 Dec 22 17:57 msgpack-core-0.8.21.jar
-rw-r--r-- 1 root root   29680 Dec 22 17:57 mxparser-1.2.2.jar
-rw-r--r-- 1 root root  124106 Dec 22 17:57 nekohtml-1.9.16.jar
-rw-r--r-- 1 root root 4080249 Dec 22 17:57 netty-all-4.1.42.Final.jar
-rw-r--r-- 1 root root  258278 Dec 22 17:57 not-yet-commons-ssl-0.3.9.jar
-rw-r--r-- 1 root root  791329 Dec 22 17:57 okhttp-4.9.0.jar
-rw-r--r-- 1 root root  243179 Dec 22 17:57 okio-2.8.0.jar
-rw-r--r-- 1 root root 1413987 Dec 22 17:57 opensaml-2.6.6.jar
-rw-r--r-- 1 root root  574762 Dec 22 17:57 openws-1.5.6.jar
-rw-r--r-- 1 root root   20235 Dec 22 17:57 osgi-resource-locator-1.0.1.jar
-rw-r--r-- 1 root root 1660960 Dec 22 17:57 protobuf-java-3.11.1.jar
-rw-r--r-- 1 root root  105909 Dec 22 17:57 rados-0.6.0.jar
-rw-r--r-- 1 root root  105848 Dec 22 17:57 reflections-0.9.12.jar
-rw-r--r-- 1 root root  332351 Dec 22 17:57 reload4j-1.2.20.jar
-rw-r--r-- 1 root root  125435 Dec 22 17:57 retrofit-2.9.0.jar
-rwxr-xr-x 1 root root    3710 Dec 22 17:57 rolling-maintenance
-rw-r--r-- 1 root root   26084 Dec 22 17:57 slf4j-api-1.7.5.jar
-rw-r--r-- 1 root root  383093 Dec 22 17:57 spring-aop-5.3.18.jar
-rw-r--r-- 1 root root  698595 Dec 22 17:57 spring-beans-5.3.18.jar
-rw-r--r-- 1 root root 1274098 Dec 22 17:57 spring-context-5.3.18.jar
-rw-r--r-- 1 root root 1482364 Dec 22 17:57 spring-core-5.3.18.jar
-rw-r--r-- 1 root root  289394 Dec 22 17:57 spring-expression-5.3.18.jar
-rw-r--r-- 1 root root   24439 Dec 22 17:57 spring-jcl-5.3.18.jar
-rw-r--r-- 1 root root 1639557 Dec 22 17:57 spring-web-5.3.18.jar
-rw-r--r-- 1 root root  161867 Dec 22 17:57 stax2-api-3.1.4.jar
-rw-r--r-- 1 root root   34431 Dec 22 17:57 swagger-annotations-2.0.0.jar
-rw-r--r-- 1 root root 1285238 Dec 22 17:57 tink-1.5.0.jar
-rw-r--r-- 1 root root  343549 Dec 22 17:57 trilead-ssh2-build-217-jenkins-27.jar
-rw-r--r-- 1 root root   85203 Dec 22 17:57 truth-0.23.jar
-rw-r--r-- 1 root root  449505 Dec 22 17:57 velocity-1.7.jar
-rw-r--r-- 1 root root  512742 Dec 22 17:57 woodstox-core-5.0.3.jar
-rw-r--r-- 1 root root 2730442 Dec 22 17:57 xalan-2.7.0.jar
-rw-r--r-- 1 root root 1207073 Dec 22 17:57 xercesImpl-2.8.0.jar
-rw-r--r-- 1 root root  674607 Dec 22 17:57 xmlgraphics-commons-2.6.jar
-rw-r--r-- 1 root root    7188 Dec 22 17:57 xmlpull-1.1.3.1.jar
-rw-r--r-- 1 root root  173293 Dec 22 17:57 xmlschema-core-2.2.5.jar
-rw-r--r-- 1 root root  655992 Dec 22 17:57 xmlsec-1.5.7.jar
-rw-r--r-- 1 root root  673001 Dec 22 17:57 xmltooling-1.4.6.jar
-rw-r--r-- 1 root root  308863 Dec 22 17:57 xom-1.2.5.jar
-rw-r--r-- 1 root root  633140 Dec 22 17:57 xstream-1.4.19.jar

@levindecaro

@levindecaro
I think you need to remove cloud-plugin-hypervisor-kvm-4.15.0.0.jar

@levindecaro

@levindecaro I think you need to remove cloud-plugin-hypervisor-kvm-4.15.0.0.jar

looks it is the root cause, but i have to wait next maintenance windows to verify it.

@DaanHoogland

I have created a 4.15.0 testing env, and upgrade the 4.17.2.0.

The old cloud-plugin-hypervisor-kvm-4.15.0.0.jar has been removed during upgrade.

[root@ref-trl-4110-k-M7-wei-zhou-kvm2 lib]# ls -l cloud-*jar
-rw-r--r--. 1 root root   97055 Dec 16 17:10 cloud-agent-4.17.2.0.jar
-rw-r--r--. 1 root root 2290269 Dec 16 17:10 cloud-api-4.17.2.0.jar
-rw-r--r--. 1 root root  576521 Dec 16 17:10 cloud-core-4.17.2.0.jar
-rw-r--r--. 1 root root  103668 Dec 16 17:10 cloud-engine-api-4.17.2.0.jar
-rw-r--r--. 1 root root   76537 Dec 16 17:10 cloud-engine-components-api-4.17.2.0.jar
-rw-r--r--. 1 root root 1873790 Dec 16 17:10 cloud-engine-schema-4.17.2.0.jar
-rw-r--r--. 1 root root  167883 Dec 16 17:10 cloud-engine-storage-4.17.2.0.jar
-rw-r--r--. 1 root root   17189 Dec 16 17:10 cloud-engine-storage-configdrive-4.17.2.0.jar
-rw-r--r--. 1 root root   64512 Dec 16 17:10 cloud-engine-storage-image-4.17.2.0.jar
-rw-r--r--. 1 root root   89243 Dec 16 17:10 cloud-engine-storage-volume-4.17.2.0.jar
-rw-r--r--. 1 root root    6665 Dec 16 17:10 cloud-framework-agent-lb-4.17.2.0.jar
-rw-r--r--. 1 root root    7622 Dec 16 17:10 cloud-framework-ca-4.17.2.0.jar
-rw-r--r--. 1 root root   78031 Dec 16 17:10 cloud-framework-cluster-4.17.2.0.jar
-rw-r--r--. 1 root root   29919 Dec 16 17:10 cloud-framework-config-4.17.2.0.jar
-rw-r--r--. 1 root root  169293 Dec 16 17:10 cloud-framework-db-4.17.2.0.jar
-rw-r--r--. 1 root root   10418 Dec 16 17:10 cloud-framework-direct-download-4.17.2.0.jar
-rw-r--r--. 1 root root   12441 Dec 16 17:10 cloud-framework-events-4.17.2.0.jar
-rw-r--r--. 1 root root   91967 Dec 16 17:10 cloud-framework-ipc-4.17.2.0.jar
-rw-r--r--. 1 root root  120836 Dec 16 17:10 cloud-framework-jobs-4.17.2.0.jar
-rw-r--r--. 1 root root   18092 Dec 16 17:10 cloud-framework-managed-context-4.17.2.0.jar
-rw-r--r--. 1 root root   12346 Dec 16 17:10 cloud-framework-rest-4.17.2.0.jar
-rw-r--r--. 1 root root   24812 Dec 16 17:10 cloud-framework-security-4.17.2.0.jar
-rw-r--r--. 1 root root  593523 Dec 16 17:10 cloud-plugin-hypervisor-kvm-4.17.2.0.jar
-rw-r--r--. 1 root root   84494 Dec 16 17:10 cloud-plugin-network-ovs-4.17.2.0.jar
-rw-r--r--. 1 root root   82833 Dec 16 17:10 cloud-plugin-storage-volume-scaleio-4.17.2.0.jar
-rw-r--r--. 1 root root  151621 Dec 16 17:10 cloud-plugin-storage-volume-storpool-4.17.2.0.jar
-rw-r--r--. 1 root root  123023 Dec 16 17:10 cloud-secondary-storage-4.17.2.0.jar
-rw-r--r--. 1 root root 3435303 Dec 16 17:10 cloud-server-4.17.2.0.jar
-rw-r--r--. 1 root root  403787 Dec 16 17:10 cloud-utils-4.17.2.0.jar

when I copied the old jar file and restart cloudstack-agent, I got same error

[root@ref-trl-4110-k-M7-wei-zhou-kvm2 lib]# cp /usr/share/cloudstack-agent-41500/lib/cloud-plugin-hypervisor-kvm-4.15.0.0.jar /usr/share/cloudstack-agent/lib/

[root@ref-trl-4110-k-M7-wei-zhou-kvm2 lib]# systemctl restart cloudstack-agent

[root@ref-trl-4110-k-M7-wei-zhou-kvm2 lib]# tail -f /var/log/cloudstack/agent/agent.log 
2023-01-05 12:02:06,621 ERROR [cloud.agent.AgentShell] (main:null) (logid:) Unable to start agent: 
com.cloud.utils.exception.CloudRuntimeException: java.lang.InstantiationException: com.cloud.hypervisor.kvm.storage.ScaleIOStorageAdaptor
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.<init>(KVMStoragePoolManager.java:116)
	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.configure(LibvirtComputingResource.java:1056)
	at com.cloud.agent.Agent.<init>(Agent.java:189)
	at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:451)
	at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:420)
	at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:404)
	at com.cloud.agent.AgentShell.start(AgentShell.java:510)
	at com.cloud.agent.AgentShell.main(AgentShell.java:545)

the host became normal after removing the old jar file

[root@ref-trl-4110-k-M7-wei-zhou-kvm2 lib]# rm -f /usr/share/cloudstack-agent/lib/cloud-plugin-hypervisor-kvm-4.15.0.0.jar

[root@ref-trl-4110-k-M7-wei-zhou-kvm2 lib]# systemctl restart cloudstack-agent

this is not a bug. @DaanHoogland @levindecaro

Navigation Menu

Repository navigation

Cannot start cloudstack-agent after upgrade from 4.16.1 to 4.17.1 #6794

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cannot start cloudstack-agent after upgrade from 4.16.1 to 4.17.1 #6794

Description

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Activity

DaanHoogland commented on Oct 2, 2022

weizhouapache commented on Oct 3, 2022

DaanHoogland commented on Oct 3, 2022

DaanHoogland commented on Oct 11, 2022

DaanHoogland commented on Nov 3, 2022

levindecaro commented on Dec 15, 2022

levindecaro commented on Dec 22, 2022

weizhouapache commented on Dec 22, 2022

levindecaro commented on Dec 23, 2022

weizhouapache commented on Dec 23, 2022

levindecaro commented on Dec 23, 2022

weizhouapache commented on Jan 5, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions