Repository navigation
Cannot start system vm on XenServer 8.2 #6148
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
API
CLOUDSTACK VERSION
4.16.0.0
CONFIGURATION
N/A
OS / ENVIRONMENT
Citrix XenServer/Hypervisor 8.2 CU1
SUMMARY
We have updated one of our clusters in Cloudstack to version 8.2 (It was 7.1 before). After the update, we were unable to start any of our system vm's on the updated cluster. We checked the logs and found out that Cloudstack searches for a local storage repository called "XenServer Tools":
2022-02-07 10:24:38,613 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-22:ctx-2d0a1646) (logid:a315cd67) Unable to start v-67620-VM due to
com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name XenServer Tools or XCP-ng Tools or Citrix Hypervisor Tools
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.findPatchIsoSR(CitrixResourceBase.java:1064)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1088)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:99)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:54)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1757)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-02-07 10:25:41,212 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-246:ctx-6940899b) (logid:a315cd67) Unable to start v-67620-VM due to
com.cloud.utils.exception.CloudRuntimeException: can not find systemvmiso
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1096)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:99)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:54)
at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1757)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
It seems Citrix changed the way how they offer XenTools on their Hypervisor. You have to manually download them from the Internet and therefore they have removed the dedicated XenTools storage repository.
STEPS TO REPRODUCE
It's simple.
- Install XenServer 8.2 CU1 on a test machine.
- Add machine to a CloudStack 4.16.0 instance.
- Try to host a system vm on that host. It will not work and you will see the above errors in the management-server.log
WORKAROUND
We have currently implemented a workaround, by just adding a custom storage repository on the affected XenServers, called "XenServer Tools". It points to an smb share on the network and it contains a file with the name "systemvm.iso". This ISO contains our current system vm template registered in Cloudstack.
Activity
nvazquez commentedon Mar 22, 2022
Hi @maikgiessler your workaround makes sense since CloudStack will look for SR with name: "XenServer Tools", "Citrix Hypervisor Tools" or "XCP-ng Tools". I think the last one should be automatically created in this case, any idea @andrijapanicsb @rohityadavcloud?
nvazquez commentedon May 3, 2022
Thanks for the workaround @maikgiessler - I'm closing this issue since Xen 8.2 is not supported: http://docs.cloudstack.apache.org/en/latest/releasenotes/compat.html
maikgiessler commentedon May 5, 2022
In the documentation, XS 8.2 is given as 'expected to work'. Isn't it planned to have official support for XS 8.2 in CS 4.17.0? It would be great if it could be added, because this is a serious issue for all customer having XenServer 8.2 in place.
Thanks!
andrijapanicsb commentedon May 5, 2022
@maikgiessler Since, XenServer binaries are not publicly available, it was not possible to test it, community only tested XCP-ng 8.1, 8.2.
If you can provide XS 8.2 binaries, we would be happy to test it?
maikgiessler commentedon May 5, 2022
Hm, ok. Sure, I could provide the ISO of XS 8.2.1. Would you have me a place where to put it?
nvazquez commentedon May 5, 2022
Hi @maikgiessler I think Github has a size limit of 100MB per file. Can you try sharing over Google Drive or other providers? We can test it for the next milestone/release. With respect to the documentation, I am updating it with your workaround on this PR: apache/cloudstack-documentation#272
maikgiessler commentedon May 5, 2022
Here is the iso:
https://sap-my.sharepoint.com/:u:/p/maik_giessler/EflCK0fuv29PhwLsWmsq5eYBkDdiP3pkLlHuzp1HbebndQ
The link is valid for 3 months.
@nvazquez: Thank you. It would be great if you could add the workaround to the official documentation.
pdion891 commentedon Oct 5, 2022
Because xentool iso is not exposed by xenserver anymore, either have to create the SR from cloudstack scripts running on xenserver or change how to present systemvm.iso to VMs under XenServer.
One way of fixing the issue seams to be using following command on a xenserver cluster:
"xe pbd-list sr-uuid=" should return a PBD for each host and have currently-attached to true.
The problem using an NFS share is if cloudstack update the systemvm.iso , because of a major release upgrade or script fix, the iso from the NFS might not be updated properly by cloudstack.