Repository navigation
VM migration with disks fails between local storage hypervisors if the VM template is deleted #10246
Description
problem
When trying to migrate a virtual machine between two local storage hypervisors, the migration fails if the VM template is removed.
I did some reasearch, and have figured out the possible root cause of it.
First, the template is not found here: StorageSystemDataMotionStrategy.java:2158
Therefore, the system thinks this VM does not have a template.
As the result, the system thinks this VM does not have any backing files, and tries to perform full clone migration: MigrateKVMAsync.java:128. This seems to be the right migration option, the documentation says it will perform the migration of all the VMs disks: VIR_MIGRATE_NON_SHARED_INC
But there is a catch: the QCOW2 images have to be precreated on the destination machine with the correct size.
There is a method that would copy the template from the secondary storage to the destination host: KvmNonManagedStorageDataMotionStrategy.java:220, but there is no method to just create a QCOW2 file with the specific size at the specific path.
versions
4.20
The steps to reproduce the bug
- Take two KVM hypervisors with local storage pools.
- Create a virtual machine from a template on one of them.
- Delete this template after the test virtual machine was started.
- Try to migrate the virtual machine with its storage to the other hypervisor.
- It will fail with an error telling that the backing file does not exist on the target machine:
2025-01-22 12:41:08,662 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:[ctx-9e293509, job-226]) (logid:61e8bd24) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd com.cloud.utils.exception.CloudRuntimeException: Failed to migrate VM [VM instance {"id":18,"instanceName":"i-2-18-VM","type":"User","uuid":"d3b043ff-716a-4bc8-b20e-639853b0babe"}] along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) [{volume: "33", from: "3", to:"4"}] and VM to host [{vm: "18", from: "2", to:"4"}] failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Exception during migrate: org.libvirt.LibvirtException: Path '/var/lib/libvirt/images/bb51c27d-ca94-4cda-a74e-cec07295ce26' is not accessible: No such file or directory].].
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1444)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
....
What to do about it?
It would be nice to fix the storage migration when the VM template was removed.
I would like to try to develop a fix myself but my knowledge of Cloudstack internals is not enough to know where to look.
I'd need some guidance how to properly develop the fix, for example point me to the right files where I can look.
Activity
weizhouapache commentedon Jan 23, 2025
@phsm
can you share the full log of job-226 ?
phsm commentedon Jan 24, 2025
Attached the full log of this job from the mgmt server
mgmt-server.log
weizhouapache commentedon Jan 24, 2025
Hi @phsm
can you test the following steps ?
phsm commentedon Jan 24, 2025
But these steps are the exact reproduce scenario, so the log I've attached earlier is the log of migration after the first 3 steps were done.
weizhouapache commentedon Jan 24, 2025
@phsm
It worked in my testing (4.20). can you re-run it ?
phsm commentedon Jan 28, 2025
I re-tested in on 4.20, and indeed it works. Not sure how it didn't work for me at the first time.
Looks like this PR has a fix for this issue: #8911
weizhouapache commentedon Jan 28, 2025
thanks for the update and the link, looks right.
Jayd603 commentedon May 30, 2025
I'm commenting here because some how, this still happened to me on a fresh 4.20 install. It happened with a VM that I migrated from VMWare. I converted the VMWare VM to a qcow file , imported it as a template in CS and it worked fine, so I force deleted the template, and it failed exactly like above when I went to migrate it to another host. Any ideas?
phsm commentedon May 31, 2025
We noticed we still have it for a part of the virtual machines.
I tried to get to the bottom of this:
However, the libvirt still throws this exception. Thus I suspect there is some bug in the qemu or libvirt process.
@Jayd603 What seems to solve it for a buggy VM is stopping and starting it in Cloudstack. Once the VM is stopped/started, it can be live-migrated.
Jayd603 commentedon May 31, 2025
Yup, stopping the instance and starting solved the problem. I notice that qemu was updated on the host and VMs were running on the old binaries, possibly related.
Indrasura20 commentedon Apr 14, 2026
Seems I encountered the exact same problem as well. I am using AC 4.20.1
I have the template deleted and tried to migrate a local storage vm to another host and it fails. I had to restart the vm to migrate it.