Skip to content

Local Storage migration fails due to storage pool not found on destination KVM host #7942

Closed
#7945
Closed
Local Storage migration fails due to storage pool not found on destination KVM host#7942
#7945
Assignees
weizhouapache
Labels
Severity:CriticalCritical bugcomponent:kvmtype:bug
Milestone
4.18.1.0

Description

@yadvr
yadvr
opened on Sep 5, 2023

On a 4.18.1.0 RC2 adv zone KVM env using local only storage and local storage for systemvms, migration fails from kvm1 to kvm2 due to the following seen in the target kvm host

2023-09-05 21:12:06,645 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:f38dafe4) Attempting to create volume 23ef733c-c26e-405a-829e-6e0010b3cbeb (Filesystem) in pool 3a11d1ea-eff4-4f5b-9b7c-e0ff89a8c9dd with size (2.20 GB) 2361393152
2023-09-05 21:12:06,691 INFO  [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:f38dafe4) No existing libvirtd connection found. Opening a new one
2023-09-05 21:12:06,692 WARN  [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) (logid:f38dafe4) Can not find a connection for Instance i-2-453-VM. Assuming the default connection.
2023-09-05 21:12:08,499 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:f38dafe4) Trying to fetch storage pool 8aa9768c-cbcf-4e8e-8875-f94a7f9445b6 from libvirt
2023-09-05 21:12:08,500 WARN  [cloud.agent.Agent] (agentRequest-Handler-4:null) (logid:f38dafe4) Caught:
com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 8aa9768c-cbcf-4e8e-8875-f94a7f9445b6 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '8aa9768c-cbcf-4e8e-8875-f94a7f9445b6'
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:278)
	at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:264)

The above can sometimes cause the VM to crash as it fails right at the last step on the dest. kvm host end, and CloudStack may further report that domain is not running.

However, systemvm migration with storage work. I'm not sure if this is an env issue or a bug.

ISSUE TYPE
  • Bug Report
  • Improvement Request
  • Enhancement Request
  • Feature Idea
  • Documentation Report
  • Other
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS

Activity

  1. yadvr
    assigned
    weizhouapache
    on Sep 5, 2023
  2. yadvr
    added
    type:bug
    component:kvm
    on Sep 5, 2023

  3. weizhouapache commented on Sep 5, 2023

    @weizhouapache
    weizhouapache
    on Sep 5, 2023
    Member

    @rohityadavcloud
    Thanks for raising the issue.
    I will try to reproduce the issue

  4. weizhouapache commented on Sep 5, 2023

    @weizhouapache
    weizhouapache
    on Sep 5, 2023
    Member

    I got an error when migrate the default template
    image

    and management-server log shows
    2023-09-05 16:47:24,215 DEBUG [c.c.a.t.Request] (AgentManager-Handler-18:null) (logid:) Seq 2-4185251428710744296: Processing: { Ans: , MgmtId: 167780532, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateAnswer":{"result":"false","details":"Exception during migrate: org.libvirt.LibvirtException: operation failed: migration of disk vda failed: Input/output error","wait":"0","bypassHostMaintenance":"false"}}] }

    and then

    2023-09-05 16:47:24,220 DEBUG [c.c.a.t.Request] (Work-Job-Executor-4:ctx-9389653d job-33/job-34 ctx-f9253e30) (logid:baac3e91) Seq 1-1506454075355431213: Sending  { Cmd , MgmtId: 167780532, via: 1(ref-trl-5592-k-Mr8-wei-zhou-kvm1), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":"3","name":"i-2-3-VM"...
...

2023-09-05 16:47:24,815 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:null) (logid:) Seq 1-1506454075355431213: Processing:  { Ans: , MgmtId: 167780532, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Could not fetch storage pool 1a30047b-99dd-4b7b-b2e5-4e862c1117a5 from libvirt due to org.libvirt.LibvirtException: Storage pool not found: no storage pool with matching uuid '1a30047b-99dd-4b7b-b2e5-4e862c1117a5'
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:278)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:264)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDisksViaVmSpec(KVMStoragePoolManager.java:239)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.handleRollback(LibvirtPrepareForMigrationCommandWrapper.java:150)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:62)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtPrepareForMigrationCommandWrapper.execute(LibvirtPrepareForMigrationCommandWrapper.java:52)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1848)
        at com.cloud.agent.Agent.processRequest(Agent.java:662)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
        at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }
  5. weizhouapache
    added this to the 4.18.1.0 milestone on Sep 5, 2023
  6. weizhouapache
    added
    Severity:CriticalCritical bug
    on Sep 5, 2023

  7. yadvr commented on Sep 5, 2023

    @yadvr
    yadvr
    on Sep 5, 2023
    MemberAuthor

    @weizhouapache I think this may be an env issue or something that affects only local storage post upgrade. If I register a new/fresh template after upgrade and then deploy VM and migrate with (local) storage, it seems to be working. The issue is the MigrationCommand fails and upon failure a prepareformigration command is sent with rollback=true.

    Could you test by registering a new template again?

    I suspect the I/O issue could be due to something about the migrate command not copying the template (backing file) or something else?

  8. yadvr commented on Sep 5, 2023

    @yadvr
    yadvr
    on Sep 5, 2023
    MemberAuthor

    @weizhouapache update - I downloaded and re-registered the same template in my env, deployed a VM and that can live migrate with storage without any issues 🤦 I suspect if you were to re-register the template you were template - it might just work - now to find out why this is happening.

  9. yadvr commented on Sep 5, 2023

    @yadvr
    yadvr
    on Sep 5, 2023
    MemberAuthor

    Could be related - #5759 cc @weizhouapache

  10. weizhouapache commented on Sep 5, 2023

    @weizhouapache
    weizhouapache
    on Sep 5, 2023
    Member

    @rohityadavcloud
    Thanks for the information

  11. weizhouapache commented on Sep 5, 2023

    @weizhouapache
    weizhouapache
    on Sep 5, 2023
    Member

    This seems to be related to #7408

    @harikrishna-patnala
    can you please have a look ?

  12. yadvr commented on Sep 5, 2023

    @yadvr
    yadvr
    on Sep 5, 2023
    MemberAuthor

    Update - for my specific template, it had some io_uring related settings removing those the vm migration seems to be working for newly deployed VMs. It could still be combination of other issues, not a blocker for me now. Thanks @weizhouapache

  13. weizhouapache commented on Sep 5, 2023

    @weizhouapache
    weizhouapache
    on Sep 5, 2023
    Member

    Update - for my specific template, it had some io_uring related settings removing those the vm migration seems to be working for newly deployed VMs. It could still be combination of other issues, not a blocker for me now. Thanks @weizhouapache

    Thanks @rohityadavcloud for you findings.
    I will investigate it tomorrow
    It would be good if we can at least find a workaround

  14. weizhouapache commented on Sep 6, 2023

    @weizhouapache
    weizhouapache
    on Sep 6, 2023
    Member

    we have found the root cause

    diff --git a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/
+resource/LibvirtConnection.java
index c70a72f399c..0f8031e3aaa 100644
--- a/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java
+++ b/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtConnection.java
@@ -21,6 +21,7 @@ import java.util.Map;

 import org.apache.log4j.Logger;
 import org.libvirt.Connect;
+import org.libvirt.Library;
 import org.libvirt.LibvirtException;

 import com.cloud.hypervisor.Hypervisor;
@@ -44,6 +45,7 @@ public class LibvirtConnection {
         if (conn == null) {
             s_logger.info("No existing libvirtd connection found. Opening a new one");
             conn = new Connect(hypervisorURI, false);
+            Library.initEventLoop();
             s_logger.debug("Successfully connected to libvirt at: " + hypervisorURI);
             s_connections.put(hypervisorURI, conn);
         } else {

    @harikrishna-patnala is working on the fix.

  15. JoaoJandre commented on Sep 6, 2023

    @JoaoJandre
    JoaoJandre
    on Sep 6, 2023
    Contributor

    @weizhouapache @rohityadavcloud It seems that this is not only affecting local storage, I had the same issue that @weizhouapache reported while migrating volumes of a running VM from NFS to SharedMountPoint.

  16. weizhouapache commented on Sep 6, 2023

    @weizhouapache
    weizhouapache
    on Sep 6, 2023
    Member

    @weizhouapache @rohityadavcloud It seems that this is not only affecting local storage, I had the same issue that @weizhouapache reported while migrating volumes of a running VM from NFS to SharedMountPoint.

    @JoaoJandre
    which cloudstack version do you use ?
    can you provide some logs if possible ?

  17. JoaoJandre commented on Sep 6, 2023

    @JoaoJandre
    JoaoJandre
    on Sep 6, 2023
    Contributor

    @weizhouapache I'm using 4.19.0.0-SNAPSHOT, that I've built from main.
    Unfortunately I've already rebuilt my environment with another version, but I had the same exception that you described here #7942 (comment).

  18. weizhouapache commented on Sep 6, 2023

    @weizhouapache
    weizhouapache
    on Sep 6, 2023
    Member

    @weizhouapache I'm using 4.19.0.0-SNAPSHOT, that I've built from main. Unfortunately I've already rebuilt my environment with another version, but I had the same exception that you described here #7942 (comment).

    Ok @JoaoJandre
    If you have not seen this issue in other versions, it should be caused by same root cause which is fixed by #7945

  19. yadvr commented on Sep 7, 2023

    @yadvr
    yadvr
    on Sep 7, 2023
    MemberAuthor

    Fixed by #7945

  20. yadvr
    closed this as completedon Sep 7, 2023
  21. weizhouapache
    linked a pull request that will close this issuekvm: fix live vm migration between local storage pools #7945on Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Severity:CriticalCritical bugcomponent:kvmtype:bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions