Repository navigation
Incorrect work of getFreeMemory() in LibvirtStartCommandWrapper.java #3526
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
LibvirtStartCommandWrapper.java
CLOUDSTACK VERSION
4.10 +
CONFIGURATION
N/A
OS / ENVIRONMENT
N/A
SUMMARY
I found error in getFreeMemory() of https://github.com/apache/cloudstack/blob/4.11/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtStartCommandWrapper.java#L170
When I create a new VM we take a list of all VMs from libvirt via getFreeMemory(). After that we iterate by this list. If amount of VMs so large(or libvirt response longer than always) this may take some time. At that time one or more VMs, which has "Running" state can start stopping by another process(job) and not displayed in libvirt anymore. So it bring to exception with "libvirt.libvirtError: Domain not found: no domain with matching id 157124." message and we move on catch with wrong error message(Failed to get free memory).
STEPS TO REPRODUCE
1. Start one proces which will create VMs.
2. When we have 100(create more if not reproduce) VMs on host should start another process which will destroy VMs.
3. Still some time we will see error message "Failed to get free memory" in logs with free memory resources on host.
management-server.log https://yadi.sk/d/wzyZVeTKS2kkDQ
EXPECTED RESULTS
Correct creation of VM without exceptions.
ACTUAL RESULTS
INFO message "Failed to get free memory" and ERROR with InsufficientCapacityEception.
Activity
yadvr commentedon Jul 29, 2019
Can you try/test against 4.11.3.0 or master @DmB991 ?
DmB991 commentedon Jul 29, 2019
@rhtyd yes, reproduced on version 4.11.3.0
https://yadi.sk/d/8ISgwyGI2D7e_A
logid:0d7c4e6c
wido commentedon Aug 26, 2019
@GabrielBrascher Could you look into this? Seems like a valid scenario. We probably need a try/catch around the point where we query the domain from libvirt.
GabrielBrascher commentedon Aug 27, 2019
@wido @rhtyd @DmB991
Commit 9a02190 added the
getFreeMemory(conn,libvirtComputingResource)method. However, it was latter removed by commit 65f31f1 on Aug 8, 2018; therefore, it has already been fixed. Upgrading to 4.11.3.0 or 4.12.0.0 should fix it :)DmB991 commentedon Aug 27, 2019
@GabrielBrascher After upgrading to 4.11.3.0 the problem(INFO message "Failed to get free memory" and ERROR with InsufficientCapacityEception) still appear when hosts has free resources (see the log 2 comments above)
GabrielBrascher commentedon Aug 27, 2019
Thanks, I will check the second log, missed that message.
DmB991 commentedon Sep 9, 2019
@GabrielBrascher Do you have any results of checking logs?
GabrielBrascher commentedon Jul 8, 2020
Sorry for the late reply, @DmB991.
Just updating that I have been trying to reproduce this error on KVM nodes but so far I had no success.
For instance, I deployed more than 150 VMs on a node (via API call s done by a script), reaching its CPU limits, but could not reproduce the issue raised here. Also, I was able to list those VMs and delete all of them at once without any problem.
Have you been able to upgrade from 4.11.3 to 4.13.0.0 or 4.14.0.0?
DmB991 commentedon Jul 23, 2020
@GabrielBrascher Are you deleting(first 100 VMs) and creating new VMs(after 100 VMs exists) at the same time?
I return on 4.11.2, it works perfect for me, except this and #3513 bug. So I recreate VMs(via script) when VM is fallen down.
GabrielBrascher commentedon Jul 24, 2020
@DmB991 I did not execute both, VMs creation and deletion, at the same time, I will re-check considering your use case.
yadvr commentedon Aug 14, 2020
cc @Spaceman1984
yadvr commentedon Aug 19, 2020
cc @Spaceman1984 can you check and see if we need to (quick) fix this?
GabrielBrascher commentedon Aug 20, 2020
@DmB991 @rhtyd @Spaceman1984 I gave another shot trying to see any loose point but I still failed to reproduce it.
Tests were done via CloudMonkey:
Additionally, the piece of code that was failing with
failed to get free memoryis not on codebase anymore.LibvirtStartCommandWrapperclass has been refactored at PR #2766 by @wido. I don`t think that the issue would happen again if both management and agent nodes get upgraded to recent CloudStack versions.yadvr commentedon Aug 27, 2020
Can you review @DmB991 and close if you're unable to hit this with latest 4.14 or master?
DmB991 commentedon Aug 31, 2020
ok, I can't test on latest 4.14 or master, so I close this one.