Repository navigation
KVM hosts are being reset by a script #2657
Description
ISSUE TYPE
Bug Report
COMPONENT NAME
KVM
CLOUDSTACK VERSION
All 4.x and 2.x versions seem to be affected.
CONFIGURATION
N/A
OS / ENVIRONMENT
This issue is related to KVM hosts. As far as I can see, XEN hosts are affected too.
SUMMARY
KVM hosts are being rebooted by kvmheartbeat.sh when the NFS-server that is being used as a primary storage becomes unreachable.
I'd like to propose change behavior of kvmheartbeat.sh, as it's not necessary to reboot a whole host when only one of VMs is having difficulties while other VMs are working fine with other storages.
STEPS TO REPRODUCE
It's easy to reproduce: just deploy a VM with a volume on an NFS-based storage, break NFS and wait.
EXPECTED RESULTS
The KVM host will be reset.
ACTUAL RESULTS
The kvmscript.sh will reboot the server.
Activity
serverchief commentedon May 21, 2018
As of 4.11 CloudStack has an enhancement for KVM HA. Instead of relying on shell script and specific primary store - we will check the write time stamp on the disks for all VMs that are known to reside on hypervisor.
All VM disks/files must not update the write timestamp for an extended amount of time(configurable) before a specifc action is taken (configurable). Even If a single vm disk will have a write timestamp and all other don’t - we will not power off the host.
You can read more about KVM HA on cloudstack homepage, there is also a video talk by Rohit Yadav that explains it in great details.
khmarochos commentedon May 21, 2018
Thank you! I didn't notice that 4.11.x have it fixed, in 4.9.2 it still can reset a host.
In any case, the script and
echo b > /proc/sysrq-triggeris still present. Won't it be executed at some moment? Looks a bit dangerous, as for me.wido commentedon May 22, 2018
The main problem is that it is hard to determine if the host isn't responding or that something else is going on.
You do not want the VM to run twice as that might (will!) cause data corruption. That's why a hard reboot is being issued if the storage is having trouble.
khmarochos commentedon May 23, 2018
Thank you for notation.
That's exactly why I proposed (d821625) to find the VMs using the mountpoint (by
lsofandawk) and then kill theirqemu-kvmprocesses.Slair1 commentedon Jun 25, 2018
Please see PR #2722 and #2472
I agree and created these PRs to address the built-in reboot
Make networkofferingid required in migrateNetwork
Make networkofferingid required in migrateNetwork
DaanHoogland commentedon Jan 29, 2020
long time no activity, closing. please reopen @Melnik13 if you think i'm at err