Repository navigation
VM HA does not work when server is stopped gracefully #4211
Description
When we stop a server (hypervisor) gracefully, vms on it will NOT be started on other servers even HA is enabled.
ISSUE TYPE
- Bug Report
COMPONENT NAME
VM HA
CLOUDSTACK VERSION
4.14/4.15
CONFIGURATION
advanced zone
OS / ENVIRONMENT
KVM
SUMMARY
STEPS TO REPRODUCE
1. create a service offering with HA enabled
2. deploy a vm
3. go to the server where vm is running on, and stop it by "poweroff" command
EXPECTED RESULTS
vm will be started on other servers.
ACTUAL RESULTS
vm is NOT started on other servers.
only when the server is Up again, cloudstack detects the vm is stopped out-of-band and starts it.
Activity
weizhouapache commentedon Jul 15, 2020
@rhtyd @DaanHoogland @PaulAngus @svenvogel have you faced this issue ? is this an expected behavior ?
DaanHoogland commentedon Jul 29, 2020
This doesn't sound familiar @weizhouapache . To recap,
I would not expect this behaviour as a user but it might be a side effect of the host-ha, hence my questions.
GabrielBrascher commentedon Aug 6, 2020
@weizhouapache adding to @DaanHoogland's questions:
weizhouapache commentedon Aug 12, 2020
@GabrielBrascher we use nfs as zone-wide storage pool.
weizhouapache commentedon Aug 12, 2020
@DaanHoogland
if kvm cloudstack-agent is stopped, there is no thread in cloudstack to determine its actual host state.
RodrigoDLopez commentedon Aug 24, 2020
hi @weizhouapache,
My environment is similar to yours, apart from the storage pool that I use cluster-wide, on this environment I could see HA working on cloudstack version
4.13.1.1with an NFS cluster-wide storage pool.On 4.13.1.1 version I could see my VMs migrate to a suitable HOST after I force shut down the HOST where those VMs were, but with gracefully shut down I couldn't see the HA working, those VMs never migrate to another suitable HOST, only when this HOST came back the cloudstack detected this VMs as stopped. But at
4.14and4.15version, I couldn't see HA working even when I force shut down, those VMs are never detected as stopped by cloudstackshwstppr commentedon Feb 3, 2021
@weizhouapache are planning/working on a fix for this?
weizhouapache commentedon Feb 3, 2021
@shwstppr yeah. I will create a pr in coming days.
weizhouapache commentedon Apr 22, 2022
this issue should have been fixed by #5164
for users who do not use NFS, there are two PRs open for review and testing:
KVM High Availability regardless of storage #4978
KVM HA Enhancements for Ceph RBD Primary Storage #5862