Repository navigation
VR DHCP lease file not persistent #3272
Description
ISSUE TYPE
- Bug Report - BLOCKER
- Improvement request
COMPONENT NAME
- VR dnsmasq DHCP
CLOUDSTACK VERSION
- ACS 4.11.2
- ACS 4.12 not tested but assumed issues are still present looking at code changes
- Regressions from / previously present in ACS 4.9 / 4.10 / 4.11
REFERENCES
- PR2427 - CLOUDSTACK-10252: Delete dnsmasq leases file on restart / CLOUDSTACK-10252: Delete dnsmasq leases file on restart #2427
- Old Jira issue: VR DNSmasq lease file not persistent across DHCP actions
/ https://issues.apache.org/jira/browse/CLOUDSTACK-10204 - Old Jira issue: New VR after network restart with cleanup is missing DNS and DHCP entries
/ https://issues.apache.org/jira/browse/CLOUDSTACK-10167 - Finite lease introduction in Fixes for VirtualRouters in Basic Networking, especially with mutliple ranges in VLANs #1547
- Also check related issue VR DHCP VPC multihomed VM broken #3273
CONFIGURATION
- ACS4.11.2
- Isolated guest networks and VPC networks
OS / ENVIRONMENT
- Any
SUMMARY
- Leases file is not persistent across network operations:
- Refer to ref fix rpm/deb build error #2 above.
- Leases continues to be not persistent across dnsmasq restarts / additions - i.e. leases file is wiped and leases are not remembered.
- This causes the guest VM dhcp client to get an "NAK" DHCP deny on DHCP renewal, and have to go through the normal procedure again of DISCOVER>OFFER>REQUEST>ACK. For some guest OS versions (RedHat has shown this) where static routes are used the initial "NAK" will break these static routes, leading to problems in the guest OS, i.e. the VM will get a new lease but will have lost it's static routes.
- Improvement request: Lease time for DHCP options is still hard coded to randint(700,760) hours:
https://github.com/apache/cloudstack/blob/master/systemvm/debian/opt/cloud/bin/cs/CsDhcp.py#L145
- Since this ends up in the packaged systemvm.iso file dynamically managing this might be a problem
- When looking at this issue could 5 mins be spent considering if the lease time could be put in as a global setting
STEPS TO REPRODUCE
- Leases file is not persistent across network operations:
- Create VM
- Check contents of /var/lib/misc/dnsmasq.leases
- Create second VM
- Observe contents of /var/lib/misc/dnsmasq.leases is wiped
- Now wipe the lease file on the first VM and restart network
- Observe the /var/log/dnsmasq.log shows entries for a full DISCOVER>OFFER>REQUEST>ACK process rather than a renew
EXPECTED RESULTS
- Leases persistent throughout lifetime of VM
ACTUAL RESULTS
- Leases not persistent causing static route problems on guest VMs
Activity
[-]VR DHCP hosts and leases issue[/-][+][WIP] VR DHCP hosts and leases issue[/+]yadvr commentedon Apr 11, 2019
Interesting discussion https://serverfault.com/questions/723292/dnsmasq-doesnt-automatically-reload-when-entry-is-added-to-etc-hosts
[-][WIP] VR DHCP hosts and leases issue[/-][+]VR DHCP hosts and leases issue[/+][-]VR DHCP hosts and leases issue[/-][+]VR DHCP lease file not peristent[/+][-]VR DHCP lease file not peristent[/-][+]VR DHCP lease file not persistent[/+]router: Persistent DHCP leases file on VRs and cleanup /etc/hosts on …
gildegoma commentedon Jun 13, 2019
FYI: I did some validation tests against CloudStack 4.11.3.0-RC1, and can confirm that this issue is now elegantly resolved 👍
❤️ ❤️ ❤️
andrijapanicsb commentedon Oct 24, 2019
Not really... (partially solved i.e. we don't wipe any more the dnsmasq.leases file - we just remove the specific IP/MAC settings from the CONFIGURATION files for dnsmasq, BUT at the wrong point in time - causing other issues). This is solved now in master - i.e. we delete specific IP/MAC CONFIGURATION only when expunging the VM (i.e. not while stopping the VM - which is the current impemenation in 4.11.3 and 4.13)
Outstanding issues: restarting Network with CleanUP - inside the new VR that is created during network restart, dnsmasq.leases file is absent - this will cause clients to receive DHCPNAK when they try to renew the lease with DHCPREQUEST. Client's will then fallback to the "initialisation process" with DHCPDISCOVER and get's it's IP address - no downtime from IP connectivity point of view - i.e. the client will NOT lose it's IP address at any point in time.
BUT, on some RHEL system, this DHCPNAK will cause RHEL system to drop its routes... thus the problem. This is an edge case.
We yet need to solve that the new VR (that is created during network restarts with CleanUp) to also get the dnsmasq.leases file regenerated somehow (if possible)....
Current 4.11.3 and 4.13 are causing issues as following:
Hope that clarifies the current situation with 4.11.3/4.13 - it's good to be documented somewhere, thus my looooooong comment here.