Skip to content

VR: dnsmasq service start failed  #5208

Closed
#5319
Closed
VR: dnsmasq service start failed #5208
#5319
Assignees
weizhouapache
Labels
component:virtual-routerneeds-testingtype:bug
Milestone
4.15.2.0

Description

@havengit
havengit
opened on Jul 14, 2021
ISSUE TYPE
  • Improvement Request
COMPONENT NAME
VR
CLOUDSTACK VERSION
cloudstack 4.15.0
CONFIGURATION
OS / ENVIRONMENT

ubuntu20.04.4

SUMMARY

Create a large number of virtual machines 10+, and the DNSMASQ service in VR will be restarted frequently, resulting in service startup failure

Need to modify parameters in VR ,To support mass creation of virtual machines

cat /etc/systemd/system.conf
default:
DefaultStartLimitIntervalSec=10s
DefaultStartLimitBurst=5
example:
DefaultStartLimitIntervalSec=10s
DefaultStartLimitBurst=100

logs:
cat /var/log/daemon.log
Jul 13 03:25:39 r-359-VM systemd[1]: user-runtime-dir@0.service: Start request r
epeated too quickly.
Jul 13 03:25:39 r-359-VM systemd[1]: user-runtime-dir@0.service: Failed with res
ult 'start-limit-hit'.
Jul 13 03:25:39 r-359-VM systemd[1]: Failed to start User Runtime Directory /run
/user/0.
Jul 13 03:25:39 r-359-VM systemd[1]: Dependency failed for User Manager for UID
0.
Jul 13 03:25:39 r-359-VM systemd[1]: user@0.service: Job user@0.service/start fa
iled with result 'dependency'.
Jul 13 03:25:39 r-359-VM systemd[1]: Started Session 12714 of user root.
Jul 13 03:25:39 r-359-VM systemd[1]: Stopping dnsmasq - A lightweight DHCP and c
aching DNS server...
Jul 13 03:25:39 r-359-VM systemd[1]: dnsmasq.service: Succeeded.
Jul 13 03:25:39 r-359-VM systemd[1]: Stopped dnsmasq - A lightweight DHCP and ca
ching DNS server.
Jul 13 03:25:39 r-359-VM systemd[1]: dnsmasq.service: Start request repeated too
quickly.
Jul 13 03:25:39 r-359-VM systemd[1]: dnsmasq.service: Failed with result 'start-
limit-hit'.
Jul 13 03:25:39 r-359-VM systemd[1]: Failed to start dnsmasq - A lightweight DHC
P and caching DNS server.
Jul 13 03:25:39 r-359-VM systemd[1]: session-12714.scope: Succeeded.

STEPS TO REPRODUCE
EXPECTED RESULTS
Create a large number of virtual machines at one time ,VR work fine
ACTUAL RESULTS
dnsmasq start faild, vm cannot get IP by dhcp.

Activity

  1. luhaijiao commented on Jul 14, 2021

    @luhaijiao
    luhaijiao
    on Jul 14, 2021

    We found this issue while creating VMs in one batch

  2. yadvr commented on Jul 15, 2021

    @yadvr
    yadvr
    on Jul 15, 2021
    Member

    @luhaijiao can you test with 4.15.1.0?

  3. yadvr
    assigned
    weizhouapache
    on Jul 15, 2021
  4. yadvr
    added this to the 4.16.0.0 milestone on Jul 15, 2021

  5. weizhouapache commented on Jul 15, 2021

    @weizhouapache
    weizhouapache
    on Jul 15, 2021
    Member

    @luhaijiao @havengit
    creating vms with 1-2s interval should solve the problem, it is a workaround.

  6. weizhouapache commented on Jul 15, 2021

    @weizhouapache
    weizhouapache
    on Jul 15, 2021
    Member

    @havengit can you post the resulf of command below in VR ?

    grep dnsmasq /var/log/cloud.log |grep Service

  7. weizhouapache commented on Jul 15, 2021

    @weizhouapache
    weizhouapache
    on Jul 15, 2021
    Member

    I did a small testing in VR

    interval=0.5
action=reload
for i in `seq 1 20`;do echo $i;systemctl $action dnsmasq && sleep $interval;done

    if action is 'reload', it works without issue (tested interval =0.5, 1 or 2)
    if action is 'restart', error if interval is 0.5 or 1, no error if interval is 2.

  8. havengit commented on Jul 16, 2021

    @havengit
    havengit
    on Jul 16, 2021
    ContributorAuthor

    @havengit can you post the resulf of command below in VR ?

    grep dnsmasq /var/log/cloud.log |grep Service

    2021-07-02 02:26:20,942 INFO Service dnsmasq restart
    2021-07-03 02:14:22,327 INFO Service dnsmasq restart
    2021-07-05 10:14:44,116 INFO Service dnsmasq restart
    2021-07-05 10:19:17,779 INFO Service dnsmasq restart
    2021-07-06 05:53:34,067 INFO Service dnsmasq restart
    2021-07-13 03:25:14,428 INFO Service dnsmasq restart
    2021-07-13 03:25:15,239 INFO Service dnsmasq restart
    2021-07-13 03:25:15,970 INFO Service dnsmasq restart
    2021-07-13 03:25:21,205 INFO Service dnsmasq restart
    2021-07-13 03:25:30,068 INFO Service dnsmasq restart
    2021-07-13 03:25:30,692 INFO Service dnsmasq restart
    2021-07-13 03:25:37,763 INFO Service dnsmasq restart
    2021-07-13 03:25:38,520 INFO Service dnsmasq restart
    2021-07-13 03:25:39,232 INFO Service dnsmasq restart
    2021-07-13 03:25:39,859 INFO Service dnsmasq restart
    2021-07-13 08:04:11,194 INFO Service dnsmasq restart
    2021-07-13 08:37:12,665 INFO Service dnsmasq restart
    2021-07-13 08:37:15,996 INFO Service dnsmasq reload
    2021-07-13 08:37:16,679 INFO Service dnsmasq reload
    2021-07-13 08:46:46,242 INFO Service dnsmasq restart
    2021-07-13 08:46:47,140 INFO Service dnsmasq restart
    2021-07-13 08:46:47,962 INFO Service dnsmasq restart
    2021-07-13 09:10:48,036 INFO Service dnsmasq reload
    2021-07-13 09:24:28,826 INFO Service dnsmasq restart
    2021-07-13 09:54:04,212 INFO Service dnsmasq restart
    2021-07-13 18:14:24,265 INFO Service dnsmasq restart
    2021-07-14 03:58:54,292 INFO Service dnsmasq restart
    2021-07-14 07:51:02,498 INFO Service dnsmasq restart
    2021-07-14 08:31:22,588 INFO Service dnsmasq restart
    2021-07-14 08:31:26,810 INFO Service dnsmasq restart
    2021-07-14 08:31:27,450 INFO Service dnsmasq restart
    2021-07-14 08:31:31,785 INFO Service dnsmasq restart

  9. havengit commented on Jul 16, 2021

    @havengit
    havengit
    on Jul 16, 2021
    ContributorAuthor

    I did a small testing in VR

    interval=0.5
action=reload
for i in `seq 1 20`;do echo $i;systemctl $action dnsmasq && sleep $interval;done

    if action is 'reload', it works without issue (tested interval =0.5, 1 or 2)
    if action is 'restart', error if interval is 0.5 or 1, no error if interval is 2.

    Somebody mentioned that service reload instead of restart, #3613

  10. nvazquez
    added
    component:virtual-router
    on Jul 16, 2021
  11. weizhouapache
    linked a pull request that will close this issuevr: reload dnsmasq when start vms #5319on Aug 19, 2021
  12. nvazquez
    added
    type:bug
    on Aug 25, 2021
  13. nvazquez
    modified the milestones: 4.16.0.0, 4.15.2.0 on Aug 25, 2021
  14. nvazquez
    added
    needs-testing
    on Aug 25, 2021
  15. nvazquez
    closed this as completedon Aug 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

component:virtual-routerneeds-testingtype:bug

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions