Repository navigation
VMs stuck in Stopping state due to NPE at org.apache.cloudstack.network.topology.BasicNetworkVisitor.visit(BasicNetworkVisitor.java:201) #3497
Description
Activity
Line 201 of BasicNetworkVisitor.java in CS 4.11.3 is:
if (router.getPodIdToDeployIn().longValue() == destination.getPod().getId()) {in this method:
@Override public boolean visit(final DhcpEntryRules dhcp) throws ResourceUnavailableException { final VirtualRouter router = dhcp.getRouter(); final Commands commands = new Commands(Command.OnError.Stop); final NicVO nicVo = dhcp.getNicVo(); final UserVmVO userVM = dhcp.getUserVM(); final DeployDestination destination = dhcp.getDestination(); final boolean remove = dhcp.isRemove(); **if (router.getPodIdToDeployIn().longValue() == destination.getPod().getId()) {** _commandSetupHelper.createDhcpEntryCommand(router, userVM, nicVo, remove, commands); return _networkGeneralHelper.sendCommandsToRouter(router, commands); } return true; }This works fine in advanced network, but seems to fail with a basic network; also all tests in the relative merge pull seem to be done with advanced networking only.
I have attached jdb and the problem seems to be that destination is NULL, specifically that dhcp.getDestination() returns NULL. From the debugger:
Breakpoint hit: "thread=API-Job-Executor-1", org.apache.cloudstack.network.topology.BasicNetworkVisitor.visit(), line=193 bci=0 API-Job-Executor-1[1] list Source file not found: BasicNetworkVisitor.java API-Job-Executor-1[1] locals Method arguments: dhcp = instance of com.cloud.network.rules.DhcpEntryRules(id=11576) Local variables: API-Job-Executor-1[1] print dhcp dhcp = "com.cloud.network.rules.DhcpEntryRules@725a95d4" API-Job-Executor-1[1] print dhcp.getNicVo() dhcp.getNicVo() = "Nic[862-609-9f0f0a12-c397-4735-84b2-a6222a929e1a-84.33.37.153]" API-Job-Executor-1[1] print dhcp.getUserVM() dhcp.getUserVM() = "VM[User|i-9-609-VM]" API-Job-Executor-1[1] print dhcp.getDestination() dhcp.getDestination() = null API-Job-Executor-1[1] print dhcp.isRemove() dhcp.isRemove() = trueSo no destination is specified and this is causing the NPE. Hope this helps.
Just one more thing: it will always be NULL when invoked from removeDhcpEntry in the BasicNetworkTopology-class, line 461:
final DhcpEntryRules dhcpRules = new DhcpEntryRules(network, nic, profile, null);The same calls happens in the AdvancedNetworkTopology-class BTW, but in the AdvancedNetworkVisitor-class destination is not used and not compared, so it works there.
Why is destination needed in Basic Networking?
You probably want to add the destination parameter to removeDhcpEntry in server/src/org/apache/cloudstack/network/topology/BasicNetworkTopology.java, e. g.:
@Override public boolean removeDhcpEntry(final Network network, final NicProfile nic, final VirtualMachineProfile profile, final DeployDestination dest, final VirtualRouter virtualRouter) throws ResourceUnavailableException { s_logger.debug("REMOVING DHCP ENTRY RULE"); final String typeString = "dhcp entry"; final Long podId = profile.getVirtualMachine().getPodIdToDeployIn(); boolean isPodLevelException = false; if (podId != null && profile.getVirtualMachine().getType() == VirtualMachine.Type.User && network.getTrafficType() == TrafficType.Guest && network.getGuestType() == Network.GuestType.Shared) { isPodLevelException = true; } final boolean failWhenDisconnect = false; final DhcpEntryRules dhcpRules = new DhcpEntryRules(network, nic, profile, dest); dhcpRules.setRemove(true); return applyRules(network, virtualRouter, typeString, isPodLevelException, podId, failWhenDisconnect, new RuleApplierWrapper<RuleApplier>(dhcpRules)); }But that is the obvious choice (e.g. compared to applyDhcpEntry), so there must be something that prevented this trivial solution from being implemented.
Is this PR really fixing the issue? Your original patch was born out of the idea that DHCP entries where left in the VR when it should not.
Now with your patch you are basically avoiding the command being executed, but that means it would fall back to the old behaviour before your patch in Basic Networking. Wouldn't it make sense to change it to something like this:
@Override public boolean visit(final DhcpEntryRules dhcp) throws ResourceUnavailableException { final VirtualRouter router = dhcp.getRouter(); final Commands commands = new Commands(Command.OnError.Stop); final NicVO nicVo = dhcp.getNicVo(); final UserVmVO userVM = dhcp.getUserVM(); final DeployDestination destination = dhcp.getDestination(); final boolean remove = dhcp.isRemove(); // If we are just removing something, no destination is necessary; we always need the router: if (router != null && (remove || (destination != null && destination.getPod() != null && router.getPodIdToDeployIn() != null && router.getPodIdToDeployIn().longValue() == destination.getPod().getId()))) { _commandSetupHelper.createDhcpEntryCommand(router, userVM, nicVo, remove, commands); return _networkGeneralHelper.sendCommandsToRouter(router, commands); } return true; }So that if remove is TRUE it does not check for destination at all?
- added a commit that references this issue
on Jul 23, 2019
Bug Report
After upgrading from 4.11.2.0 to 4.11.3.0 every operation that requires deleting an entry in a basic network seems to fail due to a NPE and therefore the VMs are stuck in Stopping state.
Possibly caused by commit b2b99ca regarding server/src/main/java/org/apache/cloudstack/network/topology/BasicNetworkTopology.java (but this is just my guess)
CloudStack version 4.11.3.0
with Basic Networking
How to reproduce: Create a VM and then try to stop or destroy it. The procedure will not complete and the VM stays in "Stopping" state. In the CloudStack Mgm Server the following error message is shown:
Already tried recreating the VR, cleaning and restarting the network etc., but it makes no difference.
The problems first appeared after upgrading from CloudStack 4.11.2 to 4.11.3.