Site to Site VPN Connections shows disconnected if there are no running vm's #13687

Activity

1. kiranchavala

   added the

   Bug

   issue type on Jul 24, 2026
2. kiranchavala

   added

   bug

   on Jul 24, 2026
3. DaanHoogland

   added this to the 4.22.2 milestone on Jul 24, 2026
4. DaanHoogland

   added this to Apache CloudStack BugFest - Issueson Jul 24, 2026
5. DaanHoogland

   added

   component:virtual-router

   component:networking

   component:vpc

   and removed

   bug

   on Jul 24, 2026
6. DaanHoogland

   moved this to Todo in Apache CloudStack BugFest - Issueson Jul 24, 2026
7. skorchir commented on Oct 1, 2026

   

   skorchir

   on Oct 1, 2026

   More actions

   Same symptom on 4.22.1.1 (VPC, VpcVirtualRouter, non-persistent tier offering). Traced it to the state poller's router selection rather than to the tunnel.

   Symptom

   A site-to-site VPN connection on a VPC with no running VM never changes state. After createVpnConnection it reads Connecting (or Disconnected when passive) indefinitely, even while the peer has completed IKE and reports its tunnel up; resetVpnConnection only rewrites the same word. Deploying one VM in a tier moves it to Connected within a minute. Destroying every VM and waiting out network GC freezes it at its last value, so a tunnel that later drops still reads Connected. listRouters shows the VPC router's guest NIC appearing and disappearing with the tier; the router itself stays Running.

   Cause (branch 4.22, head 2e63c60; the cited files are byte-identical on tag 4.22.1.1 and on main)

   • The only writer of Connected/Disconnected from live router state is CheckRouterTask → updateSite2SiteVpnConnectionState (VirtualNetworkApplianceManagerImpl.java:1077-1080, body :784-870), fed by _routerDao.listIsolatedByHostId(null).
   • That query is an INNER JOIN on router_network_ref with guest_type = Isolated (DomainRouterDaoImpl.java:110-118, :253-265). A router with no such row is never polled.
   • A VPC router has no such row at deploy. It gets one when a tier is implemented (VpcVirtualRouterElement.java:227-234 → VpcVirtualNetworkApplianceManagerImpl.java:151-166) and loses it when the tier is shut down (removeVpcRouterFromGuestNetwork).
   • A tier is implemented by the first VM start and shut down by NetworkGarbageCollector once no running VM holds a NIC in it (NetworkOrchestrator.java:3602-3662, NetworkDaoImpl.findNetworksToGarbageCollect:540-547).

   So until a VM runs, and again after the last one is gone, the task never looks at the router, and the connection keeps whatever startVpnConnection last wrote (Site2SiteVpnManagerImpl.java:417-421).

   Fix

   updateSite2SiteVpnConnectionState only does work for VPC routers: getConnectionsForRouter returns an empty list when vpcId is null (Site2SiteVpnManagerImpl.java:887-896). Selecting routers by isolated-tier membership is the wrong join for this step. CheckRouterTask already iterates _vpcDao.listAll() for its redundant-state half (:1083-1088); passing _routerDao.listByVpcId(vpcId) (DomainRouterDaoImpl.java:415-420) for each VPC that owns site-to-site connections into the VPN check, keeping its existing Running / redundant-PRIMARY / INTERNAL_LB_VM filters, polls the router for as long as it exists. A DAO method listing routers with vpc_id IS NOT NULL would do the same.
8. skorchir commented on Oct 1, 2026

   

   skorchir

   on Oct 1, 2026

   More actions

   Reproduced on a 4.22.1.1 lab with the management DB and router shells open, which confirms the mechanism above and adds the garbage-collection variant.

   Setup. Three VPCs on one KVM host: vpca and vpcb with one tier and one running VM each; vpcd with one tier and no VM. Site-to-site VPN between vpcd and vpcb, both sides passive=false.

   Observed

   • router_network_ref held one Isolated row for vpca's router and one for vpcb's. Nothing for vpcd's router, which had only its public NIC.
   • CheckRouterTask logged Found 2 routers to update status every 30 s before and after vpcd existed.
   • vpcb's connection toward vpcd read Connected. vpcd's connection toward vpcb read Connecting for as long as it was left, while ipsec statusall on both routers showed the SA ESTABLISHED and the CHILD_SA INSTALLED. vpcb's router had initiated.
   • On vpcd's router, ip route get <peer tier address> sourced from the public IP, so its own trigger ping in configure.py never matches the leftsubnet trap policy; a VM-less VPC router can respond to IKE but never initiate it.

   The reverse direction, after garbage collection (network.gc.interval and network.gc.wait lowered to 60 s to make it quick): stopping the only VM in vpcb's tier (not destroying it) moved the tier Implemented → Shutdown → Allocated, removed the router's tier NIC and its router_network_ref row, and the poller dropped to Found 3 routers. vpcb's connections kept reading Connected. Deleting the far side's connection then removed the SA from vpcb's router while the API still read Connected 40 s later. Starting the VM re-implemented the tier and the state corrected to Disconnected within one poll.

   Workaround that holds. A tier on a network offering with ispersistent=true (a clone of DefaultIsolatedNetworkOfferingForVpcNetworks with the same services and providers; createNetworkOffering accepts the flag with forvpc=true) is implemented at creation and never garbage-collected, so its router carries the row from the start. A VPC built on it reached Connected within 10 s with no VM, and moving an existing Allocated tier onto it with updateNetwork networkofferingid= implemented the tier immediately and unfroze its connection. Whether the poller should depend on tier implementation at all is still the question for the fix in the comment above.
9. Damans227

   self-assigned this

   on Oct 7, 2026