Repository navigation
VR Mising #10574
Description
I have my VR that deleted during restart network, however it not automatically created back.
Now all the VM that under this GuestNetwork are unable to start, it shows Insufficient Capacity due to no VR . Even though I want to delete this network from the VM and change another one also cannot delete it.
The only thing I can do is destroy the whole vm .
I am using Cloudstack 4.15 and Ubuntu 18.
Activity
weizhouapache commentedon Mar 14, 2025
please share the logs, thanks
hstan77 commentedon Mar 14, 2025
Starting VM that missing VR
025-03-14 18:57:40,347 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-81:ctx-e1df6ca7 job-487558/job-487559 ctx-ad6c35ae) (logid:ed251ac2) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!
at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:288)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1485)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1840)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1774)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1149)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5464)
at jdk.internal.reflect.GeneratedMethodAccessor424.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5631)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
=======+Restarting network that missing VR ===
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:251)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1392)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1327)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3166)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1994)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2012)
at jdk.internal.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy256.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-14 18:59:52,200 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-100:ctx-310b6a64 job-487561 ctx-99dba8c4) (logid:5b299583) Network id=217 failed to restart.
2025-03-14 18:59:52,223 DEBUG [c.c.a.ApiServlet] (qtp1886491834-1071:ctx-467ab645) (logid:7657
hstan77 commentedon Mar 14, 2025
Any body can let me know to manual get the VR up . or Some Db record need to update it .
DaanHoogland commentedon Mar 14, 2025
@hstan77 , is there enough capacity to start the VR ?
weizhouapache commentedon Mar 14, 2025
@hstan77
I think there are some exceptions before the log you shared. can you have a look ?
also, can you restart the network with cleanup ?
hstan77 commentedon Mar 14, 2025
Hi
It is enaugh to start VR, a lot more
Cleaning up , it will say no VR to clean up
hstan77 commentedon Mar 14, 2025
i grep WARN and ERR
**restart Network
2025-03-14 19:55:30,234 WARN [c.c.a.ApiServer] (qtp1886491834-1084:ctx-584ca2a8 ctx-a2113a51) (logid:6033a140) Unknown API command: listAndSwitchSamlAccount
2025-03-14 19:55:46,110 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-24:ctx-52d1ff6b job-488646 ctx-faec8381) (logid:2394dcae) Failed to implement network Ntwk[217|Guest|6] elements and resources as a part of network restart due to
2025-03-14 19:55:46,112 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-24:ctx-52d1ff6b job-488646 ctx-faec8381) (logid:2394dcae) Network id=217 failed to restart.
2025-03-14 19:57:30,843 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-100:ctx-8c1177ba job-488649 ctx-3c1be356) (logid:7d78035b) Didn't support redundant virtual router without public network!
** restart network and clean up
2025-03-14 19:56:58,533 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-87:ctx-8d18e4fc job-488648 ctx-d509981e) (logid:518baef4) Didn't support redundant virtual router without public network!
2025-03-14 19:56:58,560 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-87:ctx-8d18e4fc job-488648) (logid:518baef4) Unexpected exception while executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
hstan77 commentedon Mar 14, 2025
This is detail with debug log
2025-03-14 19:58:24,165 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-8c72f876 job-488650) (logid:345a719c) Add job-488650 into job monitoring
2025-03-14 19:58:24,169 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8844, power state: PowerOn
2025-03-14 19:58:24,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1886491834-1113:ctx-cdaaea5e ctx-3c16e021) (logid:a4093f5d) submit async job-488650, details: AsyncJobVO {id:488650, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3079878","id":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","ctxDetails":"{"interface com.cloud.network.Network":"a3cca856-973e-4a6a-aa9d-d4c908938cc7"}","ctxAccountId":"2","uuid":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","cmdEventType":"NETWORK.RESTART"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2130706689, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-03-14 19:58:24,175 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8848, power state: PowerOn
2025-03-14 19:58:24,176 DEBUG [c.c.a.ApiServlet] (qtp1886491834-1113:ctx-cdaaea5e ctx-3c16e021) (logid:a4093f5d) ===END=== 180.75.249.24 -- GET id=a3cca856-973e-4a6a-aa9d-d4c908938cc7&cleanup=false&makeredundant=false&command=restartNetwork&response=json
2025-03-14 19:58:24,180 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1940, power state: PowerOn
2025-03-14 19:58:24,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-50:ctx-8c72f876 job-488650) (logid:3fef167a) Executing AsyncJobVO {id:488650, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3079878","id":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","ctxDetails":"{"interface com.cloud.network.Network":"a3cca856-973e-4a6a-aa9d-d4c908938cc7"}","ctxAccountId":"2","uuid":"a3cca856-973e-4a6a-aa9d-d4c908938cc7","cmdEventType":"NETWORK.RESTART"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 2130706689, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2025-03-14 19:58:24,185 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9621, power state: PowerOn
2025-03-14 19:58:24,191 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8857, power state: PowerOn
2025-03-14 19:58:24,197 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9628, power state: PowerOn
2025-03-14 19:58:24,204 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1181, power state: PowerOn
2025-03-14 19:58:24,210 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1951, power state: PowerOn
2025-03-14 19:58:24,215 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1445, power state: PowerOn
2025-03-14 19:58:24,221 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 9642, power state: PowerOn
2025-03-14 19:58:24,228 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1196, power state: PowerOn
2025-03-14 19:58:24,235 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1199, power state: PowerOn
2025-03-14 19:58:24,242 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 433, power state: PowerOn
2025-03-14 19:58:24,249 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8627, power state: PowerOn
2025-03-14 19:58:24,253 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Restarting network 217...
2025-03-14 19:58:24,255 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7352, power state: PowerOn
2025-03-14 19:58:24,258 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Implementing the network Ntwk[217|Guest|6] elements and resources as a part of network restart without cleanup
2025-03-14 19:58:24,260 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1209, power state: PowerOn
2025-03-14 19:58:24,266 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 1212, power state: PowerOn
2025-03-14 19:58:24,271 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7358, power state: PowerOn
2025-03-14 19:58:24,277 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8640, power state: PowerOn
2025-03-14 19:58:24,283 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 960, power state: PowerOn
2025-03-14 19:58:24,289 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7620, power state: PowerOn
2025-03-14 19:58:24,295 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 965, power state: PowerOn
2025-03-14 19:58:24,300 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 967, power state: PowerOn
2025-03-14 19:58:24,306 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 970, power state: PowerOn
2025-03-14 19:58:24,311 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 5834, power state: PowerOn
2025-03-14 19:58:24,317 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 8139, power state: PowerOn
2025-03-14 19:58:24,323 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2000, power state: PowerOn
2025-03-14 19:58:24,325 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Asking SecurityGroupProvider to implemenet Ntwk[217|Guest|6]
2025-03-14 19:58:24,328 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 980, power state: PowerOn
2025-03-14 19:58:24,333 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 981, power state: PowerOn
2025-03-14 19:58:24,339 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 983, power state: PowerOn
2025-03-14 19:58:24,345 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2007, power state: PowerOn
2025-03-14 19:58:24,351 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 984, power state: PowerOn
2025-03-14 19:58:24,356 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 6106, power state: PowerOn
2025-03-14 19:58:24,361 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2012, power state: PowerOn
2025-03-14 19:58:24,367 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 991, power state: PowerOn
2025-03-14 19:58:24,372 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2017, power state: PowerOn
2025-03-14 19:58:24,375 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Asking VirtualRouter to implemenet Ntwk[217|Guest|6]
2025-03-14 19:58:24,378 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 2019, power state: PowerOn
2025-03-14 19:58:24,384 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 996, power state: PowerOn
2025-03-14 19:58:24,388 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Didn't support redundant virtual router without public network!
2025-03-14 19:58:24,389 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7142, power state: PowerOn
2025-03-14 19:58:24,395 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl] (AgentManager-Handler-12:null) (logid:) VM state report. host: 19, vm id: 7147, power state: PowerOn
2025-03-14 19:58:24,395 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Failed to implement network Ntwk[217|Guest|6] elements and resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:251)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1392)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1327)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3166)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1994)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2012)
at jdk.internal.reflect.GeneratedMethodAccessor716.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy256.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2025-03-14 19:58:24,397 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-50:ctx-8c72f876 job-488650 ctx-e17cb53d) (logid:3fef167a) Network id=217 failed to restart.
hstan77 commentedon Mar 14, 2025
When restart network, it show error cannot find router, of cause it cant find, cause the VR missing.
When restart with Clenaup, It show cannot find VR to clean also. and that is right cause VR is missing.
However my question is the VR not auto created, and how can we create it back ?
weizhouapache commentedon Mar 14, 2025
Can you upload the full log when start /creatr the VR ?
The logs you uploaded do not give enough information.
hstan77 commentedon Mar 15, 2025
Hi
Issue now is the VR not created automatically , i can only do is retart network as above.
The whole Cloudstack , there is no function to create VR to the GuestNetwork . Or Manual activate Create VR .
The case now is when VR not automatically create, it will be trouble. How to create back VR?
hstan77 commentedon Mar 15, 2025
I do change IP those function:
2025-03-15 03:31:34,288 ERROR [o.c.n.r.d.RouterDeploymentDefinition] (Work-Job-Executor-151:ctx-e7cd657e job-489150/job-489152 ctx-ec97e82e) (logid:8b532c8b) Didn't support redundant virtual router without public network!
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Can't find at least one running router!
it just not auto create back the router.
hstan77 commentedon Mar 15, 2025
Finally i found the issue . This is Bug of Cloudstack.
The GuestNetwork , redundant=1 in Mysql, and when this happen , the VR will not auto create back . It keep say cannot create redundant . And the webinterface is no way to the status of the GuestNetwork is in redundant or not , and there is no way to turn off the redundant.
I edit mysql and set redundant=0, and restart network , and VR created back.
weizhouapache commentedon Mar 15, 2025
a bit strange, why is redundant 1 ?
is it a shared network or isolated network ? what's the network offering ?
hstan77 commentedon Mar 15, 2025
I am in shared network .
Network offering : [DefaultSharedNetworkOfferingWithSGService]
i think it may cause by when restart network time, i clicked "redundant" and "clean up", after VR destroyed, it cannot up back. I guess is this, need to test how to simulate this back.
1 remaining item
hstan77 commentedon Mar 16, 2025
This issue #491 is only mentioned need or no need the features.
Even now the option is there, the Error Handling seem not handle the issue well. It shall handle the issue of mistake no VR in GuestNetwork. At least have more detail in the log . At begining, I copied all the log , and nobody know what happening.
It is very lucky for me able to "Guess" the issue . Otherwise, this become unknown.
weizhouapache commentedon Mar 17, 2025
@hstan77
4.15 is EOL, I suggest you to upgrade to latest versions.
in recent versions, the
make redundantoption is not visible anymore, even for isolated networks.sureshanaparti commentedon Apr 14, 2025
@hstan77 can you check with latest version?
DaanHoogland commentedon May 12, 2025
fixed in #10612
mjstark commentedon Jun 10, 2026
restart the network and VR will be re-created