Skip to content

Can't create or scale VM to more then 2 vcpus. XCP-NG. #7375

Closed
#7887
Closed
Can't create or scale VM to more then 2 vcpus. XCP-NG.#7375
#7887
Assignees
shwstppr
Labels
Severity:Majorcomponent:XenServercomponent:computestatus:has-work-aroundstatus:needs-investigation
Milestone
4.18.1.0

Description

@furyflash777
furyflash777
opened on Mar 28, 2023
ISSUE TYPE
  • Bug Report
COMPONENT NAME
UI
CLOUDSTACK VERSION
4.18.0
CONFIGURATION
OS / ENVIRONMENT

XCP-NG 8.2.1

Global parameter xen.vm.vcpu.max set to 64

Host has 2 sockets and 16 cores each.

SUMMARY

Unable to start VM from different templates (Debian, Ubuntu) when vcpu more then 2.
VM with attached iso started successfully with more then 2 vcpu.

Task failed! Task record: uuid: 8d2ad021-9c6d-daaa-899a-a55867cd49e7
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue Mar 28 15:08:19 MSK 2023
finished: Tue Mar 28 15:08:19 MSK 2023
status: failure
residentOn: com.xensource.xenapi.Host@a937b48e
progress: 1.0
type:
result:
errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []

    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:508)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:5143)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:111)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

GUI error:

Unable to start a VM [e430f6d7-f13e-47b0-8f7c-d8fcc491e268] due to [Unable to create a deployment for VM instance {"id":148,"instanceName":"i-8-148-VM","type":"User","uuid":"e430f6d7-f13e-47b0-8f7c-d8fcc491e268"}].

STEPS TO REPRODUCE
Create VM with 3 or more CPUs
EXPECTED RESULTS
VM created successfully
ACTUAL RESULTS
VM has failed result

Activity

  1. sbrueseke commented on Mar 29, 2023

    @sbrueseke
    sbrueseke
    on Mar 29, 2023

    I am not able to reproduce this error. I am able to start a VM with more than 3 vCPUs.
    You can set the xen.vm.vcpu.max on Cluster level, too. Did you check the value there?

  2. DaanHoogland
    added
    component:XenServer
    on Mar 30, 2023

  3. DaanHoogland commented on Mar 30, 2023

    @DaanHoogland
    DaanHoogland
    on Mar 30, 2023
    Contributor

    @furyflash777 I think you need to add more information.
    for instance: can you deploy such a VM directly on Xen?
    can you give the full stacktrace including any messages that have the same job- and/or ctx-?

  4. DaanHoogland
    added
    component:compute
    on Mar 30, 2023

  5. furyflash777 commented on Apr 6, 2023

    @furyflash777
    furyflash777
    on Apr 6, 2023
    Author

    @furyflash777 I think you need to add more information. for instance: can you deploy such a VM directly on Xen? can you give the full stacktrace including any messages that have the same job- and/or ctx-?

    Hello,
    New information :
    Compute offerings without Dynamic Scale have not this problem.

    2023-04-07 01:21:07,477 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-142:ctx-24901669) (logid:ecfc77dc) Seq 5-7472316206737908984: Executing request
    2023-04-07 01:21:07,481 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) 1. The VM i-6-202-VM is in Starting state.
    2023-04-07 01:21:07,482 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) no guest OS type, start it as HVM guest
    2023-04-07 01:21:07,486 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) No recommended value found for dynamic min
    2023-04-07 01:21:07,486 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) No recommended value found for dynamic max, setting static max and dynamic max equal
    2023-04-07 01:21:07,497 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Created VM 6c068ca5-e6fc-b712-19b9-908a6cc81268 for i-6-202-VM
    2023-04-07 01:21:07,502 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Setting boottype=BIOS and bootmode=null for VM: 6c068ca5-e6fc-b712-19b9-908a6cc81268
    2023-04-07 01:21:07,511 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) VBD a8dba020-d853-2bac-455c-d337946533a7 created for com.cloud.agent.api.to.DiskTO@1a091b58
    2023-04-07 01:21:07,520 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) VBD c98a6d77-647e-1472-97f8-cb20d43714bc created for com.cloud.agent.api.to.DiskTO@2839770
    2023-04-07 01:21:07,520 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Creating VIF for i-6-202-VM on nic [Nic:Guest-10.21.66.19-vlan://1836]
    2023-04-07 01:21:07,521 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Looking for network named trunk
    2023-04-07 01:21:07,529 DEBUG [c.c.h.x.r.XsLocalNetwork] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Found a network called trunk on host=F.A.K.E; Network=eed1eedf-986c-b05b-fd85-d91163e91343; pif=a9a5169b-163c-268d-c000-a827754c6542
    2023-04-07 01:21:07,532 DEBUG [c.c.h.x.r.XsLocalNetwork] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Found a network called VLAN-eed1eedf-986c-b05b-fd85-d91163e91343-1836 on host=F.A.K.E; Network=be8fa6aa-b6fb-a498-7381-f62f69337b98; pif=5f5219c1-4a9
    a-2e73-9d8b-2f38b20f47e6
    2023-04-07 01:21:07,538 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Created a vif 809ceb04-8dc6-ec0b-5e07-686dbe4b8b1e on 0
    2023-04-07 01:21:08,555 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []

    2023-04-07 01:21:08,562 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Unable to start VM(i-6-202-VM) on host(3605b9c3-e0b5-4a94-95b1-9220f426395d) due to Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-
    6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []

    Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []

        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:508)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:5143)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:111)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

    2023-04-07 01:21:08,568 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) unable to destroy task(com.xensource.xenapi.Task@cfeb0182) on host(3605b9c3-e0b5-4a94-95b1-9220f426395d) due to You gave an invalid object reference.
    The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
    2023-04-07 01:21:08,570 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Catch Exception: class com.cloud.utils.exception.CloudRuntimeException due to com.cloud.utils.exception.CloudRuntimeException: Unable to sta
    rt VM(i-6-202-VM) on host(3605b9c3-e0b5-4a94-95b1-9220f426395d) due to Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []

    com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(i-6-202-VM) on host(3605b9c3-e0b5-4a94-95b1-9220f426395d) due to Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []

        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:5162)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:111)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

    2023-04-07 01:21:08,571 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Unable to start i-6-202-VM due to
    com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(i-6-202-VM) on host(3605b9c3-e0b5-4a94-95b1-9220f426395d) due to Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f
    subtasks: []

        at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:5162)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:111)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

    2023-04-07 01:21:08,571 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Unable to start i-6-202-VM due to
    com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(i-6-202-VM) on host(3605b9c3-e0b5-4a94-95b1-9220f426395d) due to Task failed! Task record: uuid: 77dddee7-8a5c-cfb5-6e92-b5805af1dcb4
    nameLabel: Async.VM.start_on
    nameDescription:
    allowedOperations: []
    currentOperations: {}
    created: Fri Apr 07 01:21:30 MSK 2023
    finished: Fri Apr 07 01:21:30 MSK 2023
    status: failure
    residentOn: com.xensource.xenapi.Host@a937b48e
    progress: 1.0
    type:
    result:
    errorInfo: [VALUE_NOT_SUPPORTED, VCPUs-at-startup, 8, value greater than VCPUs-max]
    otherConfig: {}
    subtaskOf: com.xensource.xenapi.Task@aaf13f6f

            subtasks: []

    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:5162)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:111)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

    2023-04-07 01:21:08,591 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Unable to clean up VBD due to
    You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
    at com.xensource.xenapi.Types.checkResponse(Types.java:693)
    at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
    at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
    at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3677)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:222)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    2023-04-07 01:21:08,593 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Unable to clean up VBD due to
    You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
    at com.xensource.xenapi.Types.checkResponse(Types.java:693)
    at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
    at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
    at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3677)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:222)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

    2023-04-07 01:21:08,597 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Unable to cleanup VIF
    You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
    at com.xensource.xenapi.Types.checkResponse(Types.java:693)
    at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
    at com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
    at com.xensource.xenapi.VIF.unplug(VIF.java:921)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3685)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:222)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:57)
    at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
    at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1782)
    at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
    at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
    at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    2023-04-07 01:21:08,603 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) The VM is in stopped state, detected problem during startup : i-6-202-VM
    2023-04-07 01:21:08,603 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Seq 5-7472316206737908984: Response Received:
    2023-04-07 01:21:08,604 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (DirectAgent-142:ctx-24901669) (logid:7a9ab01d) Seq 5-7472316206737908984: MgmtId 46327103290396: Resp: Routing to peer
    2023-04-07 01:21:09,419 DEBUG [c.c.a.t.Request] (AgentManager-Handler-6:null) (logid:) Seq 5-7472316206737908985: Executing: { Cmd , MgmtId: 46327103290396, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCle
    anup":"false","forceStop":"false","vlanToPersistenceMap":{"1836":"true"},"volumesToDisconnect":[],"vmName":"i-6-202-VM","executeInSequence":"false","wait":"0","bypassHostMaintenance":"false"}}] }

  6. DaanHoogland
    added
    status:needs-investigation
    on Jun 22, 2023
  7. DaanHoogland
    added this to the 4.18.1.0 milestone on Jun 22, 2023
  8. DaanHoogland
    added
    Severity:Major
    on Jun 22, 2023
  9. weizhouapache
    self-assigned this
    on Jul 28, 2023

  10. weizhouapache commented on Aug 14, 2023

    @weizhouapache
    weizhouapache
    on Aug 14, 2023
    Member

    I had a quick investigation

    in PR #4672, the vcpu max is set to min of host cpu socket

    cloudstack/plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/XenServerGuru.java

    Lines 99 to 106 in 0574087

    List<HostVO> clusterHosts = hostDao.listByClusterAndHypervisorType(host.getClusterId(), host.getHypervisorType());
    HostVO hostWithMinSocket = clusterHosts.stream().min(Comparator.comparing(HostVO::getCpuSockets)).orElse(null);
    Integer vCpus = MaxNumberOfVCPUSPerVM.valueIn(host.getClusterId());
    if (hostWithMinSocket != null && hostWithMinSocket.getCpuSockets() != null &&
    hostWithMinSocket.getCpuSockets() < vCpus) {
    vCpus = hostWithMinSocket.getCpuSockets();
    }
    to.setVcpuMaxLimit(vCpus);

    if dynamic scaling is enabled, the vcpu max is set to the above value, which might be smaller than the vcpus of vm instance.

    cloudstack/plugins/hypervisors/xenserver/src/main/java/com/cloud/hypervisor/xenserver/resource/CitrixResourceBase.java

    Lines 1331 to 1343 in 0574087

    if (isDmcEnabled(conn, host) && vmSpec.isEnableDynamicallyScaleVm()) {
    // scaling is allowed
    vmr.memoryStaticMin = getStaticMin(vmSpec.getOs(), vmSpec.getBootloader() == BootloaderType.CD, vmSpec.getMinRam(), vmSpec.getMaxRam(), recommendedMemoryMin);
    vmr.memoryStaticMax = getStaticMax(vmSpec.getOs(), vmSpec.getBootloader() == BootloaderType.CD, vmSpec.getMinRam(), vmSpec.getMaxRam(), recommendedMemoryMax);
    vmr.memoryDynamicMin = vmSpec.getMinRam();
    vmr.memoryDynamicMax = vmSpec.getMaxRam();
    if (guestOsTypeName.toLowerCase().contains("windows")) {
    vmr.VCPUsMax = (long)vmSpec.getCpus();
    } else {
    if (vmSpec.getVcpuMaxLimit() != null) {
    vmr.VCPUsMax = (long)vmSpec.getVcpuMaxLimit();
    }
    }

    @shwstppr
    can you please have look ?

  11. weizhouapache
    assigned
    shwstppr
    and unassigned
    weizhouapache
    on Aug 14, 2023

  12. shwstppr commented on Aug 16, 2023

    @shwstppr
    shwstppr
    on Aug 16, 2023
    Contributor

    @weizhouapache will have a look later this week or early next week

  13. weizhouapache commented on Aug 16, 2023

    @weizhouapache
    weizhouapache
    on Aug 16, 2023
    Member

    @weizhouapache will have a look later this week or early next week

    @shwstppr
    cool, thanks.

  14. weizhouapache
    added a commit that references this issue on Aug 21, 2023

    xen/xcpng: revert java changes in PR apache#4672

    a83b902
  15. weizhouapache
    linked a pull request that will close this issuexen/xcpng: set vmr.VCPUsMax to minimum of global setting and host cpu cores #7887on Aug 21, 2023

  16. weizhouapache commented on Aug 21, 2023

    @weizhouapache
    weizhouapache
    on Aug 21, 2023
    Member

    @shwstppr
    I have created a draft PR #7887 which revert java changes in PR#4672

  17. shwstppr commented on Aug 21, 2023

    @shwstppr
    shwstppr
    on Aug 21, 2023
    Contributor

    @weizhouapache thanks. At that time the original code was causing some issues with the VM deployment and migration. We may have to keep an eye on the smoke-tests

  18. weizhouapache commented on Aug 21, 2023

    @weizhouapache
    weizhouapache
    on Aug 21, 2023
    Member

    @weizhouapache thanks. At that time the original code was causing some issues with the VM deployment and migration. We may have to keep an eye on the smoke-tests

    thanks for the info @shwstppr
    I will run some tests on xcpng-82

  19. weizhouapache
    removed a link to a pull requestxen/xcpng: set vmr.VCPUsMax to minimum of global setting and host cpu cores #7887on Aug 23, 2023
  20. weizhouapache
    linked a pull request that will close this issuexen/xcpng: set vmr.VCPUsMax to minimum of global setting and host cpu cores #7887on Aug 25, 2023

  21. weizhouapache commented on Aug 28, 2023

    @weizhouapache
    weizhouapache
    on Aug 28, 2023
    Member

    a workaround as @furyflash777 mentioned:

    Compute offerings without Dynamic Scale have not this problem.
  22. weizhouapache
    added
    status:has-work-around
    on Aug 28, 2023

  23. harikrishna-patnala commented on Aug 28, 2023

    @harikrishna-patnala
    harikrishna-patnala
    on Aug 28, 2023
    Member

    value greater than VCPUs-max

    @weizhouapache I think what we are configuring about vCPUs is correct but instead of failing the VM deployment at XenServer, we should check if vCPU max is greater than the compute offering value and warn accordingly.

    Also while setting the vCPU max by comparing "xen.vm.vcpu.max" and the minimum vCPUs available in the cluster we should log the information for debugging purposes.

  24. weizhouapache commented on Aug 28, 2023

    @weizhouapache
    weizhouapache
    on Aug 28, 2023
    Member

    value greater than VCPUs-max

    @weizhouapache I think what we are configuring about vCPUs is correct but instead of failing the VM deployment at XenServer, we should check if vCPU max is greater than the compute offering value and warn accordingly.

    Also while setting the vCPU max by comparing "xen.vm.vcpu.max" and the minimum vCPUs available in the cluster we should log the information for debugging purposes.

    @harikrishna-patnala
    actually cloudstack checks the host cpu cores and cpu speed in cloudstack code

    cloudstack/server/src/main/java/com/cloud/capacity/CapacityManagerImpl.java

    Lines 413 to 416 in f563c16

    // Check host can support the Cpu Number and Speed.
    Host host = _hostDao.findById(hostId);
    boolean isCpuNumGood = host.getCpus().intValue() >= cpuNum;
    boolean isCpuSpeedGood = host.getSpeed().intValue() >= cpuSpeed;

    I doubt the check in PR#4672 is needed.
    Also, I think the vcpu max should not be set to the max cpu sockets, but the max cpu cores (cpu sockets * cpu cores per socket).

  25. harikrishna-patnala commented on Aug 28, 2023

    @harikrishna-patnala
    harikrishna-patnala
    on Aug 28, 2023
    Member

    value greater than VCPUs-max

    @weizhouapache I think what we are configuring about vCPUs is correct but instead of failing the VM deployment at XenServer, we should check if vCPU max is greater than the compute offering value and warn accordingly.
    Also while setting the vCPU max by comparing "xen.vm.vcpu.max" and the minimum vCPUs available in the cluster we should log the information for debugging purposes.

    @harikrishna-patnala actually cloudstack checks the host cpu cores and cpu speed in cloudstack code

    cloudstack/server/src/main/java/com/cloud/capacity/CapacityManagerImpl.java

    Lines 413 to 416 in f563c16

    // Check host can support the Cpu Number and Speed.
    Host host = _hostDao.findById(hostId);
    boolean isCpuNumGood = host.getCpus().intValue() >= cpuNum;
    boolean isCpuSpeedGood = host.getSpeed().intValue() >= cpuSpeed;

    I doubt the check in PR#4672 is needed. Also, I think the vcpu max should not be set to the max cpu sockets, but the max cpu cores (cpu sockets * cpu cores per socket).

    Okay I see your point about the check which we are doing it before any ways @weizhouapache

    About the max value to be set, https://support.citrix.com/article/CTX236977/overcommitting-pcpus-on-individual-xenserver-vms this document says "Citrix recommends that you do not run a VM with more virtual CPUs (vCPUs) than the number physical CPUs (pCPUs) available on the XenServer host"
    I think this is the reason we are setting based on host details.

  26. weizhouapache commented on Aug 28, 2023

    @weizhouapache
    weizhouapache
    on Aug 28, 2023
    Member

    Okay I see your point about the check which we are doing it before any ways @weizhouapache

    About the max value to be set, https://support.citrix.com/article/CTX236977/overcommitting-pcpus-on-individual-xenserver-vms this document says "Citrix recommends that you do not run a VM with more virtual CPUs (vCPUs) than the number physical CPUs (pCPUs) available on the XenServer host"
    I think this is the reason we are setting based on host details.

    understood, thanks @harikrishna-patnala for the information.

    The following check applies on all hypervisor types, which might be good enough.

    boolean isCpuNumGood = host.getCpus().intValue() >= cpuNum;

  27. harikrishna-patnala commented on Aug 28, 2023

    @harikrishna-patnala
    harikrishna-patnala
    on Aug 28, 2023
    Member

    Agreed @weizhouapache admin anyways have the control of it using the cluster/global level setting, so it should be fine.

  28. DaanHoogland commented on Aug 28, 2023

    @DaanHoogland
    DaanHoogland
    on Aug 28, 2023
    Contributor

    implemented in #7887

  29. DaanHoogland
    closed this as completedon Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Severity:Majorcomponent:XenServercomponent:computestatus:has-work-aroundstatus:needs-investigation

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions