You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When trying to delete a Template, it fails with a message like "Error Unable to delete template with id: 294 because VM instances: [VM instance {"id":2972,"instanceName":"i-39 ..." (continuing with a long list beyond the screen).
This can be overridden using the "force" delete option, leaving a bad feeling of "doing something dangerous".
STEPS TO REPRODUCE
Create a VM template (e.g. import ISO, install VM, make Template from the root device).
Create a new VM from this template
Try to delete the template
EXPECTED RESULTS
A template should be deletable at any time without error, unless there is a actual reason against it. (which could then still be overridden using the "force" option).
ACTUAL RESULTS
Error message (see above). This gives me a clear feeling of "doing something bad", without any explanation.
This suggests that there is a hidden connection between a template and VMs created from it after creation, which the" force" option would just leave dangling.
Hmm, reading the discussion and the code, it looks more like the default value for the force parameter (if not specified) was changed from true to false, which would actually make increase the issue for me. (By also affecting the API directly, not only the Web UI which seems to default to false already)
I tried to delete the templates using the Web GUI, leaving the force in the default "false" position.
I still fail to understand what would speak against always setting "force=true", making that flag redundant?
2023-10-30 12:51:32,575 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-36:ctx-aa4d8787 job-97867) (logid:f3b3e108) Complete async job-97867, jobStatus: FAILED, resultCode: 431, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"431","errortext":"Unable to delete template with id: 322 because VM instances: [VM instance {"id":XXXX,"instanceName":"YYYYY","type":"User","uuid":"f8a0c75b-784d-4490-8e29-7fd9f179a4ca"} (and lots more) ] are using it."}
I do not understand how running VMs are using templates. On XenServer, the ROOT volume ist copied from the template image; afterwards, I see no connection between the running VM and the template.
Maybe there are other hypervisors with such a relation (e.g. running VMs in a copy-on-write fashion laid over the active template image), but with Xen/XCP that's not the case (and would be veeeery slow from NFS secondary storage if it was the case).
thin provisioned disks (copy on write principle) would indeed explain this. You could make sure none of you service offering/ disk offering are using this.
As of now we're still on XCP-ng 7.6, but the upgrade to 8.2 is indeed currently underway by our team. So I cannot check stuff on the Pool for the next hours/days.
The only Secondary Storage we have is a NFS share, so am quite certain a real copy-on-write deployment is not used here currently.
Until now, thin provisioning on Xen meant allocating only the blocks containing actual data (i.e. actually allocating the storage only on the first write), not any copy-on-write scheme.
I do not know if XCP-ng 8.2 brings any new features regarding that. But still, LVM2-backed copy-on-write has to work on a blockdevice-level (local volumes, iSCSI,...), I doubt it could use a filesystem-level image (on a NFS share) as a lower layer.
It just came to me why this is working this way @MartinEmrich ; For a running instance you can say reinstall and for that you would need the original template. I agree that the message could be somewhat more friendly but the intention is good ;)
I am not sure if we could get away with just changing documentation for this. Returning a list of VMs using the template would be useful for an operator I'd say, to help make sure if they want to force delete.
Maybe a "fix" could indeed be documentation-only: a sentence like "Deleting templates used by VMs is perfectly safe, but the VM can no longer be reinstalled from the deleted image".
And then, if the user accepts that and force-deletes old images, the Reinstall button will disappear and the restoreVirtualMachine command will obviously return some template-not-found error message.
@MartinEmrichforce param is probably there to prevent wrongful deletion of template. As some operations such as VM restore rely on the availability of the template personally I feel it is not correct to right away delete the template when there are active VMs that use the template. User can always delete the template even in that condition by specifying force=true. From the content of your issue I feel we probably need to make the error message more meaningful.
@shwstppr yes I agree, I just would love to see it made more clear what "in use" actually means.
I use CloudStack for about 10 years now, and always thought I would actually break the running VMs because the template is "in use by them", so there's a big heap of old images.
Only now after doing some digging (resulting in this issue), I notice that keeping them makes no sense for us.
Activity
DaanHoogland commentedon Oct 30, 2023
@MartinEmrich, #7731 should address your concern.
MartinEmrich commentedon Oct 30, 2023
Hmm, reading the discussion and the code, it looks more like the default value for the force parameter (if not specified) was changed from true to false, which would actually make increase the issue for me. (By also affecting the API directly, not only the Web UI which seems to default to false already)
I tried to delete the templates using the Web GUI, leaving the force in the default "false" position.
I still fail to understand what would speak against always setting "force=true", making that flag redundant?
DaanHoogland commentedon Oct 30, 2023
right? do you see any stacktraces in the logs during the call to delete
MartinEmrich commentedon Oct 30, 2023
No stacktraces in the logs.
This is the error message:
I do not understand how running VMs are using templates. On XenServer, the ROOT volume ist copied from the template image; afterwards, I see no connection between the running VM and the template.
Maybe there are other hypervisors with such a relation (e.g. running VMs in a copy-on-write fashion laid over the active template image), but with Xen/XCP that's not the case (and would be veeeery slow from NFS secondary storage if it was the case).
DaanHoogland commentedon Oct 30, 2023
thin provisioned disks (copy on write principle) would indeed explain this. You could make sure none of you service offering/ disk offering are using this.
what xcp version are you using?
MartinEmrich commentedon Oct 30, 2023
As of now we're still on XCP-ng 7.6, but the upgrade to 8.2 is indeed currently underway by our team. So I cannot check stuff on the Pool for the next hours/days.
The only Secondary Storage we have is a NFS share, so am quite certain a real copy-on-write deployment is not used here currently.
Until now, thin provisioning on Xen meant allocating only the blocks containing actual data (i.e. actually allocating the storage only on the first write), not any copy-on-write scheme.
I do not know if XCP-ng 8.2 brings any new features regarding that. But still, LVM2-backed copy-on-write has to work on a blockdevice-level (local volumes, iSCSI,...), I doubt it could use a filesystem-level image (on a NFS share) as a lower layer.
DaanHoogland commentedon Oct 30, 2023
It just came to me why this is working this way @MartinEmrich ; For a running instance you can say reinstall and for that you would need the original template. I agree that the message could be somewhat more friendly but the intention is good ;)
I am not sure if we could get away with just changing documentation for this. Returning a list of VMs using the template would be useful for an operator I'd say, to help make sure if they want to force delete.
MartinEmrich commentedon Oct 30, 2023
Ah ok, that makes sense.
Maybe a "fix" could indeed be documentation-only: a sentence like "Deleting templates used by VMs is perfectly safe, but the VM can no longer be reinstalled from the deleted image".
And then, if the user accepts that and force-deletes old images, the Reinstall button will disappear and the restoreVirtualMachine command will obviously return some template-not-found error message.
shwstppr commentedon Oct 30, 2023
@MartinEmrich
forceparam is probably there to prevent wrongful deletion of template. As some operations such as VM restore rely on the availability of the template personally I feel it is not correct to right away delete the template when there are active VMs that use the template. User can always delete the template even in that condition by specifying force=true. From the content of your issue I feel we probably need to make the error message more meaningful.MartinEmrich commentedon Oct 30, 2023
@shwstppr yes I agree, I just would love to see it made more clear what "in use" actually means.
I use CloudStack for about 10 years now, and always thought I would actually break the running VMs because the template is "in use by them", so there's a big heap of old images.
Only now after doing some digging (resulting in this issue), I notice that keeping them makes no sense for us.
DaanHoogland commentedon Oct 31, 2023
So now we have two issues if I understand correctly:
@MartinEmrich can you make a PR for the documentation change?
Clarify consequences of deleting templates in use by VMs.
MartinEmrich commentedon Oct 31, 2023
Yes, I gave it a try: apache/cloudstack-documentation#355
DaanHoogland commentedon Nov 1, 2023
@MartinEmrich can we close this issue?
MartinEmrich commentedon Nov 2, 2023
yes