Repository navigation
After upgrade to 4.22.0.0 snapshot.delta.max not respected #12524
Description
problem
I upgraded to 4.22.0.0 on Nov 11. I have many volumes that are set to 3 snapshots max stored by my users.
I noticed that all snapshots since Nov 11 (all existing and new ones) seem to create a massive chain of snapshots.
I had 60+ days snapshot files on the disk for every volume with daily snapshots, while the snapshot.delta.max is only set to 7. And snapshot policy is 3 snapshots only.
It seems the full snapshot before Nov 11 in my case was the last full snapshot. After that there is no more full snapshots, only incremental snapshots.
What I see in database:
in snapshots table:
And in snapshot_store_ref (have hidden rows in between - but they all show as "Hidden"
Please help! I cannot even clear my storage which is now hit 95% used capacity and about to blow up!! Its tens of extra terabytes of data!! with no way to clear it other than deleting all snapshots which is a huge problem for users...
versions
ACS 4.22.0
XCP-NG 8.2
The steps to reproduce the bug
1.Upgraded from 4.19 .1 to 4.22
2.
3.
...
What to do about it?
Identify what is causing the snapshot.delta.max to not be respected.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
nvazquez commentedon Jan 27, 2026
Hi @nxsbi thanks for reporting the issue. An additional question, before the upgrade did you see these 'Hidden' entries on database, or were them removed?
On 4.21 the KVM incremental snapshot feature has been introduced (#8907) which seems to do a similar approach leaving Hidden entries on database but not removing them from storage, is it possible that its also the default behavior now for Xen/XCP? @sureshanaparti @JoaoJandre @DaanHoogland
nxsbi commentedon Jan 27, 2026
Hi @nvazquez
Before upgrade the data reflected only two values - Destroyed or BackedUp (or Ready)
I did not do any extra steps around this.
Related to this - I had originally reported a related issue - #9501
This one is same issue, except that now there are no Full snapshots every 7 days at all. I used to be able to determine which is a Full snapshot based on Parent ID = 0 (in snapshot_store_ref) and now I no longer see that.
I also reviewed the actual snapshot sizes and there is no large files except the one from before upgrade.
Not sure if any other xcp-ng + cloudstack users can comment
JoaoJandre commentedon Feb 4, 2026
@nvazquez As far as I remember the logic for snapshot creation and removal is the same for KVM and Xen. I'll try to reproduce the issue on 4.22 using XCP-ng.
JoaoJandre commentedon Feb 4, 2026
Hello, @nxsbi
I was able to reproduce the problem. This happens when the number of snapshots configured to be kept for the policies is lower than the
snapshot.delta.maxconfiguration, what happens is that the chain is never ended and thus the older snapshots are never removed in order to keep the newer ones valid. I will create a PR to fix this in 4.22.1.0.The workaround that I would suggest is setting the maximum number of snapshots for the user snapshot policies to be the same as the value of
snapshot.delta.max. This will ensure that a full snapshot chain is created and finished each cycle and the older snapshot chains will be deleted. Could you test and check if this workaround works for you?@nvazquez This affects only XCP-ng, the snapshot creation logic is different for KVM.
nvazquez commentedon Feb 5, 2026
Thanks @JoaoJandre for your investigation and the fix PR - will review it
nxsbi commentedon Feb 27, 2026
I can confirm that the workaround works..
Here is what I did:
Suggestion for fix ---
sureshanaparti commentedon Apr 10, 2026
Fixed in #12597