Repository navigation
Snapshot chain - garbage collection not working as expected #9501
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
UI
CLOUDSTACK VERSION
4.19.0.1
CONFIGURATION
XCP-Ng backend
OS / ENVIRONMENT
NA
SUMMARY
In Xenserver/XCP_NG backend, when a chain of snapshots exists, the cleanup of old snapshot files is not taking place correctly. Old snapshots contnue to exist, while some in the middle are destroyed and files are deleted as well, but older ones continue to exist.
STEPS TO REPRODUCE
I have not found a sure way to recreate this issue. Here is what I have though - Setup xcp-ng backend, ver 8.2. Set snapshot.delta.max = 7, setup a Volume with Daily Snapshot and Keep =3.
Over time, you will start seeing this discrepency
EXPECTED RESULTS
The Keep = 3 should get respected, and besides the required chain of the last 3 snapshots, all old snapshot files should physically be deleted.
ACTUAL RESULTS
old files continue to exist.
Example -
and Physical file view
I have Keep = 3, I expect that the Aug 6 and Aug 7 files will exist,
I also expect that Aug 5 file will exist (keep = 3), but since Aug 5 file is part of a chain (by looking at file size), all Files from Jul 30 to Aug 5 need to be kept.
However, Jul 3, 4 and 6 files should not exist. These files also show as "Ready" in the snapshot_store_ref above.
Activity
nxsbi commentedon Aug 8, 2024
Re- uploading screenshots as they did not show up as desired
I have Keep = 3, I expect that the Aug 6 and Aug 7 files will exist,
I also expect that Aug 5 file will exist (keep = 3), but since Aug 5 file is part of a chain (by looking at file size), all Files from Jul 30 to Aug 5 need to be kept.
However, Jul 3, 4 and 6 files should not exist. These files also show as "Ready" in the snapshot_store_ref .
So issue is - why are the snapshots for Jul5 and those between Jul 7 and Jul 29 destroyed, but not the Jul 3, 4 and 6.
PS - I did notice that the created date in the snapshot_store_ref shows as Jul 23 for many days. Not sure why...
File view:
snapshot_store_ref view
shwstppr commentedon Aug 8, 2024
@nxsbi can you please check if this is not due to #9446 and try 4.19.1.1 release?
nxsbi commentedon Aug 8, 2024
I read through #9446 - This is a separate issue. In that one, the whole chain is getting deleted, while in my issue, files are not getting deleted.
nxsbi commentedon Sep 5, 2024
@shwstppr @DaanHoogland - Is there a way to trigger the garbage collection (to clear up deleted snapshot files) manually? Can you provide info on how to trigger and monitor?
DaanHoogland commentedon Sep 5, 2024
@nxsbi , I am not sure. Can you delete those snapshots from the UI?
nxsbi commentedon Sep 5, 2024
@DaanHoogland - The old snapshots dont show up on the Web interface. In the UI it only shows the 3 snapshots (based on the snapshot policy setting). The Snapshot_store_ref table shows many old snapshots as "Ready".
To manually do the clean up I update the snapshots table for the ones I want to delete by setting the status = 'BackedUp' and then delete from UI.
Even after doing this, the snapshot_store_ref still shows it as Ready. So I have to run a manual update on that table as well.
11 remaining items
github-actions commentedon Aug 31, 2026
This issue is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.
weizhouapache commentedon Aug 31, 2026
Context: This assumes CloudStack 4.19.0.1, XCP-ng/XenServer 8.2,
snapshot.delta.max=7, with a daily snapshot policy and Keep=3.Diagnosis: Known bug. The retention/GC path is in
DefaultSnapshotStrategy.deleteSnapshotChain()/deleteSnapshotInfo()(engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java). When the retention sweep tries to delete a snapshot that is still a live "parent" in the VHD chain (i.e. it has achild),deleteSnapshotChain()correctly skips the physical delete to avoid corrupting the child's backing file — but its caller unconditionally firesSnapshot.Event.OperationSucceededanyway, flipping the top‑levelSnapshotDB record toDestroyedwhile the underlyingSnapshotDataStoreVOrow (and the VHD file itself) is leftdisplay=falsebut stillReady. Nothing in 4.19.x later revisits that row to retry cleanup once the child is retired, so it leaks indefinitely. This matches your report: some intermediate snapshots (no live child at delete time) get cleaned up fine, while others (still a live parent at delete time) are marked "gone" in the UI/DB but never actually removed from storage. Asnapshot.delta.max=7chain with Keep=3 makes this trigger often, since the oldest snapshot chosen for deletion frequently still has a live child somewhere in the still-growing 7-deep chain.Resolution:
Ready-row leak described here.Hiddenstate for "retired but still needed as chain backing storage." This landed in 4.22.1 — there is no backport to 4.19/4.20 at this time.snapshot_store_reffor rows withdisplay=0andstate=Readywhose parentsnapshotrow isDestroyed, confirm no active volume/template depends on that VHD, and clean up manually — do this carefully and only after confirming the chain is truly orphaned, since incorrect manual deletion on a live chain can cause data loss (the exact failure mode Fix snapshot chain being deleted on XenServer #9447 fixed).Sources: #9446 / #9447 (merged 4.19.2.0, commit
9033ab709e6); #12524 / #12597 (merged 4.22.1); code read at tag4.19.0.1inDefaultSnapshotStrategy.java.Next steps: If you can reproduce reliably after uprgade, attaching the
snapshot_store_refstate for one orphaned chain (snapshot IDs,state,display,parent_snapshot_id) here would help confirm this is the same mechanism rather than a distinct regression.