Repository navigation
snapshot as Abnormal residue on rbd #12002
Description
problem
when use ceph rbd store to primary store. and turn off [snapshot.backup.to.secondary]
take Instance volume snapshot, and then Destroy Instance and expunge,
the snapshot will Abnormal residue.
because Destroy Instance and expunge, will remove rbd image and its snap. after this the cloudstack job go [ StorpoolSnapshotStrategy] to delete snap, however ,this time the snap is not exit
so cloudstack will hung,and the snapshot whill Abnormal residue in MySQL and UI. and can not delete!
versions
CloudStack 4.21.0.0
The steps to reproduce the bug
1.use ceph rbd store to primary store.
2.turn off [snapshot.backup.to.secondary]
3.take a Instance volume snapshot
4.Destroy Instance and expunge,
5. check the snapshot
What to do about it?
to fix this ,you can fix the msyql snapshot ‘status’ ，fix ‘backedup' to 'CreatedOnPrimary'，it may work
Activity
boring-cyborg commentedon Nov 6, 2025
Thanks for opening your first issue here! Be sure to follow the issue template!
[-]when use rbd store , Take Instance volume snapshot, then Destroy Instance,the snapchot is Abnormal residue[/-][+]snapshot as Abnormal residue on rbd[/+]Damans227 commentedon Mar 13, 2026
@lubxun @DaanHoogland @abh1sar Ok, just reproduced and investigated this issue.. so, basically, when
snapshot.backup.to.secondary=false, snapshots live only on primary as RBD snapshots.And as per Ceph docs, an RBD image can't be removed until its snapshots are removed; CloudStack handles this correctly by purging all RBD snapshots before deleting the image. So no orphans on the Ceph side BUT the DB retains stale
snapshotsandsnapshot_store_refrecords for these primary-only snapshots even after the data is gone from Ceph.Since there are no copies elsewhere, these entries become undeletable in the UI (fails with "Problem with condition: state" in DefaultSnapshotStrategy.java#L438).
Proposed fix:
Imo, it's best to clean up snapshot DB records for primary-only snapshots before the volume is deleted as they'll be destroyed with the image anyway.
kvm: delete primary-only RBD snapshots before the volume