Repository navigation
Deleting a storage object created via createLargeObject() won't delete the segments that make up the file #286
Description
If you create an object via createLargeObject() and then delete it via this library, it won't actually delete all the segments that make up the object, just the manifest.
I believe there is a way to do the delete request such that OpenStack will delete the segments for you when deleting the object, but I believe that depends on your OpenStack instance having particular middleware. I'm not 100% on either of those points, unfortunately. :-/
In any case, it is possible to ask a manifest where it's segments are, by doing a HEAD on the object and looking at the 'X-Object-Manifest' header. Unfortunately, there isn't a super easy way to get that header - see #285
Currently, in my application, I have a wrapper around delete that first does a HEAD on the object, checks the 'X-Object-Manifest' header, then lists and deletes all the segments before deleting the manifest. However, it would be really, really great, if this library could handle that automatically! There is builtin support for uploading large objects (via createLargeObject()) so IMO there should be support for deleting them as well.
Activity
dsnopek commentedon Jul 16, 2019
A little bit of research follows...
Looking at the code for python-swiftclient (which I know does magically delete the segments for a large object) it appears to grab the 'X-Object-Manifest' header and delete the segments itself (as opposed to asking Swift to do it on the server-side):
https://github.com/openstack/python-swiftclient/blob/2fcd4d872713dc30e7352845c37515280f1d21ab/swiftclient/service.py#L2515
For SLO objects (up until now we're talking about DLO objects) it does ask Swift to delete the underlying segments for it, using a special query string parameter (
"multipart-manifest=delete"). I don't necessarily think this library needs to support deleting SLO objects, since it only supports creating DLO ones, but that could be a nice feature at some point.Another interesting thing about the Python implementation, is that it uses a thread pool to delete the segments, which would probably be great for performance. If you have 500 segments and each request takes 500ms, it would take 250 seconds to do them serially, but only 50 seconds if done in parallel with 5 threads.
dsnopek commentedon Jul 16, 2019
Based on the above research, I'd like to propose the following:
delete()method get a new argument likedelete(delete_segments=FALSE)which, if set to true, will grab the 'X-Object-Manifest' header via whatever gets implemented in Can't easily get ALL headers for a storage object #285, and then serially deletes the segments before deleting the manifest (doing it in parallel with a thread pool could be done manually by the library user if they choose)The reason I'm proposing that we don't delete the segments by default is that it could be very slow. Although, if you guys aren't worried about that, I think it'd be better for the usability of this library if it defaulted to deleting the segments too.
I'll try to work on some PRs later...
Lejo1 commentedon Jul 10, 2023
Hey the issue still exists after this long time.
I for example encounter it on my nextcloud, large files in the _segements bucket are just never deleted.
I would really appreciate if you could look into this again.