Skip to content

S3 - Previews not deleted #20344

Closed
Closed
S3 - Previews not deleted#20344
Labels
0. Needs triagePending check for reproducibility or if it fits our roadmapbugfeature: object storagefeature: previews and thumbnailsneeds info

Description

@SimplyCorbett
SimplyCorbett
opened on Apr 7, 2020

This is related to #20333

When removing images previews are not deleted from the S3 bucket if using primary storage.

Before upload:

"num_objects": 1421

After uploading 10 images and generating previews:

"num_objects": 1451

With previews that brings it up to 30 total images/files.

After removing them, deleting the trashbin and running the S3 garbage collection process this is the amount of files leftover:

"num_objects": 1441

Previews are not removed and I didn't realize this because my primary use case is not images.

Configuration:
18.0.3
Primary S3 multibucket storage | path style false
Debian 10
PHP 7.4-FPM
HAProxy + Nginx
Postgresql

Activity

  1. SimplyCorbett
    added
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    bug
    on Apr 7, 2020
  2. solracsf
    added
    feature: object storage
    on Apr 17, 2020

  3. MorrisJobke commented on Jul 29, 2020

    @MorrisJobke
    MorrisJobke
    on Jul 29, 2020
    Member

    Mind to run this query: (it's might be a bit costly, but it should give us some hints if there is already a mismatch in the DB or just on the object store)

    SELECT fileids FROM (
	SELECT DISTINCT(SUBSTRING_INDEX(SUBSTRING_INDEX(f1.path, '/', -2), '/', 1)) AS fileids
	FROM oc_filecache f1
	WHERE f1.storage = (SELECT numeric_id FROM oc_storages WHERE id LIKE "local::%" LIMIT 1) AND f1.path LIKE 'appdata_%/preview/%.png'
) as r
WHERE r.fileids NOT IN (SELECT f2.fileid FROM oc_filecache f2);

    If you have multiple local storages (when using the external storage app with local storage for example) then maybe replace the (SELECT numeric_id FROM oc_storages WHERE id LIKE "local::%" LIMIT 1) with the specific numeric ID of local::/YOUR/DATA/DIR/PATH.

    This query basically extracts all the file IDs from the preview paths and then checks them against the entries in the file cache. That then should give us a hint on which previews exist for files that don't exist anymore.

  4. solracsf commented on Nov 24, 2020

    @solracsf
    solracsf
    on Nov 24, 2020
    Member

    @MorrisJobke as i'm having these issues (as pointed in the 1st post on #20333 ) your query returns an empty result:

    MariaDB [nextcloud]> SELECT fileids FROM (
    -> SELECT DISTINCT(SUBSTRING_INDEX(SUBSTRING_INDEX(f1.path, '/', -2), '/', 1)) AS fileids
    -> FROM oc_filecache f1
    -> WHERE f1.storage = (SELECT numeric_id FROM oc_storages WHERE id LIKE "local::%" LIMIT 1) AND f1.path LIKE 'appdata_%/preview/%.png'
    -> ) as r
    -> WHERE r.fileids NOT IN (SELECT f2.fileid FROM oc_filecache f2);
Empty set (0.004 sec)

    I don't think this are (only) previews, as i pointed at #20333 (comment) because i can see objects in the bucket >10Mb so clearly these are not previews.

  5. SimplyCorbett commented on Nov 25, 2020

    @SimplyCorbett
    SimplyCorbett
    on Nov 25, 2020
    Author

    I don't think this are (only) preview

    Then your issue is different from mine.

    I'm no longer on S3 though so I'm going to go ahead and close this ticket.

  6. SimplyCorbett
    closed this as completedon Nov 25, 2020

  7. fnkr commented on Nov 25, 2020

    @fnkr
    fnkr
    on Nov 25, 2020

    @SimplyCorbett Can you please keep this open? It's still an issue. You can unsubscribe if you want. :)

  8. solracsf
    reopened this on Nov 25, 2020

  9. SimplyCorbett commented on Nov 26, 2020

    @SimplyCorbett
    SimplyCorbett
    on Nov 26, 2020
    Author

    @SimplyCorbett Can you please keep this open? It's still an issue. You can unsubscribe if you want. :)

    Is it? Nobody else has actually posted anything that replicates my issue.

    Is the issue what @acsfer posted in #20333 or is the issue preview-only as mentioned in my OP?

  10. solracsf commented on Nov 26, 2020

    @solracsf
    solracsf
    on Nov 26, 2020
    Member

    I can replicate your exact issue. I'm just saying this is bigger than simply the previews with my test case.

    Please keep in mind that Git issues are not your issues or my issues but issues that are lacking triage.

    You're not using S3 anymore so you can't contribute anymore to debug it. No problem. Just unsubscribe from it and thanks for having report it 👍

  11. SimplyCorbett commented on Nov 26, 2020

    @SimplyCorbett
    SimplyCorbett
    on Nov 26, 2020
    Author

    I can replicate your exact issue. I'm just saying this is bigger than simply the previews with my test case.

    Please keep in mind that Git issues are not your issues or my issues but issues that are lacking triage.

    You're not using S3 anymore so you can't contribute anymore to debug it. No problem. Just unsubscribe from it and thanks for having report it +1

    Well, I stopped using S3 because of this issue so I'm still invested.

    But I did not have the issues you mention in #20333 so perhaps you are seeing a placebo effect with this one?

    Or one of the nextcloud updates introduced a new problem..

    I'll go ahead and keep this open then.

  12. szaimen commented on Jun 18, 2021

    @szaimen
    szaimen
    on Jun 18, 2021
    Contributor

    Related #26014

  13. szaimen
    added
    1. to developAccepted and waiting to be taken care of
    feature: previews and thumbnails
    and removed
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    on Jun 18, 2021

  14. charleypaulus commented on Oct 14, 2022

    @charleypaulus
    charleypaulus
    on Oct 14, 2022

    I have the same issue with Swift primary object storage.

  15. charleypaulus commented on Oct 28, 2022

    @charleypaulus
    charleypaulus
    on Oct 28, 2022

    I wrote this script. It:

    • gets from database all previews that no longer have their full original image
    • checks that these full images are actually absent from object storage
    • deletes these previews both from object storage and from database

  16. szaimen commented on Jan 9, 2023

    @szaimen
    szaimen
    on Jan 9, 2023
    Contributor

    Hi, please update to 24.0.8 or better 25.0.2 and report back if it fixes the issue. Thank you!

  17. szaimen
    added
    needs info
    0. Needs triagePending check for reproducibility or if it fits our roadmap
    and removed
    1. to developAccepted and waiting to be taken care of
    on Jan 9, 2023
  18. szaimen
    closed this as completedon Mar 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    0. Needs triagePending check for reproducibility or if it fits our roadmapbugfeature: object storagefeature: previews and thumbnailsneeds info

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions