Skip to content

Remove binary blobs from history? #11

Closed
Closed
Remove binary blobs from history?#11
Assignees
aozarov
Labels
🚨 criticalP0 critical issue. Requires immediate fixtriage meI really want to be triaged.

Description

@mbrukman
mbrukman
opened on Feb 17, 2015

The repo is currently 274MB when cloned, and just the src tree (i.e., excluding past versions in .git) is 85MB — the actual code takes up negligible space, and the vast majority of that space is taken up by src/test/resources/gcd-head.zip.

This makes it take an unusually large amount of time to clone the repo, and it will likely grow every time this blob is updated to a new version.

Would it be possible to not version these large files (and ideally remove them from history while there are few forks), and get them from Maven or other sources, pinned to a specific version?

Activity

  1. aozarov commented on Feb 18, 2015

    @aozarov
    aozarov
    on Feb 18, 2015
    Contributor

    I am not sure how to exclude past versions of a file (and was unaware that they are actually pulled when repo is cloned). I don't think it is likely that we will have much changes in gcd-head but the plan is to be depended on gcloud SDK which in the future will include the gcd tool and hopefully will be available via maven.

  2. ludoch commented on May 19, 2015

    @ludoch
    ludoch
    on May 19, 2015
    Contributor

    I think it's about time to do this clean up before we are too successful:-)
    Never done that, but I guess http://git-scm.com/book/en/v2/Git-Internals-Maintenance-and-Data-Recovery#Removing-Objects will help?

  3. aozarov commented on May 19, 2015

    @aozarov
    aozarov
    on May 19, 2015
    Contributor

    This sounds dangerous (and I wonder if such a change, that is not a changelist could then be applied back on the github repository...)

    In any case, I followed the directions there on my local branch.

    $ git verify-pack -v .git/objects/pack/pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.
    pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.idx pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.pack

    $ git verify-pack -v .git/objects/pack/pack-0c7478248833c2e7e29bb2bb0de8670669d2482c.idx | sort -k 3 -n | tail -3
    997af117c72074594a71ed6b141f9675709ec948 blob 931373 52784 90364920
    484475ed6fc310dc3eaf115309719344fcda6791 blob 89098873 89111817 273395
    ac05237a254e2892d0eb3b497d12f54515df775c blob 99807981 99811510 127031733

    $ git rev-list --objects --all | grep ac05237a254e2892d0eb3b497d12f54515df775c
    ac05237a254e2892d0eb3b497d12f54515df775c src/test/resources/gcd-v1beta2-rev1-2.1.1.zip

    (gcd-v1beta2-rev1-2.1.1.zip is what we want to prune).

    $ git log --oneline --branches -- src/test/resources/gcd-v1beta2-rev1-2.1.1.zip
    e68feca replace local gcd hack with a fixed version of gcd
    c79887b Automatically invoke gcd for testing

    $ git filter-branch --index-filter 'git rm --ignore-unmatch --cached src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' -- c79887b^..
    $ rm -Rf .git/refs/original
    $ rm -Rf .git/logs
    $ git prune --expire now
    $ git count-objects -v

    But could not see any difference (I verified and repeated the steps but still no change).

    I then found this: https://help.github.com/articles/remove-sensitive-data/
    and use it to do the cleanup of the old /gcd-v1beta2-rev1-2.1.1.zip
    $ git filter-branch --force --index-filter \

    'git rm --cached --ignore-unmatch src/test/resources/gcd-v1beta2-rev1-2.1.1.zip'
    --prune-empty --tag-name-filter cat -- --all

    But this created a scary/nasty change-list per forked branch (I tried it in my fork) touching all files (this may be safe but was too scary for me, so I dropped it). Not to mention that for this to work all forks has to be re-based (rather than merge) to avoid re-introducing the object again...

  4. aozarov commented on May 19, 2015

    @aozarov
    aozarov
    on May 19, 2015
    Contributor

    Any more suggestions on how to remove src/test/resources/gcd-v1beta2-rev1-2.1.1.zip which was introduced by this changelist and later got replaced with src/test/resources/gcd-head.zip by this changelist?

  5. jgeewax commented on May 20, 2015

    @jgeewax
    jgeewax
    on May 20, 2015

    @ryanseys : Any input here? Is there a "best practice way" to remove large binaries from history? The link @ludoch provided seems pretty straightforward.

    We'll be totally re-writing history (which sucks, I know) and therefore fucking over the 11 forks that exist today, but I think making 11 people unhappy is better than having giant binaries hanging around in the git repository...

  6. jgeewax commented on May 20, 2015

    @jgeewax
    jgeewax
    on May 20, 2015

    @aozarov

    But this created a scary/nasty change-list per forked branch (I tried it in my fork) touching all files (this may be safe but was too scary for me, so I dropped it).

    I think thats supposed to happen. You're effectively re-writing history, no?

  7. ryanseys commented on May 20, 2015

    @ryanseys
    ryanseys
    on May 20, 2015

    Yep that github link to remove sensitive data looks on point. You are rewriting all history when deleting the references to the files so the SHAs will change and screw over any forks or clones so potentially way more than 11 people. But this is the only way to permanently remove large blobs from your repo.

  8. jgeewax commented on May 20, 2015

    @jgeewax
    jgeewax
    on May 20, 2015

    Yep, sounds like a necessary evil :(

  9. aozarov commented on May 20, 2015

    @aozarov
    aozarov
    on May 20, 2015
    Contributor

    Is it really a necessary evil? When I clone my fork where I applied all this git scary magic I get:

    remote: Counting objects: 26043, done.
remote: Compressing objects: 100% (204/204), done.
remote: Total 26043 (delta 17), reused 0 (delta 0), pack-reused 25829
Receiving objects: 100% (26043/26043), 195.57 MiB | 42.28 MiB/s, done.
Resolving deltas: 100% (22834/22834), done.

    When I clone the non-fork (GoogleCloudPlatform/gcloud-java) I get:

    cloning into 'gcloud-java'...
remote: Counting objects: 70531, done.
remote: Compressing objects: 100% (1818/1818), done.
remote: Total 70531 (delta 1666), reused 0 (delta 0), pack-reused 68705
Receiving objects: 100% (70531/70531), 224.72 MiB | 42.19 MiB/s, done.
Resolving deltas: 100% (65692/65692), done.

    So the % difference is not that big and honestly I am not comfortable to do it blindly (as
    my knowledge in git is very basic, I just follow the steps without really understanding them
    or their implication).

    I still didn't manage to recover my fork which keep saying that it is "This branch is 171 commits ahead of GoogleCloudPlatform:master" (without any file change). Yes, I understand that it is re-writing history (though I would expect less
    history to be behind not ahead).

    Any advice on how to reset my fork to a fresh start would be much appreciated. After that (and after I know how to recover from it) I don't mind to give it another try (and let someone that understand review that push request or better yet do it himself).

  10. jgeewax commented on May 21, 2015

    @jgeewax
    jgeewax
    on May 21, 2015

    OK I think the bigger problem is that this library is 200 MB...

    That's an order of magnitude larger than gcloud-python, which has been around for quite a bit longer:

    Cloning into 'gcloud-python'...
remote: Counting objects: 16892, done.
remote: Compressing objects: 100% (139/139), done.
remote: Total 16892 (delta 91), reused 0 (delta 0), pack-reused 16753
Receiving objects: 100% (16892/16892), 29.53 MiB | 5.69 MiB/s, done.
Resolving deltas: 100% (12957/12957), done.

    It also seems weird that the number of objects is so large. From the logs above, it looks like your fork had 26,043, whereas GCP/gcloud-java had 70,531? How did we get a 3x multiplier there? Are we sure there aren't more blobs floating around?

    The following stats just don't look quite right to me...

    Library Services Commits Start date Size
    gcloud-java 2 259 Nov 2014 224.72 MiB
    gcloud-python 3 1408 Jan 2014 29.53 MiB

  11. ryanseys commented on May 21, 2015

    @ryanseys
    ryanseys
    on May 21, 2015

    Yeah there's a definite difference.

    Before:

    image

    After:

    $ git clone https://github.com/ryanseys/gcloud-java.git ryanseys-gcoud-java

    Results:

    Cloning into 'ryanseys-gcoud-java'...
remote: Counting objects: 4017, done.
remote: Compressing objects: 100% (795/795), done.
remote: Total 4017 (delta 1994), reused 4017 (delta 1994), pack-reused 0
Receiving objects: 100% (4017/4017), 788.63 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1994/1994), done.
Checking connectivity... done.

  12. jgeewax commented on May 21, 2015

    @jgeewax
    jgeewax
    on May 21, 2015

    @ryanseys : Which commits / blobs did you chop out ?

  13. ryanseys commented on May 21, 2015

    @ryanseys
    ryanseys
    on May 21, 2015
    src/test/resources/gcd-v1beta2-rev1-2.1.1.zip
src/test/resources/gcd-head.zip

  14. ryanseys commented on May 21, 2015

    @ryanseys
    ryanseys
    on May 21, 2015

    You don't really chop out commits, you chop out files (listed above) from the index and as a result, you affect every commit that follows the earliest commit that added those files to the index.

    How to reproduce:

    Step 0. Fork and clone repo.

    git clone git@github.com:{{username}}/gcloud-java.git

    Step 1. Remove src/test/resources/gcd-v1beta2-rev1-2.1.1.zip from index.

    git filter-branch --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' --tag-name-filter cat -- --all

    Step 2. Remove src/test/resources/gcd-head.zip from index.

    git filter-branch --force --prune-empty --index-filter 'git rm -rf --cached --ignore-unmatch src/test/resources/gcd-head.zip' --tag-name-filter cat -- --all

    Step 3. Ignore those files from now on, make a new commit.

    echo 'src/test/resources/gcd-head.zip' >> .gitignore
echo 'src/test/resources/gcd-v1beta2-rev1-2.1.1.zip' >> .gitignore
git commit -am 'Remove files'

    Step 4. Garbage collect outdated blobs and force push back to your fork.

    (WARNING THIS WILL REWRITE HISTORY IN YOUR FORK)

    git for-each-ref --format='delete %(refname)' refs/original | git update-ref --stdin
git reflog expire --expire=now --all
git gc --prune=now
git push origin --force --all

    Step 5. Clone it under a new name, enjoy the speed up!

    git clone git@github.com:{{username}}/gcloud-java.git gcloud-java-new

    Let me know if this doesn't work for you!

  15. jgeewax commented on May 21, 2015

    @jgeewax
    jgeewax
    on May 21, 2015

    Rather than Step 4 (delete fork, new repo) could you just do git push origin -f ? Or no?

  16. 60 remaining items

  17. meltsufin
    added 14 commits that reference this issue on Apr 29, 2026

    fix: update ignored diffs (#206)

    240e141

    fix: adding deprecated tags to deprecated methods (#472)

    91b5152

    feat: expose default client configs (#541)

    01e6daf

    chore: generate topic retention changes (#798)

    a1c8aa6

    fix: do not block forever if message size > flow control limits (#847)

    f4ca4b2

    Modifying Publish example in README to match other examples given, and (

    22735ed

    fix: shutdown grpc stubs properly when a subscriber is stopped (#74)

    81ea2e1

    fix: ensure all publishes complete before shutting down publisher (#76)

    f18bc3a

    fix: changing integration test push endpoint to be valid https (#170)

    faeb580

    fix: update ignored diffs (#206)

    eee310a

    fix: adding deprecated tags to deprecated methods (#472)

    9d2fefb

    feat: expose default client configs (#541)

    c42b32f

    chore: generate topic retention changes (#798)

    2edb58e

    fix: do not block forever if message size > flow control limits (#847)

    e1c6cbc
  18. benjaminp
    added a commit that references this issue on Jul 13, 2026

    chore: add and run synthtool (googleapis#11)

    b128bfe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

🚨 criticalP0 critical issue. Requires immediate fixtriage meI really want to be triaged.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions