Per our discussion, git-fat needs a new package and deployment to the cluster (exact list of hosts tbd, figuring out now)
Current version is 0.1.1-2
New version can be 0.1.2-1 (I think? There's not much upstream version numbering here, so mostly guessing)
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Ensuring explicit version of git-fat to upgrade to 0.1.2 | operations/puppet | production | +10 -2 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Ottomata | T147856 Scap deploy failed to sync git-fat artifacts | |||
| Resolved | Ottomata | T155856 Package + deploy new version of git-fat |
0.1.2-1 sounds reasonable to me. Changelog is reasonably thousandth-level-y
Is there anything on the releng side we need to do to push this forward? @Ottomata are you the right person to bother? :)
For context this will likely solve T147856: Scap deploy failed to sync git-fat artifacts and (probably) T98962: git fat/git deploy doesn't always unstub files [Trebuchet]
yay!
Thanks in advance. Feel free to poke me in IRC if you have questions/problems/need a post-deploy checker.
Done! https://apt.wikimedia.org/wikimedia/pool/main/g/git-fat/
Reopen if it doesn't work! :)
Ok, I'm just clicking buttons way too fast over here. It's been packaged. To deploy, we need to run a apt-get install git-fat everywhere. Can we check this somewhere before I do that? It'd be a big ol' salt command running everywhere, so let's test to make sure this works in one place first.
\o/
OK -- here's a good test and a good demonstration of what we're trying to correct with the new version:
Create and setup a new git fat directory in your home directory that uses /tmp as its remote and add some large file therein:
$ cd "$HOME" $ export GIT_FAT_VERBOSE=1 $ mkdir -p tmp/fattest $ cd tmp/fattest/ $ git init . Initialized empty Git repository in /home/thcipriani/tmp/fattest/.git/ $ git fat init Initialized git fat $ cat > .gitfat [rsync] remote = /tmp/thcipriani-fattest $ mkdir /tmp/thcipriani-fattest $ echo '*.bin filter=fat -crlf' > .gitattributes $ git add .gitfat .gitattributes $ git commit -m'Initial repository' [master (root-commit) 103e4dd] Initial repository 2 files changed, 3 insertions(+) create mode 100644 .gitattributes create mode 100644 .gitfat $ head -c 10m /dev/urandom > big1.bin $ git add big1.bin git-fat filter-clean: caching to .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297 $ git commit -m'add bin file' [master f2910a7] add bin file git-fat filter-clean: cache already exists .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297 1 file changed, 1 insertion(+) create mode 100644 big1.bin
Push the big file over to the directory under /tmp that is your remote
$ git fat push Pushing to /tmp/thcipriani-fattest Executing: rsync --progress --ignore-existing --from0 --files-from=- .git/fat/objects/ /tmp/thcipriani-fattest/ building file list ... 1 file to consider f5a7a13e60219f3a36954468f714198023479297 10,485,760 100% 216.71MB/s 0:00:00 (xfr#1, to-chk=0/1)
Now make a script that clones this git repo and tries to execute a git fat pull. The git fat pull will likely fail with the old version of git fat as it does in the example here (note 74 as the filesize of big1.bin in bytes initially before the 2nd, manual git fat pull)
$ cd .. $ cat > fattest.sh #!/bin/bash set -u NEWDIR="$HOME/tmp/fattest2" OLDDIR="$HOME/tmp/fattest" /usr/bin/git clone "$OLDDIR" "$NEWDIR" cd "$NEWDIR" /usr/bin/git fat init /usr/bin/git fat pull $ /bin/bash fattest.sh Cloning into '/home/thcipriani/tmp/fattest2'... done. Initialized git fat Pulling from /tmp/thcipriani-fattest Executing: rsync --progress --ignore-existing --from0 --files-from=- /tmp/thcipriani-fattest/ .git/fat/objects/ building file list ... 1 file to consider f5a7a13e60219f3a36954468f714198023479297 10,485,760 100% 199.38MB/s 0:00:00 (xfr#1, to-chk=0/1) Restoring f5a7a13e60219f3a36954468f714198023479297 -> big1.bin $ ls -lh fattest2/ total 4.0K -rw-rw-r-- 1 thcipriani wikidev 74 Feb 9 16:28 big1.bin $ cd fattest2/ $ git fat pull Pulling from /tmp/thcipriani-fattest Executing: rsync --progress --ignore-existing --from0 --files-from=- /tmp/thcipriani-fattest/ .git/fat/objects/ building file list ... 0 files to consider Restoring f5a7a13e60219f3a36954468f714198023479297 -> big1.bin git-fat filter-smudge: restoring from .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297 git-fat filter-clean: cache already exists .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297 $ ls -lh total 10M -rw-rw-r-- 1 thcipriani wikidev 10M Feb 9 16:28 big1.bin git-fat filter-clean: cache already exists .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297
So the big test is if the script manages to git fat pull without having to manually git fat pull!
I just tried the above ^ on a labs project machine with the new git fat version and it worked!
Here's the tl;dr:
$ /bin/bash fattest.sh Cloning into '/home/thcipriani/tmp/fattest2'... done. Initialized git fat Pulling from /tmp/thcipriani-fattest Executing: rsync --progress --ignore-existing --from0 --files-from=- /tmp/thcipriani-fattest/ .git/fat/objects/ building file list ... 1 file to consider 41bac1f5205725b0c81e13214877de7c96d54dd7 10,485,760 100% 136.56MB/s 0:00:00 (xfr#1, to-chk=0/1) Restoring 41bac1f5205725b0c81e13214877de7c96d54dd7 -> big1.bin git-fat filter-smudge: restoring from .git/fat/objects/41bac1f5205725b0c81e13214877de7c96d54dd7 $ ls -lh fattest2/ total 10M -rw-r--r-- 1 thcipriani wikidev 10M Feb 9 17:35 big1.bin thcipriani@ci-staging-gerrit:~/tmp$ ls -lh fattest2/ total 10M -rw-r--r-- 1 thcipriani wikidev 10M Feb 9 17:35 big1.bin $ apt-cache policy git-fat git-fat: Installed: 0.1.2-1~jessie1 Candidate: 0.1.2-1~jessie1 Version table: *** 0.1.2-1~jessie1 0 1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages 100 /var/lib/dpkg/status
hrm. I usually test in beta and don't have a production test repo for this kind of thing :(
There are a number of users of git fat (@Eevans runs into this problem all the time :(), but I don't know if any are planning on deploying anytime soon :\
/srv/deployment/analytics/refinery/.gitfat /srv/deployment/cassandra/logstash-logback-encoder/.gitfat /srv/deployment/cassandra/metrics-collector/.gitfat /srv/deployment/cassandra/twcs/.gitfat /srv/deployment/dropwizard/metrics/.gitfat /srv/deployment/elasticsearch/plugins/.gitfat /srv/deployment/prometheus/jmx_exporter/.gitfat /srv/deployment/wdqs/wdqs/.gitfat
Change 336887 had a related patch set uploaded (by Ottomata):
Ensuring explicit version of git-fat to upgrade to 0.1.2
Change 336887 merged by Ottomata:
Ensuring explicit version of git-fat to upgrade to 0.1.2