Page MenuHomePhabricator

Package + deploy new version of git-fat
Closed, ResolvedPublic3 Estimate Story Points

Description

Per our discussion, git-fat needs a new package and deployment to the cluster (exact list of hosts tbd, figuring out now)

Current version is 0.1.1-2
New version can be 0.1.2-1 (I think? There's not much upstream version numbering here, so mostly guessing)

Details

Related Gerrit Patches:

Event Timeline

demon created this task.Jan 20 2017, 9:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 20 2017, 9:39 PM

0.1.2-1 sounds reasonable to me. Changelog is reasonably thousandth-level-y

Changelog

  • Add whitespace for readability
  • Some code pep8-ifying
  • Ensured that a file touch pre-filter actually triggers mtime update

Is there anything on the releng side we need to do to push this forward? @Ottomata are you the right person to bother? :)

For context this will likely solve T147856: Scap deploy failed to sync git-fat artifacts and (probably) T98962: git fat/git deploy doesn't always unstub files [Trebuchet]

I can help!

yay!

Thanks in advance. Feel free to poke me in IRC if you have questions/problems/need a post-deploy checker.

Ottomata reopened this task as Open.Feb 9 2017, 3:54 PM

Ok, I'm just clicking buttons way too fast over here. It's been packaged. To deploy, we need to run a apt-get install git-fat everywhere. Can we check this somewhere before I do that? It'd be a big ol' salt command running everywhere, so let's test to make sure this works in one place first.

Ottomata edited projects, added Analytics-Kanban; removed Analytics.Feb 9 2017, 3:54 PM
Ottomata set the point value for this task to 3.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
thcipriani added a comment.EditedFeb 9 2017, 4:40 PM

Ok, I'm just clicking buttons way too fast over here. It's been packaged. To deploy, we need to run a apt-get install git-fat everywhere. Can we check this somewhere before I do that? It'd be a big ol' salt command running everywhere, so let's test to make sure this works in one place first.

\o/

OK -- here's a good test and a good demonstration of what we're trying to correct with the new version:

Create and setup a new git fat directory in your home directory that uses /tmp as its remote and add some large file therein:

$ cd "$HOME"
$ export GIT_FAT_VERBOSE=1
$ mkdir -p tmp/fattest
$ cd tmp/fattest/
$ git init .
Initialized empty Git repository in /home/thcipriani/tmp/fattest/.git/
$ git fat init
Initialized git fat
$ cat > .gitfat
[rsync]
remote = /tmp/thcipriani-fattest
$ mkdir /tmp/thcipriani-fattest
$ echo '*.bin filter=fat -crlf' > .gitattributes
$ git add .gitfat .gitattributes
$ git commit -m'Initial repository'
[master (root-commit) 103e4dd] Initial repository
 2 files changed, 3 insertions(+)
 create mode 100644 .gitattributes
 create mode 100644 .gitfat
$ head -c 10m /dev/urandom > big1.bin
$ git add big1.bin
git-fat filter-clean: caching to .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297
$ git commit -m'add bin file'
[master f2910a7] add bin file
git-fat filter-clean: cache already exists .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297
 1 file changed, 1 insertion(+)
 create mode 100644 big1.bin

Push the big file over to the directory under /tmp that is your remote

$ git fat push
Pushing to /tmp/thcipriani-fattest
Executing: rsync --progress --ignore-existing --from0 --files-from=- .git/fat/objects/ /tmp/thcipriani-fattest/
building file list ...
1 file to consider
f5a7a13e60219f3a36954468f714198023479297
     10,485,760 100%  216.71MB/s    0:00:00 (xfr#1, to-chk=0/1)

Now make a script that clones this git repo and tries to execute a git fat pull. The git fat pull will likely fail with the old version of git fat as it does in the example here (note 74 as the filesize of big1.bin in bytes initially before the 2nd, manual git fat pull)

$ cd ..
$ cat > fattest.sh
#!/bin/bash
set -u
NEWDIR="$HOME/tmp/fattest2"
OLDDIR="$HOME/tmp/fattest"
/usr/bin/git clone "$OLDDIR" "$NEWDIR"
cd "$NEWDIR"
/usr/bin/git fat init
/usr/bin/git fat pull
$ /bin/bash fattest.sh
Cloning into '/home/thcipriani/tmp/fattest2'...
done.
Initialized git fat
Pulling from /tmp/thcipriani-fattest
Executing: rsync --progress --ignore-existing --from0 --files-from=- /tmp/thcipriani-fattest/ .git/fat/objects/
building file list ...
1 file to consider
f5a7a13e60219f3a36954468f714198023479297
     10,485,760 100%  199.38MB/s    0:00:00 (xfr#1, to-chk=0/1)
Restoring f5a7a13e60219f3a36954468f714198023479297 -> big1.bin
$ ls -lh fattest2/
total 4.0K
-rw-rw-r-- 1 thcipriani wikidev 74 Feb  9 16:28 big1.bin
$ cd fattest2/
$ git fat pull
Pulling from /tmp/thcipriani-fattest
Executing: rsync --progress --ignore-existing --from0 --files-from=- /tmp/thcipriani-fattest/ .git/fat/objects/
building file list ...
0 files to consider
Restoring f5a7a13e60219f3a36954468f714198023479297 -> big1.bin
git-fat filter-smudge: restoring from .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297
git-fat filter-clean: cache already exists .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297
$ ls -lh
total 10M
-rw-rw-r-- 1 thcipriani wikidev 10M Feb  9 16:28 big1.bin
git-fat filter-clean: cache already exists .git/fat/objects/f5a7a13e60219f3a36954468f714198023479297

So the big test is if the script manages to git fat pull without having to manually git fat pull!

I just tried the above ^ on a labs project machine with the new git fat version and it worked!

Here's the tl;dr:

$ /bin/bash fattest.sh
Cloning into '/home/thcipriani/tmp/fattest2'...
done.
Initialized git fat
Pulling from /tmp/thcipriani-fattest
Executing: rsync --progress --ignore-existing --from0 --files-from=- /tmp/thcipriani-fattest/ .git/fat/objects/
building file list ...
1 file to consider
41bac1f5205725b0c81e13214877de7c96d54dd7
     10,485,760 100%  136.56MB/s    0:00:00 (xfr#1, to-chk=0/1)
Restoring 41bac1f5205725b0c81e13214877de7c96d54dd7 -> big1.bin
git-fat filter-smudge: restoring from .git/fat/objects/41bac1f5205725b0c81e13214877de7c96d54dd7
$ ls -lh fattest2/
total 10M
-rw-r--r-- 1 thcipriani wikidev 10M Feb  9 17:35 big1.bin
thcipriani@ci-staging-gerrit:~/tmp$ ls -lh fattest2/
total 10M
-rw-r--r-- 1 thcipriani wikidev 10M Feb  9 17:35 big1.bin
$ apt-cache policy git-fat
git-fat:
  Installed: 0.1.2-1~jessie1
  Candidate: 0.1.2-1~jessie1
  Version table:
 *** 0.1.2-1~jessie1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

Ok, great. Can I install on a remote target and we try in prod for
something?

Ok, great. Can I install on a remote target and we try in prod for
something?

hrm. I usually test in beta and don't have a production test repo for this kind of thing :(

There are a number of users of git fat (@Eevans runs into this problem all the time :(), but I don't know if any are planning on deploying anytime soon :\

/srv/deployment/analytics/refinery/.gitfat
/srv/deployment/cassandra/logstash-logback-encoder/.gitfat
/srv/deployment/cassandra/metrics-collector/.gitfat
/srv/deployment/cassandra/twcs/.gitfat
/srv/deployment/dropwizard/metrics/.gitfat
/srv/deployment/elasticsearch/plugins/.gitfat
/srv/deployment/prometheus/jmx_exporter/.gitfat
/srv/deployment/wdqs/wdqs/.gitfat

I think we’ll have a refinery deploy to do soon. Will check…

Change 336887 had a related patch set uploaded (by Ottomata):
Ensuring explicit version of git-fat to upgrade to 0.1.2

https://gerrit.wikimedia.org/r/336887

Change 336887 merged by Ottomata:
Ensuring explicit version of git-fat to upgrade to 0.1.2

https://gerrit.wikimedia.org/r/336887

Updating here: git-fat 0.1.2 now installed everywhere by puppet.

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Feb 10 2017, 2:52 PM
thcipriani closed this task as Resolved.Feb 10 2017, 6:06 PM

Thanks @Ottomata!