Make git 2.2.0+ (preferably 2.8.x) available
Closed, ResolvedPublic

Description

Ideally we could use some newer features in git to speed up some operations on the deployment master. Specifically, I'm looking at the --jobs parameter to git fetch (added in 2.8.0) and the --dissociate parameter to git clone (added in 2.2.0). The former allows us to parallelize our git fetches in submodules, and the latter lets us re-use on-disk clones that still need an upstream (saves time fetching over the network).

Jessie (seems?) to have 2.1.x available, although tin is currently running 1.9.1. stretch/sid/experimental all have 2.8.1 which would be *fantastic*. Current latest upstream (just for comparison) is 2.9.2.

git submodule has been ported to C with 2.9.0 instead of the shell / sed spam. It can now downloads in parallel by passing --job X or setting submodule.fetchJobs = X.

demon created this task.Jul 20 2016, 6:42 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 20 2016, 6:42 PM

2.9.2 is backported to trusty at https://launchpad.net/~git-core/+archive/ubuntu/ppa

It seems fairly straightforward to backport git and git has good backward compatibility.

demon added a comment.Jul 28 2016, 4:36 PM

Would be nice for tool users too, who are routinely doing git operations and are stuck on an ancient 1.9.x as well.

I also want git fsck --name-objects

We could just build git our selfs by creating a git repo in gerrit and then adding the debian/ folder to build a deb for Jessie / trusty / other os too.

We could just build git our selfs by creating a git repo in gerrit and then adding the debian/ folder to build a deb for Jessie / trusty / other os too.

That's a huge waste of time when packages already exist that can be backported.

We could just build git our selfs by creating a git repo in gerrit and then adding the debian/ folder to build a deb for Jessie / trusty / other os too.

That's a huge waste of time when packages already exist that can be backported.

Oh yep, we could just use sid / stretch as it looks just like the name is different and it should work on Jessie.

hashar updated the task description. (Show Details)Dec 12 2016, 11:02 PM
hashar added a subscriber: hashar.

Mentions git-submodule got ported to C with 2.9.0. git submodule update also learns --jobs to fetch changes in parallel (default: 1). foreach does not support parallel operations though.

demon raised the priority of this task from Low to Normal.Jan 25 2017, 9:00 PM

Actually, stretch/testing has 2.11 available. That would solve all of the above and is easily backported to jessie. All dependencies (with pinned versions) seem to already be in jessie :)

Raising priority a bit because we keep hitting things we'd like to use in git.

@fgiunchedi Is this something you could possibly look at for us?

@demon I was indeed able to build stretch's git as-is on jessie, resulting in 2.11.0-2~bpo8+1. Uploading it internally to jessie-wikimedia is a possibility but it would also mean all jessie machines get the upgrade. ATM I don't think we have a good way of pinning a particular version only for e.g. tin/mira. We could upload git to experimental component though and enable said component on tin/mira though.

demon added a comment.Feb 1 2017, 3:50 PM

That seems like a reasonable course to go for now. Then after further testing, perhaps roll it out further to the rest of the servers?

Like I said in our meeting Monday I'm not particularly worried about this going out, but a small test (with the highest users of git) would help alleviate any concerns we may have.

hashar added a comment.Feb 1 2017, 9:33 PM

beta cluster and CI can definitely benefit from a newer git version. Can't it be pushed to jessie-wikimedia/backports and then we can apt:pin git?

jessie-wikimedia/backports has the same priority as main, if it gets added there, it also upgrades all labs instances with git installed (and would likely cause some disruptions). This should really be opt-in for some selected use cases like tin/mira only.

FWIW we're using the same method via jessie-wikimedia/experimental on cache boxes for e.g. nginx or the kernel, this is the relevant puppet config:

apt::repository { 'wikimedia-experimental':
    uri        => 'http://apt.wikimedia.org/wikimedia',
    dist       => "${::lsbdistcodename}-wikimedia",
    components => 'experimental',
}

We can do the same for tin/mira and upload git there.

hashar added a comment.Feb 2 2017, 9:36 AM

ARGHGHGHG. At the risk of derailing completely:

Debian.org has 500 by default and backports at 100

codename/componentPriorityOriginArchiveCodenameLabelComponent
jessie/main500o=Debiana=stablen=jessiel=Debianc=main
jessie-backports/main100o=Debian Backportsa=jessie-backportsn=jessie-backportsl=Debian Backportsc=main

When we have everything pinned at 1001:

codename/componentPriorityOriginArchiveCodenameLabelComponent
jessie-wikimedia/thirdparty1001o=Wikimediaa=jessie-wikimedian=jessie-wikimedial=Wikimediac=thirdparty
jessie-wikimedia/backports1001o=Wikimediaa=jessie-wikimedian=jessie-wikimedial=Wikimediac=backports
jessie-wikimedia/main1001o=Wikimediaa=jessie-wikimedian=jessie-wikimedial=Wikimediac=main

Which is because we pin 1001 whatever has Origin Wikimedia:

/etc/apt/preferences.d/wikimedia.pref
Package: *
Pin: release o=Wikimedia
Pin-Priority: 1001

From man 5 apt_preferences:

1001: causes a version to be installed even if this constitutes a downgrade of the package
500: causes a version to be installed unless there is a version available belonging to the target release or the installed version is more recent
100: causes a version to be installed unless there is a version available belonging to some other distribution or the installed version is more recent

thirdparty I guess it is to separate packages that are not in Debian.org so that one probably make sense. I am not sure though why jessie-wikimedia has backports and main components since they end up being at the same priority anyway.

I am not sure though why jessie-wikimedia has backports and main components since they end up being at the same priority anyway.

That distinction it meant to simplify the migration to a new Debian release, packages from backports are taken from the subsequent Debian release, so when we prepare stretch-wikimedia we probably can all ditch these. While "main" contains stuff that's not a direct backport, like custom packages or else. In reality that distinction isn't really enforced much :-)

mmodell edited projects, added Scap; removed scap2.Feb 10 2017, 6:22 PM

Change 337605 had a related patch set uploaded (by Filippo Giunchedi):
deployment::server: enable jessie-wikimedia/experimental

https://gerrit.wikimedia.org/r/337605

Change 337605 merged by Filippo Giunchedi:
deployment::server: enable jessie-wikimedia/experimental

https://gerrit.wikimedia.org/r/337605

Mentioned in SAL (#wikimedia-operations) [2017-02-15T11:20:27Z] <godog> upgrade git on tin/mira - T140927

fgiunchedi closed this task as Resolved.Feb 16 2017, 8:41 AM
fgiunchedi claimed this task.

This is completed, use_experimental can be removed once deployment servers are migrated to stretch

Change 353954 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add apt pinning for git on deployment servers

https://gerrit.wikimedia.org/r/353954

Change 353954 merged by Muehlenhoff:
[operations/puppet@production] Add apt pinning for git on deployment servers

https://gerrit.wikimedia.org/r/353954