Page MenuHomePhabricator

Upload git 2.20 package from stretch-backports to component/git
Closed, ResolvedPublic

Description

Debian Stretch ships with git 2.11.0 which does not support the enhance git-protocol-v2 , we need it for CI.

stretch-backports has git 2.20.1 however backports will eventually be sunset before the eol of the distribution (T256877).

This task asks for git 2.20.1 from stretch-backports to be uploaded to stretch component component/git.

Note: in puppet modules/aptrepo/files/distributions-wikimedia , component/git is already mentioned but was for Jessie.

Event Timeline

Before touching this, I would like Moritz CC'd to ok this request.

What is meant to include this, only CI images or also anything in prod?

For all production machines as well, that will benefit scap deployed repositories and make it slightly faster.

Is there a measurable performance benefit of protocol-2 over the default git? We can do that, but it comes at a cost as we need to backport future git vulnerabilties to that version.

Is there a measurable performance benefit of protocol-2 over the default git? We can do that, but it comes at a cost as we need to backport future git vulnerabilties to that version.

Yeah, and I expect it to match up pretty well with our usages in large repos:

The main motivation for the new protocol was to enable server side filtering of references (branches and tags). Prior to protocol v2, servers responded to all fetch commands with an initial reference advertisement, listing all references in the repository. This complete listing is sent even when a client only cares about updating a single branch, e.g.: git fetch origin master. For repositories that contain 100s of thousands of references (the Chromium repository has over 500k branches and tags) the server could end up sending 10s of megabytes of data that get ignored. This typically dominates both time and bandwidth during a fetch, especially when you are updating a branch that's only a few commits behind the remote, or even when you are only checking if you are up-to-date, resulting in a no-op fetch.

We recently rolled out support for protocol version 2 at Google and have seen a performance improvement of 3x for no-op fetches of a single branch on repositories containing 500k references. Protocol v2 has also enabled a reduction of 8x of the overhead bytes (non-packfile) sent from googlesource.com servers. A majority of this improvement is due to filtering references advertised by the server to the refs the client has expressed interest in.

⁠— https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html

Ok, fair enough! Let's go ahead with component/git, then. Importing the last stretch-backports isn't good idea, though, it's quite outdated (Sat, 05 Jan 2019 14:12:21 -0800). I'll do a quick backport of the current version in buster-security.

I'll do a quick backport of the current version in buster-security.

I will then not touch this and leave it to your discretion.

I'll do a quick backport of the current version in buster-security.

I will then not touch this and leave it to your discretion.

Ack, I'll take this, it's part of the wider T256877 I'm working on.

Change 610289 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Add component/git for stretch-wikimedia

https://gerrit.wikimedia.org/r/610289

Is there a measurable performance benefit of protocol-2 over the default git? We can do that, but it comes at a cost as we need to backport future git vulnerabilties to that version.

Yes definitely and I fully understand cherry pick a package causes an overhead to keep track of security issues :-\ Beside the link @CDanis
posted above, I have a published a short blog post resuming the new protocol: Blog Post: Faster source code fetches thanks to git protocol version 2 :]

Change 610289 merged by Muehlenhoff:
[operations/puppet@production] Add component/git for stretch-wikimedia

https://gerrit.wikimedia.org/r/610289

Mentioned in SAL (#wikimedia-operations) [2020-07-08T13:31:26Z] <moritzm> imported git 2.20.1-2+deb10u3~wmf1 for stretch-wikimedia component/git T257308

MoritzMuehlenhoff claimed this task.

2.20.1-2+deb10u3~wmf1 built and imported to component/git

I gave it a try, it seems it is working as intended:

$ apt-cache policy git
git:
  Installed: 1:2.20.1-2+deb10u3~wmf1
  Candidate: 1:2.20.1-2+deb10u3~wmf1
  Version table:
 *** 1:2.20.1-2+deb10u3~wmf1 500
        500 http://apt.wikimedia.org/wikimedia stretch-wikimedia/component/git amd64 Packages
                                                                 ^^^^^^^^^^^^^
        100 /var/lib/dpkg/status
     1:2.20.1-1~bpo9+1 100
        100 http://mirrors.wikimedia.org/debian stretch-backports/main amd64 Packages
     1:2.11.0-3+deb9u7 500
        500 http://security.debian.org stretch/updates/main amd64 Packages
     1:2.11.0-3+deb9u5 500
        500 http://mirrors.wikimedia.org/debian stretch/main amd64 Packages

Thank you ;)