Page MenuHomePhabricator

mirror-repos.sh snapshots created at different times will have different train release tags
Open, MediumPublic

Description

Since old train release tags eventually get deleted from mediawiki/core, someone running mirror-repos.sh on their own (instead of relying on what's in https://files.liw.fi/train-dev/repos.tar.xz) will end up with a snapshot that does not contain the branch mentioned in vdc/README.md (1.35.0-wmf.34) . Generally speaking, this makes it difficult to have a static walkthrough document.

Event Timeline

dancy created this task.Aug 17 2020, 6:45 PM

I'd really like to get to the point where you simply select the train version from a list of available versions, then that gets set as an environment variable so that you don't have to type the version again during a deployment session.

LarsWirzenius removed LarsWirzenius as the assignee of this task.Aug 18 2020, 2:00 PM
brennen moved this task from Backlog to Watching on the User-brennen board.Aug 19 2020, 5:03 PM
LarsWirzenius triaged this task as Medium priority.Aug 24 2020, 11:56 AM

We should refresh the train-dev git repositories from Gerrit, or the Gerrit replica. If it's fast enough, we can do it every time we build the train-dev VM, but if it takes more then, say, 15 seconds, there should be a way to make it optional. Further, there should be a way to refresh the repositories without rebuilding the train-dev VM.

A couple thoughts:


speed up git fetches with protocol v2

When refreshing repositories, the script would definitely benefit from using git protocol v2. The enhanced protocol dramatically reduces the network traffic originating from Gerrit when doing a fetch, Google has a 30'000 feet overview at https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html

For Stretch, that requires git to be installed from stretch-wikimedia component/git which bring in version 2.20.1 instead of the old 2.11. Can be done by adding the following apt config:

/etc/apt/sources.list.d/wikimedia-git.list
deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/git

And then enable the new protocol globally:

/etc/gitconfig
[protocol]
    version = 2

get list of repos and latest wmf branch from Gerrit

I had a quick look at mirror-repos.sh, it lists all repositories explicitly which would require it to be updated anytime the source of trust is adjusted (that is in mediawiki/tools/release : https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/release/+/refs/heads/master/make-release/settings.yaml ). Ultimately that ends up in a wmf/* branch inside mediawiki/core, so maybe the script could instead check that out and we get the exact same list of repositories?

The latest wmf branch in mediawiki/core can be retrieved by listing branches in Gerrit and sort them by version:

$ git ls-remote --heads --sort="version:refname" https://gerrit.wikimedia.org/r/mediawiki/core 'refs/heads/wmf/*'
329fdc3c3ea8704960d8f6c64585b87636f31fcf	refs/heads/wmf/1.35.0-wmf.40
21c7485231ad45e0b0b7994918e8c25de702ee7f	refs/heads/wmf/1.35.0-wmf.41
2d57f715b91fdcc12a5391f3db7f15eca34f91bb	refs/heads/wmf/1.36.0-wmf.1
a1d9eb98403ac45979a92a945bec29407c009cdd	refs/heads/wmf/1.36.0-wmf.2
418d130932458026a7660af706fad9f512687227	refs/heads/wmf/1.36.0-wmf.3
7c80b68e0e17f87e411eeb4d9e1df4ad413a0c5d	refs/heads/wmf/1.36.0-wmf.4
ddfbb3cc6a23117bf33a588e7f5babdd69a6c386	refs/heads/wmf/1.36.0-wmf.5

With tail -n1 | awk '{ print $2 } | tr -d '\n' that gives: wmf/1.36.0-wmf.5.

From there update the repo, reset to that branch and git submodule update --jobs=8 --init would fetch the updates in parallel.