Page MenuHomePhabricator

Remove CI dependencies on git.wikimedia.org
Closed, ResolvedPublic

Description

At least bin/mw-core-get.sh and tools/fetch-mw-ext from integration/jenkins.git use git.wikimedia.org to get an archive of the source from a git repository. Sadly git.w.o regularly has downtime. So for our jenkins jobs not to fail in that case we should remove that dependency.

To make things difficult gerrit does not support downloading an archive either from the UI nor from the SSH endpoint:

$ git archive --remote="ssh://jzerebecki@gerrit.wikimedia.org:29418/mediawiki/core" "eab45e37b66582a94e5a7700021e41a59560e665" >test.tar
Gerrit Code Review: git-upload-archive: not found
fatal: The remote end hung up unexpectedly

That leaves us with the following possibilities:

  1. Add a gerrit mirror to every jenkins slave that uses those scripts.
  2. Add an on demand mirror (git clone/fetch) of the repos needed.
  3. Use a gerrit mirror that has git archive --remote support. (Does that work with the intelligent git http protocol? If not this is complicated by the fact that each jenkins slave then needs ssh access to that mirror.)

Any other ways? Any preferences for a solution?


Version: wmf-deployment
Severity: enhancement

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:50 AM
bzimport set Reference to bz72001.
bzimport added a subscriber: Unknown Object (MLST).

Why not just clone with --depth 1? Seems reasonable fast to me:

hoo@wikidata-hhvm:/tmp/git-test$ time git clone https://gerrit.wikimedia.org/r/mediawiki/core --depth 1
Cloning into 'core'...
remote: Counting objects: 4146, done
remote: Finding sources: 100% (4146/4146)
remote: Getting sizes: 100% (3955/3955)
remote: Compressing objects: 99% (54822/54823)
remote: Total 4146 (delta 533), reused 1846 (delta 189)
Receiving objects: 100% (4146/4146), 19.18 MiB | 5.05 MiB/s, done.
Resolving deltas: 100% (568/568), done.
Checking connectivity... done.

real 0m9.855s
user 0m2.675s
sys 0m1.122s

We have two slave scripts (integration/jenkins.git) which still depends on git.wikimedia.org

bin/mw-core-get.sh
tools/fetch-mw-ext

Most jobs have been migrated to Zuul cloner which does a full clone. The left over jobs are misty:

mwext-browertests-* jobs which were meant to trigger browser tests on patch proposal. We abandoned that for now

mwext-Wikibase* and mwext-Wikidata* jobs which I have yet to migrate to Zuul cloner.

There is a few other jobs, will need to investigate.

Krinkle renamed this task from ci/jenkins: remove dependency on git.wikimedia.org to Remove dependency on git.wikimedia.org.Jan 8 2015, 1:33 PM
Krinkle lowered the priority of this task from Medium to Low.
Krinkle added a project: Technical-Debt.
Krinkle set Security to None.
Krinkle removed a subscriber: Unknown Object (MLST).

This regularly ensures that a ton of jobs fail. Could this get higher priority?

Change 200122 had a related patch set uploaded (by Legoktm):
Use zuul-cloner for Wikibase/Wikidata jobs

https://gerrit.wikimedia.org/r/200122

Change 200122 merged by jenkins-bot:
Use zuul-cloner for Wikibase/Wikidata jobs

https://gerrit.wikimedia.org/r/200122

Wikibase/Wikidata jobs are now using zuul-cloner. Only usage left of the 'mw-setup-extension' macro is browser tests ('{name}-{ext-name}-{browser}')

The parsoidsvc-php-parsertests job is also using the mw-core-get.sh script.

Change 234519 had a related patch set uploaded (by Hashar):
Remove legacy definition

https://gerrit.wikimedia.org/r/234519

Change 234520 had a related patch set uploaded (by Hashar):
Remove obsolete tools/fetch-mw-ext

https://gerrit.wikimedia.org/r/234520

Patches above get rid of tools/fetch-mw-ext which used git.wikimedia.org to fetch extensions.

Next step will be to dish bin/mw-core-get.sh which is still used by the job parsoidsvc-php-parsertests.

Change 234519 merged by jenkins-bot:
Remove legacy definition

https://gerrit.wikimedia.org/r/234519

Change 234520 merged by jenkins-bot:
Remove obsolete tools/fetch-mw-ext

https://gerrit.wikimedia.org/r/234520

greg renamed this task from Remove dependency on git.wikimedia.org to Remove CI dependencies on git.wikimedia.org.Aug 28 2015, 3:26 PM

Change 236215 had a related patch set uploaded (by Legoktm):
Remove unused get-mw-core macro

https://gerrit.wikimedia.org/r/236215

Change 236216 had a related patch set uploaded (by Legoktm):
Remove unused mw-core-get.sh

https://gerrit.wikimedia.org/r/236216

Change 236215 merged by jenkins-bot:
Remove unused get-mw-core macro

https://gerrit.wikimedia.org/r/236215

Change 236216 merged by jenkins-bot:
Remove unused mw-core-get.sh

https://gerrit.wikimedia.org/r/236216

Legoktm claimed this task.

Nothing left AFAIS :D