Page MenuHomePhabricator

Migrate wikimedia-portals-build to Docker container
Closed, ResolvedPublic

Description

The CI job wikimedia-portals-build should be migrated to use Docker container. As of July 2019 it runs on Jenkins instances based on Debian Jessie which are provisioned with puppet. That infrastructure is deprecated and has to be dismissed in favor of using Docker containers.

The job wikimedia-portals-build runs every Monday and roughly:

  • git clone wikimedia/portals.git
  • run npm install && npm test && npm run-script build-all-portals, that results in assets intended to power https://www.wikipedia.org/
  • Craft a commit adding the content of ./prod and send that to Gerrit for review against wikimedia/portals/deploy.git
  • A human +2 the change, once merged operations/mediawiki-config.git has the submodule bumped and the portal assets are deployed using a wrapper around scap, similar to how mediawiki-config changes are deployed.

We should be able to use the Release Pipeline to orchestrate the CI build and ultimately switch https://www.wikipedia.org/ to be served by kubernetes.

See also T210286: Create docker based CI job to build the wikidata-query-gui, the job is similar but the solution will certainly be different.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedBawolff
ResolvedAnomie
ResolvedBawolff
ResolvedBawolff
ResolvedLegoktm
ResolvedLucas_Werkmeister_WMDE
ResolvedBawolff
Resolvedsbassett
Resolvedsbassett
ResolvedJdforrester-WMF
Resolvedsbassett
Resolvedsbassett
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedReedy
ResolvedReedy
ResolvedReedy
ResolvedJdforrester-WMF
Resolvedhashar
Resolvedhashar
ResolvedJdforrester-WMF
Resolvedhashar
DeclinedMoritzMuehlenhoff
Invalidthcipriani
Resolvedmmodell
Resolvedhashar
ResolvedJoe
ResolvedJMeybohm
ResolvedJMeybohm
DuplicateDzahn
DeclinedDzahn
StalledJdforrester-WMF
ResolvedMoritzMuehlenhoff
ResolvedJdforrester-WMF
Resolvedthcipriani
OpenNone
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
OpenNone
Resolvedjeena
ResolvedJdforrester-WMF
ResolvedJdrewniak

Event Timeline

hashar created this task.Jan 15 2019, 10:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2019, 10:59 AM

I might look at migrating it to a CI Docker container. Longterm it would probably migrate to Blubber / Pipeline / Kubernetes which is T198901

greg triaged this task as Medium priority.Jan 16 2019, 4:38 PM
hashar updated the task description. (Show Details)
hashar moved this task from Backlog to Migration on the Release Pipeline board.
hashar added a subscriber: Jdrewniak.

To be clear: the current process should still work until the new process is in place.

debt added a comment.Aug 5 2019, 6:06 PM

To be clear: the current process should still work until the new process is in place.

it does seem to be working (wikipedia.org portal automatic updates) - or - at least, I'm not getting anymore failure emails. :)

Change 552144 had a related patch set uploaded (by Hashar; owner: Thcipriani):
[wikimedia/portals@master] pipeline: add a .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/552144

Change 552144 merged by jenkins-bot:
[wikimedia/portals@master] pipeline: add a .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/552144

We're now reaching crunch time for removing Jessie instances, so in a week's time I'm going to disable this job from jenkins (and remove the executors it runs on). You'll still be able to build in manually, until a replacement is built.

Mentioned in SAL (#wikimedia-releng) [2020-03-20T18:07:15Z] <James_F> jjb: Manually disabled wikimedia-portals-build cronjob. T213806

The jessie boxes this tries to run on have now been disabled. You will need to port this job for it to work again, sorry.

I believe Jeena has been working on this.

OK, let's provisionally say this is Resolved, and re-open or follow-up if it doesn't work.

Jdforrester-WMF closed this task as Resolved.Apr 27 2020, 11:46 PM