Page MenuHomePhabricator
Create Task
Maniphest T213806

Migrate wikimedia-portals-build to Docker container
Closed, ResolvedPublic
Actions

Description

The CI job wikimedia-portals-build should be migrated to use Docker container. As of July 2019 it runs on Jenkins instances based on Debian Jessie which are provisioned with puppet. That infrastructure is deprecated and has to be dismissed in favor of using Docker containers.

The job wikimedia-portals-build runs every Monday and roughly:

  • git clone wikimedia/portals.git
  • run npm install && npm test && npm run-script build-all-portals, that results in assets intended to power https://www.wikipedia.org/
  • Craft a commit adding the content of ./prod and send that to Gerrit for review against wikimedia/portals/deploy.git
  • A human +2 the change, once merged operations/mediawiki-config.git has the submodule bumped and the portal assets are deployed using a wrapper around scap, similar to how mediawiki-config changes are deployed.

We should be able to use the Release Pipeline to orchestrate the CI build and ultimately switch https://www.wikipedia.org/ to be served by kubernetes.

See also T210286: Create docker based CI job to build the wikidata-query-gui, the job is similar but the solution will certainly be different.

Details

SubjectRepoBranchLines +/-
pipeline: add a .pipeline/blubber.yamlwikimedia/portalsmaster+22 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedakosiarisT249724 Track and remove jessie based container images from production
 ResolvedJdforrester-WMFT224908 Drop jessie testing support
 ResolvedJdforrester-WMFT224907 Drop php55 testing support
 ResolvedReedyT205039 Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205041 Tracking bug for 1.27.6/1.30.2/1.31.2/1.32.2 security release
 ResolvedBawolffT197279 Direct POST to Special:ChangeEmail will bypass reauth check
 ResolvedAnomieT204729 Potential enwiki DOS due to slow WatchedItemStore::countVisitingWatchersMultiple
 ResolvedBawolffT207603 CVE-2019-12471: Loading JS from user space where the username is not a registered account is dangerous and should be banned
 ResolvedBawolffT208881 CSS using var() to create exponential sized calc() on wiki page will crash visitor's browser
 ResolvedLegoktmT199540 Forbid blocking IP ranges as big as /1 and /2, as done on ruwikiquote using the API
 ResolvedLucas_Werkmeister_WMDET212118 API responses for unpatrolled or (not) autopatrolled recent changes require privileges but may be cached publicly
 ResolvedBawolffT209794 Need to make a limit of count of attempts to change email address
 ResolvedsbassettT222036 Exposed suppressed username or log in Special:EditTags
 ResolvedsbassettT222038 Exposed suppressed log in RevisionDelete page
 ResolvedJdforrester-WMFT221739 Patch jQuery due to CVE-2019-11358
 ResolvedsbassettT25227 Use token when logging out
 ResolvedsbassettT221868 Send out wikitech-l post for T25227 ("Use token when logging out")
 ResolvedReedyT205042 Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedReedyT205043 Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT205044 Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205046 Update HISTORY in master after 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205047 Tag MW 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT224499 RELEASE-NOTES for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT224912 Update MediaWiki.org links and versions
 ResolvedJdforrester-WMFT224913 EOL REL1_27 and REL1_30 onwiki
 ResolvedReedyT225201 Formally announce EOL of MW 1.27 and 1.30
 ResolvedReedyT225149 Update CVEs and publish them
 ResolvedReedyT205048 Obtain CVEs for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedJdforrester-WMFT224906 Drop php56 testing support
 ResolvedhasharT223348 Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x
 ResolvedhasharT210287 Migrate wikimedia-fundraising-civicrm to a Docker container
 ResolvedJdforrester-WMFT224905 Move wikiba.se tests from php56 to php72
 ResolvedhasharT224591 Migrate contint* hosts to Buster
 DeclinedMoritzMuehlenhoffT226236 Upload docker-ce 18.06.3 upstream package for Stretch
 InvalidthciprianiT239880 Replacement hardware for buster/stretch upgrade of contint1001 and contint2001
 Resolved• mmodellT215458 Convert zuul to use scap
 ResolvedhasharT240551 Remove Zuul Debian package from WMCS instances
 ResolvedJoeT249110 Build and publish a python2 based container to build wheels
 ResolvedJMeybohmT249812 Rebuild helm/helm-diff for buster-wikimedia
 ResolvedJMeybohmT250479 Rebuild helmfile for buster-wikimedia
 DuplicateDzahnT210008 upgrade krypton (webserver_misc_apps) to stretch
 DeclinedDzahnT224194 switch webserver_misc_apps to PHP 7.2 (7.1)
 ResolvedJdforrester-WMFT211784 Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects
 ResolvedMoritzMuehlenhoffT241719 Migrate remaining self-hosted puppet masters to Puppet 5 / facter 3
 ResolvedJdforrester-WMFT236576 Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch
 ResolvedthciprianiT216039 jenkins / zuul backing up due to jenkins slaves down
 ResolvedhasharT216244 Don't hardcode castor url in castor docker container
 ResolvedJdforrester-WMFT225031 On CI Jenkins, audit worker labels and remove unused ones
 ResolvedJdforrester-WMFT239981 Delete Jenkins label DebianJessie
 OpenNoneT238747 Migrate www.wikipedia.org (and other www portals) to be its own service
 ResolvedjeenaT213806 Migrate wikimedia-portals-build to Docker container
 ResolvedJdforrester-WMFT237479 Update the wikimedia-portals repo's CI/linting code for various security issues
 ResolvedJdrewniakT247996 Fix issues with Gulp 4 migration

Event Timeline

hashar created this task.Jan 15 2019, 10:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2019, 10:59 AM
hashar mentioned this in T210286: Create docker based CI job to build the wikidata-query-gui.Jan 15 2019, 11:00 AM
hashar updated the task description. (Show Details)
hashar mentioned this in T198901: Migrate production services to kubernetes using the pipeline.Jan 15 2019, 5:43 PM

I might look at migrating it to a CI Docker container. Longterm it would probably migrate to Blubber / Pipeline / Kubernetes which is T198901

greg triaged this task as Medium priority.Jan 16 2019, 4:38 PM
hashar added a parent task: T216244: Don't hardcode castor url in castor docker container.Feb 20 2019, 2:18 PM
hashar mentioned this in T216244: Don't hardcode castor url in castor docker container.Feb 20 2019, 2:22 PM
hashar mentioned this in T227448: wikimedia-portals-build job failing to create castor workspace.Jul 8 2019, 9:57 AM
hashar merged a task: T228907: Migrate the wikimedia-portals-build timed CI job to node10.Jul 25 2019, 7:31 AM
hashar added a parent task: T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects.
hashar mentioned this in T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects.
hashar added subscribers: Jdforrester-WMF, debt.
hashar added projects: Release Pipeline, Release-Engineering-Team-TODO.Jul 25 2019, 7:44 AM
hashar updated the task description. (Show Details)
hashar moved this task from Backlog to Migration on the Release Pipeline board.
hashar added a subscriber: Jdrewniak.
greg moved this task from Should be empty (use Release-Engineering-Team) to Watching/External on the Release-Engineering-Team-TODO board.Jul 25 2019, 4:46 PM
thcipriani subscribed.Aug 5 2019, 4:23 PM

To be clear: the current process should still work until the new process is in place.

debt added a comment.Aug 5 2019, 6:06 PM
In T213806#5393427, @thcipriani wrote:

To be clear: the current process should still work until the new process is in place.

it does seem to be working (wikipedia.org portal automatic updates) - or - at least, I'm not getting anymore failure emails. :)

hashar added a project: Release-Engineering-Team (CI & Testing services).Aug 26 2019, 5:34 AM
hashar mentioned this in T236576: Move all Wikimedia CI (WMCS integration project) instances from jessie to stretch.Oct 28 2019, 10:52 AM
Jdforrester-WMF mentioned this in T238747: Migrate www.wikipedia.org (and other www portals) to be its own service.Nov 20 2019, 2:54 PM
hashar added a subtask: T238747: Migrate www.wikipedia.org (and other www portals) to be its own service.Nov 20 2019, 5:15 PM
gerritbot added a comment.Nov 27 2019, 7:37 PM

Change 552144 had a related patch set uploaded (by Hashar; owner: Thcipriani):
[wikimedia/portals@master] pipeline: add a .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/552144

gerritbot added a project: Patch-For-Review.Nov 27 2019, 7:37 PM
hashar added a parent task: T239981: Delete Jenkins label DebianJessie.Dec 20 2019, 8:55 AM
Jdforrester-WMF mentioned this in T243792: Upgrade dev dependencies of repositiory wikimedia/portals.Jan 28 2020, 4:00 PM
Jdforrester-WMF added a parent task: T237479: Update the wikimedia-portals repo's CI/linting code for various security issues.
Ladsgroup subscribed.Jan 30 2020, 12:41 PM
gerritbot added a comment.Feb 4 2020, 2:01 AM

Change 552144 merged by jenkins-bot:
[wikimedia/portals@master] pipeline: add a .pipeline/blubber.yaml

https://gerrit.wikimedia.org/r/552144

Maintenance_bot removed a project: Patch-For-Review.Feb 4 2020, 2:10 AM
Jdforrester-WMF removed a parent task: T237479: Update the wikimedia-portals repo's CI/linting code for various security issues.Feb 26 2020, 6:25 PM
Jdforrester-WMF added a subtask: T237479: Update the wikimedia-portals repo's CI/linting code for various security issues.
Jdforrester-WMF closed subtask T237479: Update the wikimedia-portals repo's CI/linting code for various security issues as Resolved.Mar 3 2020, 6:17 PM
Jdforrester-WMF added a comment.Mar 6 2020, 7:40 PM

We're now reaching crunch time for removing Jessie instances, so in a week's time I'm going to disable this job from jenkins (and remove the executors it runs on). You'll still be able to build in manually, until a replacement is built.

Stashbot added a comment.Mar 20 2020, 6:07 PM

Mentioned in SAL (#wikimedia-releng) [2020-03-20T18:07:15Z] <James_F> jjb: Manually disabled wikimedia-portals-build cronjob. T213806

Jdforrester-WMF removed a subtask: T238747: Migrate www.wikipedia.org (and other www portals) to be its own service.Mar 20 2020, 6:09 PM
Jdforrester-WMF added a parent task: T238747: Migrate www.wikipedia.org (and other www portals) to be its own service.

The jessie boxes this tries to run on have now been disabled. You will need to port this job for it to work again, sorry.

Jdforrester-WMF added a subtask: T247996: Fix issues with Gulp 4 migration.Apr 2 2020, 9:18 PM
Jdrewniak closed subtask T247996: Fix issues with Gulp 4 migration as Resolved.Apr 8 2020, 6:01 PM
Jdforrester-WMF assigned this task to jeena.Apr 8 2020, 7:13 PM

I believe Jeena has been working on this.

Jdforrester-WMF mentioned this in T247996: Fix issues with Gulp 4 migration.Apr 15 2020, 6:23 PM
Jdforrester-WMF added a comment.Apr 27 2020, 11:46 PM

OK, let's provisionally say this is Resolved, and re-open or follow-up if it doesn't work.

Jdforrester-WMF closed this task as Resolved.Apr 27 2020, 11:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL