Page MenuHomePhabricator
Create Task
Maniphest T217114

Migrate Proton to k8s and nodejs 10
Closed, ResolvedPublic
Actions

Description

Per T210704: Migrate node-based services in production to node10.

  • Test proton source code against node 10
  • Implement pipeline
  • Profile Proton in deployment-charts
  • Deploy Proton in k8s
  • Switch traffic to the new deployment

Details

SubjectRepoBranchLines +/-
jjb: Drop chromium-render-deploy-npm-node-6-docker, unusedintegration/configmaster+0 -6
layout: [mediawiki/services/chromium-render/deploy] Archiveintegration/configmaster+4 -7
blubber: bump version to v4mediawiki/services/chromium-rendermaster+3 -2
layout: [mediawiki/services/chromium-render] Enable pipeline testing and publishingintegration/configmaster+6 -4
Pass use_nodejs10 to protonoperations/puppetproduction+2 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedakosiarisT249724 Track and remove jessie based container images from production
 ResolvedJdforrester-WMFT224908 Drop jessie testing support
 ResolvedJdforrester-WMFT224907 Drop php55 testing support
 ResolvedReedyT205039 Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205041 Tracking bug for 1.27.6/1.30.2/1.31.2/1.32.2 security release
 ResolvedBawolffT197279 Direct POST to Special:ChangeEmail will bypass reauth check
 ResolvedAnomieT204729 Potential enwiki DOS due to slow WatchedItemStore::countVisitingWatchersMultiple
 ResolvedBawolffT207603 CVE-2019-12471: Loading JS from user space where the username is not a registered account is dangerous and should be banned
 ResolvedBawolffT208881 CSS using var() to create exponential sized calc() on wiki page will crash visitor's browser
 ResolvedLegoktmT199540 Forbid blocking IP ranges as big as /1 and /2, as done on ruwikiquote using the API
 ResolvedLucas_Werkmeister_WMDET212118 API responses for unpatrolled or (not) autopatrolled recent changes require privileges but may be cached publicly
 ResolvedBawolffT209794 Need to make a limit of count of attempts to change email address
 ResolvedsbassettT222036 Exposed suppressed username or log in Special:EditTags
 ResolvedsbassettT222038 Exposed suppressed log in RevisionDelete page
 ResolvedJdforrester-WMFT221739 Patch jQuery due to CVE-2019-11358
 ResolvedsbassettT25227 Use token when logging out
 ResolvedsbassettT221868 Send out wikitech-l post for T25227 ("Use token when logging out")
 ResolvedReedyT205042 Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedReedyT205043 Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT205044 Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205046 Update HISTORY in master after 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205047 Tag MW 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT224499 RELEASE-NOTES for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT224912 Update MediaWiki.org links and versions
 ResolvedJdforrester-WMFT224913 EOL REL1_27 and REL1_30 onwiki
 ResolvedReedyT225201 Formally announce EOL of MW 1.27 and 1.30
 ResolvedReedyT225149 Update CVEs and publish them
 ResolvedReedyT205048 Obtain CVEs for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedJdforrester-WMFT224906 Drop php56 testing support
 ResolvedhasharT223348 Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x
 ResolvedhasharT210287 Migrate wikimedia-fundraising-civicrm to a Docker container
 ResolvedJdforrester-WMFT224905 Move wikiba.se tests from php56 to php72
 ResolvedhasharT224591 Migrate contint* hosts to Buster
 DeclinedMoritzMuehlenhoffT226236 Upload docker-ce 18.06.3 upstream package for Stretch
 InvalidthciprianiT239880 Replacement hardware for buster/stretch upgrade of contint1001 and contint2001
 Resolved• mmodellT215458 Convert zuul to use scap
 ResolvedhasharT240551 Remove Zuul Debian package from WMCS instances
 ResolvedJoeT249110 Build and publish a python2 based container to build wheels
 ResolvedJMeybohmT249812 Rebuild helm/helm-diff for buster-wikimedia
 ResolvedJMeybohmT250479 Rebuild helmfile for buster-wikimedia
 DuplicateDzahnT210008 upgrade krypton (webserver_misc_apps) to stretch
 DeclinedDzahnT224194 switch webserver_misc_apps to PHP 7.2 (7.1)
 ResolvedJdforrester-WMFT211784 Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects
 OpenNoneT210704 Migrate node-based services in production to node10
 ResolvedMSantosT217114 Migrate Proton to k8s and nodejs 10

Event Timeline

Tgr created this task.Feb 26 2019, 4:17 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 26 2019, 4:17 AM
Tgr added a parent task: T210704: Migrate node-based services in production to node10.Feb 26 2019, 4:17 AM
• Mholloway subscribed.Feb 26 2019, 4:21 AM
Tgr added a comment.Feb 26 2019, 4:22 AM

https://gerrit.wikimedia.org/r/c/mediawiki/services/chromium-render/+/489102 adds URL handling with the legacy URL API (the WHATWG one is not available on nodejs 6.11.0, although it is available on later versions of node 6). Would be nice to fix when the migration happens.

Tgr added a project: Product-Infrastructure-Team-Backlog-Deprecated.Feb 27 2019, 4:20 PM
• Jhernandez triaged this task as High priority.Feb 27 2019, 4:37 PM
• Jhernandez moved this task from Needs triage to Upcoming on the Product-Infrastructure-Team-Backlog-Deprecated board.
Tgr added a subscriber: MoritzMuehlenhoff.Mar 7 2019, 4:56 AM

@MoritzMuehlenhoff on what time scale do you expect this to happen?

MoritzMuehlenhoff added a comment.Mar 7 2019, 8:12 AM

See https://phabricator.wikimedia.org/T210704, ideally this would happen in the next quarter.

Jdforrester-WMF mentioned this in T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects.Jun 12 2019, 5:42 PM
• Mholloway edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.Jul 2 2019, 6:52 PM
• Mholloway mentioned this in T227124: [Bug] Proton fails in CI and locally for node 6.x.
• Mholloway added a comment.Jul 5 2019, 2:17 PM

@Tgr Would you mind memorializing the tricky bits about this in the description, for the benefit of whoever picks it up?

Jdforrester-WMF added a parent task: T211784: Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects.Jul 18 2019, 8:30 PM
gerritbot added a comment.Jul 18 2019, 9:42 PM

Change 524353 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/puppet@production] Pass use_nodejs10 to proton

https://gerrit.wikimedia.org/r/524353

gerritbot added a project: Patch-For-Review.Jul 18 2019, 9:42 PM
gerritbot added a comment.Jul 19 2019, 9:46 AM

Change 524353 merged by Alexandros Kosiaris:
[operations/puppet@production] Pass use_nodejs10 to proton

https://gerrit.wikimedia.org/r/524353

Maintenance_bot removed a project: Patch-For-Review.Jul 19 2019, 10:10 AM
MSantos claimed this task.Jul 31 2019, 8:35 PM
MSantos moved this task from To Do to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.EditedAug 1 2019, 7:14 PM
MSantos added a subscriber: Jdforrester-WMF.

So, I think our part is done:

  1. .pipeline is created
  2. Proton is tested against nodejs 10 and has a beta environment set for it. Proton is ready for a nodejs 10 environment.

@MoritzMuehlenhoff and @Jdforrester-WMF what are the next steps? Does that conclude this task?

Jdforrester-WMF added a comment.Aug 5 2019, 7:09 PM
In T217114#5385568, @MSantos wrote:

So, I think our part is done:

  1. .pipeline is created
  2. Proton is tested against nodejs 10 and has a beta environment set for it. Proton is ready for a nodejs 10 environment.

@MoritzMuehlenhoff and @Jdforrester-WMF what are the next steps? Does that conclude this task?

CI is running node 6; there's no testing for node 10 set up (and every time you V+2 you bypass all the testing anyway). Next step is to add the pipeline build step in CI, which I'll do now.

gerritbot added a comment.Aug 5 2019, 7:12 PM

Change 528226 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/services/chromium-render] Enable pipeline testing and publishing

https://gerrit.wikimedia.org/r/528226

gerritbot added a project: Patch-For-Review.Aug 5 2019, 7:12 PM
gerritbot added a comment.Aug 5 2019, 7:22 PM

Change 528226 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/services/chromium-render] Enable pipeline testing and publishing

https://gerrit.wikimedia.org/r/528226

Stashbot added a comment.Aug 5 2019, 7:22 PM

Mentioned in SAL (#wikimedia-releng) [2019-08-05T19:22:27Z] <James_F> Zuul: [mediawiki/services/chromium-render] Enable pipeline testing and publishing T217114

Jdforrester-WMF added a comment.Aug 5 2019, 7:25 PM

OK, it's now running the pipeline step for CI, but unfortunately that's failing as your pipeline file still tries to use v3 of blubber which is no longer supported; you need to bump to v4.

gerritbot added a comment.Aug 5 2019, 7:31 PM

Change 528232 had a related patch set uploaded (by MSantos; owner: MSantos):
[mediawiki/services/chromium-render@master] blubber: bump version to v4

https://gerrit.wikimedia.org/r/528232

gerritbot added a comment.Aug 5 2019, 7:44 PM

Change 528232 abandoned by MSantos:
blubber: bump version to v4

Reason:
duplicate of If0f9c4d7f2732358b8ee01ed0c6a7e55f5be15fb

https://gerrit.wikimedia.org/r/528232

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2019, 8:11 PM
• Mholloway closed this task as Resolved.Sep 30 2019, 3:55 PM
MoritzMuehlenhoff added a comment.Sep 30 2019, 4:01 PM

But nodejs on the proton* hosts is still on nodejs 6?

• Mholloway reopened this task as Open.Sep 30 2019, 4:02 PM
• Mholloway moved this task from Sign off to To Do on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.
MSantos lowered the priority of this task from High to Medium.Oct 7 2019, 2:50 PM
MSantos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated; removed Product-Infrastructure-Team-Backlog-Deprecated (Kanban).
MSantos updated the task description. (Show Details)
MSantos moved this task from Upcoming to Kanban on the Product-Infrastructure-Team-Backlog-Deprecated board.Nov 21 2019, 2:08 PM
MSantos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.
• Mholloway moved this task from To Do to Code Review on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Mar 5 2020, 9:03 PM
• Mholloway moved this task from Code Review to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Mar 10 2020, 1:53 PM
MSantos closed this task as Resolved.Jun 23 2020, 3:33 PM
MSantos updated the task description. (Show Details)
Jdforrester-WMF added a comment.Jun 25 2020, 10:01 AM

Does this mean we can remove the node 6 CI job and the entire mediawiki/services/chromium-render/deploy repo now?

gerritbot added a comment.Jun 25 2020, 10:02 AM

Change 607746 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/services/chromium-render/deploy] Archive

https://gerrit.wikimedia.org/r/607746

gerritbot added a project: Patch-For-Review.Jun 25 2020, 10:02 AM

Change 607747 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Drop chromium-render-deploy-npm-node-6-docker, unused

https://gerrit.wikimedia.org/r/607747

• Mholloway reopened this task as Open.Jun 29 2020, 1:40 PM

I don't think the k8s deployment of Proton is actually serving any traffic yet, so we should wait on those until it's switched over.

• Mholloway renamed this task from Migrate Proton to nodejs 10 to Migrate Proton to k8s and nodejs 10.Jun 29 2020, 1:40 PM
• Mholloway updated the task description. (Show Details)
• Mholloway added a project: serviceops.
Jdforrester-WMF closed this task as Resolved.Jul 10 2020, 10:12 AM

Now done, AIUI.

gerritbot added a comment.Jul 10 2020, 10:14 AM

Change 607746 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/services/chromium-render/deploy] Archive

https://gerrit.wikimedia.org/r/607746

Stashbot added a comment.Jul 10 2020, 10:15 AM

Mentioned in SAL (#wikimedia-releng) [2020-07-10T10:15:42Z] <James_F> Zuul: [mediawiki/services/chromium-render/deploy] Archive T217114

gerritbot added a comment.Jul 10 2020, 10:18 AM

Change 607747 merged by jenkins-bot:
[integration/config@master] jjb: Drop chromium-render-deploy-npm-node-6-docker, unused

https://gerrit.wikimedia.org/r/607747

MSantos awarded a token.Jul 10 2020, 12:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL