Node 6 upgrade planning
Closed, ResolvedPublic0 Story Points

Description

Node 6 was recently transitioned to LTS status, and will be the active LTS release until April 2018. Node 4 will enter maintenance-only mode in April 2017. In the maintenance period until April 2018, Node 4 will continue to receive critical (security) fixes. It will however be fairly out of date in terms of functionality and performance.

Current status

Benefits of upgrading early

  • We are seeing very significant performance improvements (~50%) for services like RESTBase. Initial Parsoid parser test runs show no peformance changes.
  • Solid ES6 support.

Issues with upgrading early

  • Additional packaging / maintenance work, especially if we get significantly ahead of Debian.

Compatibility testing

  • RESTBase, hyperswitch & dependencies
  • service-runner
  • mathoid
  • ChangeProp
  • Analytics Query Service: merged, travis enabled, but no test run yet. Based on hyperswitch & doesn't use any binary dependencies, so very unlikely to have any issues. Calling this done -- @GWicke
  • graphoid -- Manually tested
  • kartotherian -- merged. Mapnik's "geojson" plugin is missing (T152131)
  • citoid -- Does not use travis, manually tested
  • cxserver -- Does not use travis, manually tested
  • MCS -- Manually tested
  • parsoid: pending
  • Electron render service -- Manually tested by @GWicke
  • Trending Edits -- Manually tested, OK
  • etherpad - Tested by @akosiaris, OK
There are a very large number of changes, so older changes are hidden. Show Older Changes

Should also add the new trending-edits service.

mobrovac edited the task description. (Show Details)Jan 9 2017, 6:50 PM

Scheduled to happen the week of 2017-01-16

@MoritzMuehlenhoff can you upload the new node version to our APT? We'd like to move on this issue this week.

@mobrovac nodejs 6.9.1 has been uploaded to carbon.

@Gehel Note that Karthoterian isn't ready for node 6 yet, so we need to be careful to not upgrade it until this is fixed. There's a few options, but it's probably best to set it on "hold" for now, we can discuss that tomorrow on IRC.

nodejs 6.9.1 has been uploaded to carbon.

I have learned about that upgrade literally a minute ago via T155443. That is a breaking change for CI!

On CI: NodeJS will be automatically updated tomorrow starting at 14:14 UTC when Nodepool refreshes the images . The Jenkins jobs assert the version of Node. Luckily @Paladox hadn't noticed that upgrade and even sent a config update https://gerrit.wikimedia.org/r/#/c/332382/

But I cant tell what are going to be the side effects. Seems the most important repositories have been tested already.

Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-operations) [2017-01-17T10:35:35Z] <hashar> CI switched NodeJS from v4 to v6 T155443 T149331

The CI Jessie instances are now using NodeJS version 6 as provided by apt.wikimedia.org. I kind of freaked out yesterday until I saw this task and all the compatibility testing that have been done over the last few months. Kudos on that!

Mentioned in SAL (#wikimedia-operations) [2017-01-17T17:48:15Z] <mobrovac> restbase installing node v6.9.1 on the cluster T149331

mobrovac claimed this task.Jan 17 2017, 5:48 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-17T18:03:27Z] <mobrovac> restbase deploying a0e542b, switching to Node v6 T149331

Change 332532 had a related patch set uploaded (by Mobrovac):
Switch to Node v6.9.1

https://gerrit.wikimedia.org/r/332532

Change 332532 merged by jenkins-bot:
Switch to Node v6.9.1

https://gerrit.wikimedia.org/r/332532

Over the last ~10 hours we have not seen any issues with node 6 & RESTBase. As expected, the most noticeable impact is significantly reduced memory usage:

Temporary heap limit warnings in logstash have dropped to almost zero, and there were no worker restarts at all so far.

That is great! I had T121850 about RESTBase emitting Heap memory limit exceed. The last one is at 2017-01-17T18:10:12 (logstash for 7 days https://logstash.wikimedia.org/goto/47e8919b60534319ba85768ba12db2e5 ) \O/

Change 332768 had a related patch set uploaded (by Muehlenhoff):
Stick with node 4.6 on maps due to karthotherian not being ready for node 6

https://gerrit.wikimedia.org/r/332768

Volans added a subscriber: Volans.Jan 18 2017, 7:02 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-18T19:12:29Z] <volans> Upgrading nodejs to v6 on ruthenium T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-18T19:25:11Z] <volans> Upgrading nodejs to v6 on wtp2001 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-18T20:47:56Z] <volans> Upgraded nodejs to v6 on wtp1001 T149331

Change 333028 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333028

Change 333030 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333030

Change 333031 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333031

Change 333032 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333032

Change 333034 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333034

Change 333036 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333036

Change 333038 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333038

Change 333039 had a related patch set uploaded (by Ppchelko):
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333039

Change 333036 merged by jenkins-bot:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333036

Mentioned in SAL (#wikimedia-operations) [2017-01-19T20:26:44Z] <volans> upgrading node to v6 on wtp2002 T149331

Change 333028 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333028

Change 333030 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333030

Change 333031 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333031

Change 333032 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333032

Mentioned in SAL (#wikimedia-operations) [2017-01-19T20:45:15Z] <volans> upgrading node to v6 on wtp2003 T149331

Change 333038 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333038

Change 333039 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333039

Mentioned in SAL (#wikimedia-operations) [2017-01-19T21:03:10Z] <volans> upgrading node to v6 on wtp1002 T149331

Change 333034 merged by Ppchelko:
Upgrade to Node v6.9.1

https://gerrit.wikimedia.org/r/333034

hashar removed a subscriber: hashar.Jan 19 2017, 9:11 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:43:59Z] <ppchelko@tin> Starting deploy [changeprop/deploy@ffd0b8b]: Canary deploy for switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:44:07Z] <mobrovac@tin> Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:45:18Z] <mobrovac@tin> Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 10s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:45:21Z] <mobrovac@tin> Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:47:03Z] <mobrovac@tin> Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 43s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:51:35Z] <ppchelko@tin> Finished deploy [changeprop/deploy@ffd0b8b]: Canary deploy for switching to node 6 T149331 (duration: 07m 36s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:51:46Z] <ppchelko@tin> Starting deploy [changeprop/deploy@ffd0b8b]: Deploy for switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:51:50Z] <mobrovac@tin> Starting deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:52:45Z] <ppchelko@tin> Finished deploy [changeprop/deploy@ffd0b8b]: Deploy for switching to node 6 T149331 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:53:14Z] <ppchelko@tin> Starting deploy [citoid/deploy@95df861]: Deploy for switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:53:26Z] <mobrovac@tin> Finished deploy [graphoid/deploy@f872f94]: Switching to node 6 T149331 (duration: 01m 39s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:53:40Z] <mobrovac@tin> Starting deploy [mobileapps/deploy@cacb3c9]: Switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:54:53Z] <mobrovac@tin> Starting deploy [electron-render/deploy@f1df2d3]: Switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:55:32Z] <ppchelko@tin> Finished deploy [citoid/deploy@95df861]: Deploy for switching to node 6 T149331 (duration: 02m 18s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:55:49Z] <ppchelko@tin> Starting deploy [cxserver/deploy@ff0225e]: Deploy for switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:56:22Z] <mobrovac@tin> Finished deploy [mobileapps/deploy@cacb3c9]: Switching to node 6 T149331 (duration: 02m 41s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:56:31Z] <mobrovac@tin> Starting deploy [trending-edits/deploy@0abcf25]: Switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:56:52Z] <mobrovac@tin> Finished deploy [electron-render/deploy@f1df2d3]: Switching to node 6 T149331 (duration: 01m 58s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:58:02Z] <ppchelko@tin> Finished deploy [cxserver/deploy@ff0225e]: Deploy for switching to node 6 T149331 (duration: 02m 12s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:58:21Z] <ppchelko@tin> Starting deploy [eventstreams/deploy@fe77f19]: Deploy for switching to node 6 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:58:31Z] <mobrovac@tin> Finished deploy [trending-edits/deploy@0abcf25]: Switching to node 6 T149331 (duration: 01m 59s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T22:59:52Z] <ppchelko@tin> Finished deploy [eventstreams/deploy@fe77f19]: Deploy for switching to node 6 T149331 (duration: 01m 30s)

Mentioned in SAL (#wikimedia-operations) [2017-01-19T23:32:18Z] <volans> upgrading node to v6 on wtp1003 T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-20T00:22:09Z] <volans> apt-upgrading nodejs to v6 on the rest of parsoid hosts (a deploy with restart will follow) T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-20T00:35:45Z] <mobrovac@tin> Starting deploy [parsoid/deploy@465f9c4]: Restarting Parsoid everywhere for Node v6 switch T149331

Mentioned in SAL (#wikimedia-operations) [2017-01-20T00:40:05Z] <mobrovac@tin> Finished deploy [parsoid/deploy@465f9c4]: Restarting Parsoid everywhere for Node v6 switch T149331 (duration: 04m 21s)

Change 332768 merged by Gehel:
Stick with node 4.6 on maps due to karthotherian not being ready for node 6

https://gerrit.wikimedia.org/r/332768

Mentioned in SAL (#wikimedia-operations) [2017-01-25T20:36:56Z] <volans> upgrading nodejs-legacy (it is just the symlink) to v6 on parsoid hosts T149331

mobrovac closed this task as "Resolved".

All of the services but Maps have been upgraded to Node 6, so I'm declaring victory here. Thanks to everyone that helped! The Maps status is being tracked in T150354: Implement Node6 support for Kartotherian/Tilerator.

akosiaris reopened this task as "Open".Jan 26 2017, 6:58 PM

Actually, there's etherpad left. I 'll do the upgrade tomorrow though :-). Reopening in the meantime

akosiaris closed this task as "Resolved".Jan 27 2017, 10:16 AM

And etherpad is now upgraded to 1.6.0-2 running on nodejs 6.9.1~dfsg-1+wmf1. Resolving the task once more

akosiaris edited the task description. (Show Details)Jan 27 2017, 10:16 AM
Jdforrester-WMF set the point value for this task to 0.Feb 2 2017, 7:01 PM
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptWed, Mar 22, 12:57 PM

Vagrant was updated to node 6 as well.