Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Krinkle | T267889 Update Fresh from Node.js 10 LTS to Node.js 14 LTS | |||
| Open | None | T267890 Upgrade all CI jobs for WMF-deployed projects from Node 12 to Node 14 LTS | |||
| Resolved | Jdforrester-WMF | T267888 Create WMF CI image for Node.js 14 | |||
| Declined | None | T267891 Create Debian packages for Node.js 14 upgrade |
Event Timeline
Not sure what this task rationale is.
Debian buster has node10,
https://packages.debian.org/buster/nodejs
and will provide security updates until at least 2022.
The next version of debian, bullseye, currently has node 12
https://packages.debian.org/testing/nodejs
and will eventually upgrade to node 14, probably before release.
So if the only reason for building node 14 packages, I consider the request declined. I'm happy to revisit if there are other reasons for which we need a production-ready node 14 installation.
I don't think Debian provides security support for the 1,446,739 packages on npmjs.org. It won't be long before our production services or CI tooling will no longer function on a supported version of its dependencies on Node 10.
The rationale is that developers are adopting newer versions on a different timeline than the Debian releases. Either we are ahead (the case for NodeJS) and/or we stick to an older version (the case for PHP: Stretch has 7.0, Buster has 7.3, we use 7.2).
For the recent NodeJS upgrades:
- The service team had a task filed to adopt NodeJS 6 even before Debian Stretch got released with NodeJS 4.8.2 and we switched our services to NodeJS 6 in January 2017 using Jessie as a baseline (T149331).
- The upgrade to NodeJS 10 was similar, we have migrated CI and services using Debian Stretch as a baseline (T210704) by having NodeJS 10 packaged for Stretch (T203239).
This request is similar: provide a NodeJS version which is ahead of Debian schedule since developers will surely adopt NodeJS 14 before it gets included in a Debian release.
Debian testing / unstable currently has NodeJS 12, experimental has NodeJS 14. The next Debian release will be frozen two months from now (January 2021). We thus have two different timelines:
A) NodeJS 14 get promoted to testing before the freeze in January 2020 and thus will be included in Debian Bullseye which should be released in July 2021. We would thus freeze this task and hold our adoption of NodeJS 14 for ~ 8 months.
B) They stick to NodeJS 12 and we would have to manage a backport ourselves. Either on:
B.1) Buster (which gives us NodeJS 14 "immediately")
B.2) Bullseye when it is released (and wait ~ 8 months).
I have asked the package uploader (Jérémy Lal).
Reply from the package uploader:
indeed, all other things being equal, nodejs 12.x will be in debian 11.
(unless a developer starts working full time on transitioning nodejs 14 to testing, which is unlikely to happen before release freeze).
Which implies Bullseye will ship NodeJS 12 and discards timeline A from above. Thus to support Node JS 14 either we backport to Buster or wait for Bullseye to be released and backport to it.
While nothing above seems to contradict that we don't have a compelling reason to install node 14 packages for production now, I want to make a point: developers don't just decide to use a newer version of some software in production in a void.
We want to stick to versions that are supported by our distribution of choice to reduce the already overwhelming burden of keeping up with security updates for our team. Now that we work in containerized environments it's faster to switch software distribution and node version, keeping what's currently supported by our distribution, which means we won't have to maintain the package and follow closely releases from upstream.
If we need to use node14 in CI for some reason, you can download whatever version of nodejs in your container in CI as you see fit, btw.
I didn't notice the task was opened again, I will decline again as I don't see a need for a node 14 package to use in production. CI can easily use the binaries from nodejs.org, as I stated above.
I disagree, strongly. The purpose of CI is to simulate the reality of production as closely as possible so that there is confidence ahead of deployments, rather than fire-fighting after the event. Using a jerry-rigged, unofficial package in CI diverges from this assurance, and undermines engineering productivity and confidence.
Staying on node 10 is obviously a no-go. Our support framework of all the tools we use will start dropping very quickly. It would not be reasonable to fork the dozens of upstream packages different teams use, trying to maintain node 10 compatibility whilst adding their security fixes/etc.. We're all part of a wider ecosystem. We can't just decide not to migrate to newer versions of node in a void.
CI needs to support node14 (or 12 or whatever) some time ahead of it being available in production (ideally a couple of months), so that developers can ensure that the re-platforming works and doesn't introduce novel issues. I think we all want to avoid the situation where production re-deployment from stretch onto buster ran at risk, ahead of any testing at all in CI due to lack of packaging until the last minute. I appreciate that this requirement brings forward some work for SREs, and we shouldn't rush this, but this problem is getting increasingly acute.
[Prompted by Node 10 going end-of-life today.]
Agreed.
Staying on node 10 is obviously a no-go. Our support framework of all the tools we use will start dropping very quickly. It would not be reasonable to fork the dozens of upstream packages different teams use, trying to maintain node 10 compatibility whilst adding their security fixes/etc.. We're all part of a wider ecosystem. We can't just decide not to migrate to newer versions of node in a void.
Right, there are two different needs here.
- Running services in production, in which we're pretty conservative about dependencies and maintain strict control over what runs. Relying on the version of nodejs in Debian Stable works well for this usecase.
- Running CI tests, which generally try and stay up to date with upstream. And it's also run by developers on their laptops (though ideally they'd use our containers/fresh/etc.!) so staying reasonably in sync with newer nodejs versions is good.
CI needs to support node14 (or 12 or whatever) some time ahead of it being available in production (ideally a couple of months), so that developers can ensure that the re-platforming works and doesn't introduce novel issues. I think we all want to avoid the situation where production re-deployment from stretch onto buster ran at risk, ahead of any testing at all in CI due to lack of packaging until the last minute. I appreciate that this requirement brings forward some work for SREs, and we shouldn't rush this, but this problem is getting increasingly acute.
So bullseye has nodejs 12, would that be acceptable? If so I can try to build a wikimedia-bullseye base image and then you could create ci-bullseye, nodejs12 images, etc. Should we create separate tasks for that?
I recommended Node 14 because I doubt we'll manage to get most Node services to even start using Node 12 in production through priorization/planning/CI/staging/verify before it goes EOL in 11 months time. (We also skipped an LTS when we moved from Node 6 to Node 10.)
... but, yes Node 12 would be a start I suppose. And with kubernetes nowdays (as you mentioned on IRC) it's not as tough of an upgrade as it used to be. Still though, we do need a strategy for how we'll support Node 14 next year in our pipeline, given that presumably there still won't be a stable debian.org package for Node 14 then just like there isn't today. I'm basically suggesting that whatever we do then, we should (consider) doing sometime this year, instead of (also) doing work this year for a Node 12 image.