@hashar gave me some info about this, recommending we avoid using -backports if possible. That's because backports will be phased out before buster hits end of life. So we need to get npm 6.14 onto the image another way, like installing the version needed using blubber, or making a new base image.

I don't know much about backports to understand how they will be phased out! The article on backports says:

When a release transitions to Long Term Support phase (LTS-phase), security is no longer handled by the main Debian security team.[96] Only a subset of Debian architectures are eligible for Long Term Support, and there is no support for packages in backports.[97]

Since Buster transitions to LTS sometime in 2022, wouldn't that that mean buster-backports would stop being supported at the same time?

TLDR do request SRE to import the npm 6.14.8 package from buster-backports to apt.wikimedia.org under a specific component.**

Since Buster transitions to LTS sometime in 2022, wouldn't that that mean buster-backports would stop being supported at the same time?

Yes that is correct, the transition means security support is dropped for most packages, and the associated backports repository is phased out. For Jessie that happened a few days after the security supported ended (July 2018 https://lists.debian.org/debian-backports-announce/2018/07/msg00000.html ). The reasoning is most people already upgraded to Stretch which had that time had been released a year before.

buster-backports will disappear one year and a half from now which might sounds far away in time but from experience is not that far. Eventually we will have to either:

• rebuild/import the package for Buster to apt.wikimedia.org

Or:

• force an upgrade to Debian Bullseye which in turns would force an an upgrade of NodeJS 10 to 12 and change various other packages which most probably would require code / dependencies to be updated etc.

Adding the npm package to apt.wikimedia.org under a specific component essentially freeze the state, drops the dependency upon buster-backports and lets ones track the version across all built containers (https://debmonitor.wikimedia.org/packages/npm). We can thus move at our own pace and can later on even upgrade to another npm version.

Another thing to be aware of is that NodeJS is released together with npm, the matches can be found at https://nodejs.org/en/download/releases/ . On Debian that is two separate packages and they are not kept in sync. It is probably not an issue anyway and if we have npm 6.14.8 on NodeJS 10.21 from Buster that sounds fine to me.

On CI images (not the one generated by the Pipeline) we are using a snapshot repository integration/npm.git (do NOT use that, it might just be phased out, is not monitored for security and is upgraded arbitrarily) which currently ships 6.14.5. The reasons for the repository are:

• we needed a npm version ahead of what Debian provided
• updating the Debian package is out of our reach / capabilities
• we need the same npm version across Debian version whenever we transition from a Debian release to another one (which is well, pretty much a constant migration)

On Debian that is two separate packages and they are not kept in sync. I was actually really curious about that, I was originally confused why npm and nodejs were installed separately, but didn't realize that it was a debian-specific thing. Also didn't know https://debmonitor.wikimedia.org/packages/npm existed, thats a cool tool for tracking!

Thanks a ton for all the background @hashar, your suggestion all makes sense. Will put in a request to add 6.14.8, but last question: what exactly do you mean when you say "under a specific component"....

Thanks a ton for all the background @hashar, your suggestion all makes sense.

Glad it helped :]

Will put in a request to add 6.14.8, but last question: what exactly do you mean when you say "under a specific component"....

Sorry that is the jargon from Debian. The packages are in suites (Stretch, Buster) and split by components (main, contrib, non-free).

On our apt repository (http://apt.wikimedia.org/wikimedia) we have a wikimedia specific suite for each of Debian suite (stretch-wikimedia, buster-wikimedia) which is configured on all hosts and Docker base images. That let us complement Debian by providing additional packages that are not provided by them. But since that is a global state (a new package would affect every machines and images), we actually place the packages in components which can then enabled solely on the hosts or image on which they are needed.

So for example, if one needs NodeJS 10 in a Stretch image, we have placed the package under component/node10 and it can be enabled for the image by using:

deb http://apt.wikimedia.org/wikimedia stretch-wikimedia component/node10

And that will take priority over the one provided by Debian Stretch.

Our repository structure is described at https://wikitech.wikimedia.org/wiki/APT_repository

So I guess one will provide component/npm that has all the required packages.

AHHHHH. Got it. Thanks!!

I came across this issues from T270321 and after speaking with @MoritzMuehlenhoff we think its better to use buster-backports. The reason for this is that the npm package has a bunch of dependencies (see below for a list of just the first level dependencies) which would also need to be managed and have fixes backported in any new component we create. Using backports means we dont have to worry about the dependencies and security fixes.

In relation to the concern regarding buster-backports vanishing I would first say that any productions services should have allready been migrated to bullseye by that point. however there will always be instances where that hasn't happened and its a problem we will need to solve across the fleet. As such we will do a one of import from buster-backports to buster-wikimedia of all packages which are currently in use. This should be fairly simple to script using the data in debmonitor (i may still have the script is used when jessie-backports went away)

first level dependencies

$ apt-cache depends npm                                                                                                                
npm
  Depends: nodejs
  Depends: ca-certificates
  Depends: node-abbrev
  Depends: node-ansi-regex
  Depends: node-ansistyles
  Depends: node-aproba
  Depends: node-archy
  Depends: node-cacache
  Depends: node-bluebird
  Depends: node-call-limit
  Depends: node-chownr
  Depends: node-config-chain
  Depends: node-detect-indent
  Depends: node-detect-newline
  Depends: node-editor
  Depends: node-fs-vacuum
  Depends: node-fs-write-stream-atomic
  Depends: node-glob
  Depends: node-graceful-fs
  Depends: node-has-unicode
  Depends: node-hosted-git-info
  Depends: node-iferr
  Depends: node-inflight
  Depends: node-inherits
  Depends: node-ini
  Depends: node-npm-package-arg
  Depends: node-promzard
  Depends: node-jsonstream
  Depends: node-json-parse-better-errors
  Depends: node-lazy-property
  Depends: node-libnpx
  Depends: node-lockfile
  Depends: node-lru-cache
  Depends: node-mississippi
  Depends: node-mkdirp
  Depends: node-move-concurrently
  Depends: node-nopt
  Depends: node-normalize-package-data
  Depends: node-gyp
  Depends: node-resolve-from
  Depends: node-encoding
  Depends: node-errno
  Depends: node-npmlog
  Depends: node-once
  Depends: node-opener
  Depends: node-osenv
  Depends: node-path-is-inside
  Depends: node-promise-inflight
  Depends: node-ansi
  Depends: node-qw
  Depends: node-read
  Depends: node-read-package-json
  Depends: node-request
  Depends: node-retry
  Depends: node-rimraf
  Depends: node-safe-buffer
  Depends: node-semver
  Depends: node-sha
  Depends: node-slide
  Depends: node-sorted-object
  Depends: node-from2
  Depends: node-stream-iterate
  Depends: node-ssri
  Depends: node-strip-ansi
  Depends: node-tar
  Depends: node-text-table
  Depends: node-uid-number
  Depends: node-unique-filename
  Depends: node-unpipe
  Depends: node-boxen
  Depends: node-import-lazy
  Depends: node-is-npm
  Depends: node-latest-version
  Depends: node-semver-diff
  Depends: node-xdg-basedir
  Depends: node-validate-npm-package-name
  Depends: node-which
  Depends: node-wrappy
  Depends: node-write-file-atomic

Thanks @jbond for the reasoning and explanation! Just because I'm not very familiar with package management, wouldn't this be an issue for anytime we wanted to upgrade versions of npm, or really any big package? And wouldn't this need to be upgraded eventually? Or do we not do usually major updates of big packages (like npm) unless absolutely necessary.

In T269957#6740577, @nnikkhoui wrote:

Thanks @jbond for the reasoning and explanation! Just because I'm not very familiar with package management, wouldn't this be an issue for anytime we wanted to upgrade versions of npm, or really any big package? And wouldn't this need to be upgraded eventually? Or do we not do usually major updates of big packages (like npm) unless absolutely necessary.

By using buster-backports we will be able to take advantage of the updates that automaticity make there way to that repo, so we don't have to manually manage and backport the security fixes our self. once the backports repo goes away then we will have to manage backports in the same way we would if the packages where in there own component. similar to what we currently do for jessies.

If one wanted to uses a version of software that was not in buster, or some other official Debian repo then we would also have to manage the package (and all dependencies) our self. We do this for some core/critical packages however it dose add significant overhead so its better to uses something that is allready in an official debian repo and benefit from the backporting work done by the general debian security team.

@jbond Gotcha! So just to make sure I understand, no matter the approach we'll have to manage the security of the package and its dependencies. With using backports, we get support from Debian until 2022. But by adding npm to our own debian package repo, we just have to manage dependencies ourselves starting now. Is that a correct understanding?

In T269957#6740990, @nnikkhoui wrote:

@jbond Gotcha! So just to make sure I understand, no matter the approach we'll have to manage the security of the package and its dependencies. With using backports, we get support from Debian until 2022. But by adding npm to our own debian package repo, we just have to manage dependencies ourselves starting now. Is that a correct understanding?

That is correct. Debian backports isn't security supported to the same extent as regular Debian (regular Debian gets backported fixes, while backports syncs to new backported versions from later Debian releases on a best effort basis). But for npm this is a perfectly fine tradeoff. It's always a case-by-case assessment, Antoine's earlier advice is completely correct for most packages, but in the case of npm is makes sense to simply follow backports.

Thanks @MoritzMuehlenhoff and @jbond for your explanations, will use buster backports on my own local Docker images. @jeena or @hashar would it be possible for releng is able install via backports on the blubber-generated images?

Change 661236 had a related patch set uploaded (by Nikki Nikkhoui; owner: Nikki Nikkhoui):
[wvui@master] Install NPM from buster backports

https://gerrit.wikimedia.org/r/661236

Change 661236 merged by jenkins-bot:
[wvui@master] Install NPM from buster backports

https://gerrit.wikimedia.org/r/661236

Change 682196 had a related patch set uploaded (by VolkerE; author: VolkerE):

[mediawiki/core@master] Update WVUI to v0.1.1

https://gerrit.wikimedia.org/r/682196

Change 682196 merged by jenkins-bot:

[mediawiki/core@master] Update WVUI to v0.1.1

https://gerrit.wikimedia.org/r/682196

@nnikkhoui What else is to do here?

@nnikkhoui You've filed it, and you've made it work. So I find both resolutions acceptable.