Page MenuHomePhabricator

Investigate reminding people to update `npm audit`
Open, Stalled, MediumPublic

Description

We need a solution to keep us on top of required npm package updates.

Acceptance Criteria:

  • people on the Wikibase campsite should know about package that no longer pass npm audit in a reasonable timespan (e.g. a day)

Suggested solution:

  • run npm audit as a daily job on the Wikibase repository
    • If tainted-references is an npm dependency of Wikibase (specified in package.json) then npm audit on Wikibase will also warn about packages in tainted-references
  • run the daily job from jenkins
    • this means we don't fragment our ci code further
    • hopefully it follows a similar pattern to existing jobs
      • The daily selenium WikibaseLexeme job is run daily
      • there is an existing npm audit job
  • email wikidata-ci-status@wikimedia.de on the first failure and fixing of this job
    • This will stop us from spamming these emails and eventually becoming acclimatised to them
    • This email should be checked by campers since it is where our existing failing CI reports go
    • It follows a known pattern (rather than something potentially nicer like automatically making phabricator tickets that we don't know about yet)

Steps:

  • add tainted-references as a development dependency of Wikibase ()
  • create job on jenkins that runs npm audit daily on Wikibase and emails people at (wikidata-ci-status@wikimedia.de)

Event Timeline

Tarrow created this task.Jan 31 2020, 11:00 AM
Restricted Application added a subscriber: Aklapper. Β· View Herald TranscriptJan 31 2020, 11:00 AM

It might be npm would not be out the box able to audit all packages in "monorepo" Wikibase.git, i.e. running npm audit in wikibase root directory wouldn't make it "discover" package.json files in other subdirectory.
Looks like using something like lerna could be of help. That being said, lerna does not seem to have "audit" command (yet) and people have come up with different sorts of workarounds.

WMDE-leszek added a comment.EditedFeb 5 2020, 1:35 PM

or maybe lerna run audit would do just that? (https://github.com/lerna/lerna/tree/master/commands/run#readme)

Tarrow claimed this task.Feb 12 2020, 10:59 AM
Tarrow moved this task from To Do to Doing on the Wikidata-Tainted-References-Sprint10 board.

Lerna looks interesting but as far as I can see we'd need to commit to it for all packages in our "monorepo" (and it's really not clear to me how that would work with submodules like termbox).

Restricted Application added a project: Wikidata. Β· View Herald TranscriptFeb 17 2020, 10:43 AM

Change 572828 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[mediawiki/extensions/Wikibase@master] TR: Add as dev dependency of Wikibase

https://gerrit.wikimedia.org/r/572828

Tarrow updated the task description. (Show Details)Feb 19 2020, 9:52 AM

Change 573235 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[integration/config@master] Add daily job to run npm audit on Wikibase

https://gerrit.wikimedia.org/r/573235

Addshore triaged this task as Medium priority.Feb 19 2020, 10:21 AM
Restricted Application added a project: User-Addshore. Β· View Herald TranscriptMar 6 2020, 2:05 PM

Assigning to myself to make the CI change happen!

Change 573235 merged by jenkins-bot:
[integration/config@master] Add daily job to run npm audit on Wikibase

https://gerrit.wikimedia.org/r/573235

Addshore added a comment.EditedMar 23 2020, 1:02 PM

Job merged and running.

See https://integration.wikimedia.org/ci/job/wikibase-daily-npm-audit-daily-node10-npmaudit-docker/7/console

Currently email notifications are turned off pending @Tarrow review of this and getting it green :)

Additional merged gerrit config patches:

Addshore reassigned this task from Addshore to Tarrow.Mar 25 2020, 10:18 AM
Addshore added a subscriber: Addshore.

The most recent run says these two need updating:

# Run  npm install --save-dev stylelint-config-wikimedia@0.10.1  to resolve 1 vulnerability
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Low           β”‚ Prototype Pollution                                          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Package       β”‚ yargs-parser                                                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dependency of β”‚ stylelint-config-wikimedia [dev]                             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Path          β”‚ stylelint-config-wikimedia > stylelint > meow > yargs-parser β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ More info     β”‚ https://nodesecurity.io/advisories/1500                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜


# Run  npm update yargs --depth 2  to resolve 1 vulnerability
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Low           β”‚ Prototype Pollution                                          β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Package       β”‚ yargs-parser                                                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Dependency of β”‚ @wdio/cli [dev]                                              β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Path          β”‚ @wdio/cli > yargs > yargs-parser                             β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ More info     β”‚ https://nodesecurity.io/advisories/1500                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Change 602169 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] bridge: Update dependencies to pick up security fixes

https://gerrit.wikimedia.org/r/602169

Change 602175 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] TR: Update dependencies to pick up security fixes

https://gerrit.wikimedia.org/r/602175

@Addshore Maybe I'm missing something obvious but looking at https://gerrit.wikimedia.org/r/c/integration/config/+/573235/3/jjb/job-templates.yaml it does only checks the main wikibase package.json and not the important ones like bridge, TR, termbox (honestly, running npm audit on these submodules exploded majestically)

Is that intended?

Change 602177 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Update wikimedia-stylint-config

https://gerrit.wikimedia.org/r/602177

I see there’s already a home-brewed solution to this problem but nevertheless I'd like to leave something here:
https://dependabot.com/ - this is a bot that regularly checks for updates in a repo's dependencies. If any are found it updates them and creates a pull request. It works for various languages.
This is how it looks (on a private project of mine) - https://github.com/tzhelyazkova/schema-form-prototype/pull/3
It should be possible to be configured to work with gerrit, here's the core lib - https://github.com/dependabot/dependabot-script

I see there’s already a home-brewed solution to this problem but nevertheless I'd like to leave something here:

Maybe I'm missing something obvious, but that supposed to work in Github only and the ones here are in gerrit (can we run dependabot in gerrit? I'd be really surprised).

Change 602169 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] bridge: Update dependencies to pick up security fixes

https://gerrit.wikimedia.org/r/602169

Maybe I'm missing something obvious, but that supposed to work in Github only and the ones here are in gerrit (can we run dependabot in gerrit? I'd be really surprised).

It is now native to GitHub, but I cannot think of a reason this bot cannot work with gerrit.
The last link in my message describes how it works in more detail, and it mentions how to set the bot up with GitLab.

Again, I just wanted to let y'all know a solution already exists, and it does more than what our thing does. So we might want to weigh in the pros and cons of both.
I did not spend a lot of time researching dependabot, so I don't know how to integrate it with gerrit.

For all I care this could be a Beer and Cake project :)

So, there is a bot for gerrit that does various upgrades for extensions etc.
And this now works for Wikibase https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/605191
It also reports vulnerable packages to https://libraryupgrader2.wmflabs.org/vulns/npm

Having this work on multiple package.json files in a single repo is T228527: Support nested package.json files

WMDE-leszek changed the task status from Open to Stalled.Tue, Dec 1, 2:48 PM

Potentially will be solved magically by T228527