Page MenuHomePhabricator

[T-REF] Wikibase CI broken due to Tainted Reference Node 18 indirect crypto dependency incompatibility
Open, Needs TriagePublic

Description

Currently, Wikibase CI build fails the new Node18 job with this error coming from the Tainted References app/widget:

16:45:03 node:internal/crypto/hash:69
16:45:03   this[kHandle] = new _Hash(algorithm, xofLen);
16:45:03                   ^
16:45:03 
16:45:03 Error: error:0308010C:digital envelope routines::unsupported
16:45:03     at new Hash (node:internal/crypto/hash:69:19)
16:45:03     at Object.createHash (node:crypto:133:10)
16:45:03     at module.exports (/src/view/lib/wikibase-tainted-ref/node_modules/webpack/lib/util/createHash.js:135:53)
16:45:03     at NormalModule._initBuildHash (/src/view/lib/wikibase-tainted-ref/node_modules/webpack/lib/NormalModule.js:417:16)
16:45:03     at handleParseError (/src/view/lib/wikibase-tainted-ref/node_modules/webpack/lib/NormalModule.js:471:10)
16:45:03     at /src/view/lib/wikibase-tainted-ref/node_modules/webpack/lib/NormalModule.js:503:5
16:45:03     at /src/view/lib/wikibase-tainted-ref/node_modules/webpack/lib/NormalModule.js:358:12
16:45:03     at /src/view/lib/wikibase-tainted-ref/node_modules/loader-runner/lib/LoaderRunner.js:373:3
16:45:03     at iterateNormalLoaders (/src/view/lib/wikibase-tainted-ref/node_modules/loader-runner/lib/LoaderRunner.js:214:10)
16:45:03     at Array.<anonymous> (/src/view/lib/wikibase-tainted-ref/node_modules/loader-runner/lib/LoaderRunner.js:205:4) {
16:45:03   opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
16:45:03   library: 'digital envelope routines',
16:45:03   reason: 'unsupported',
16:45:03   code: 'ERR_OSSL_EVP_UNSUPPORTED'
16:45:03 }
16:45:03 
16:45:03 Node.js v18.17.0
16:45:03 ERROR: "build:app -- --dest /tmp/tainted-refs-build" exited with 1.
16:45:03 ERROR: "test:distnodiff" exited with 1.

Event Timeline

Change 989897 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/Wikibase@master] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/989897

Change 989897 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/989897

From my side, feel free to do so. Though, I would hope that just updating all the dependencies would also update whatever outdated legacy open-ssl thing that is still referenced somehwere. So export NODE_OPTIONS=--openssl-legacy-provider maybe should just be a stop-gap.

From my side, feel free to do so. Though, I would hope that just updating all the dependencies would also update whatever outdated legacy open-ssl thing that is still referenced somehwere. So export NODE_OPTIONS=--openssl-legacy-provider maybe should just be a stop-gap.

Good point. Yeah, maybe it makes sense to start with the updates to the dependencies and then it also makes sense to have a separate ticket for Page-Previews. Thanks!

The mwgate-node18-docker job uses the container docker-registry.wikimedia.org/releng/node18-test:0.2.0-s2. It is based on Debian Bullseye. It has:

ii  libssl1.1:amd64 1.1.1w-0+deb11u1 amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl         1.1.1w-0+deb11u1 amd64        Secure Sockets Layer toolkit - cryptographic utility

And I guess Node JS 18 broke it?

The mwgate-node18-docker job uses the container docker-registry.wikimedia.org/releng/node18-test:0.2.0-s2. It is based on Debian Bullseye. It has:

ii  libssl1.1:amd64 1.1.1w-0+deb11u1 amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl         1.1.1w-0+deb11u1 amd64        Secure Sockets Layer toolkit - cryptographic utility

And I guess Node JS 18 broke it?

Could be. At least it must be a very recent change.

Mh, maybe? Though then I'm surprised that only so few apps are breaking. Could be that this is an issue where multiple things come together, some outdated npm dependencies + old openssl package in the base image? Updating that sounds also worthwhile if there is an update available.

The jobs got migrated from Node 16 to Node 18 by https://gerrit.wikimedia.org/r/c/integration/config/+/989842

Then checking docker-registry.wikimedia.org/releng/node16-test , it is based on Bullseye as well and thus comes with the same ssl libs:

ii  libssl1.1:amd64 1.1.1w-0+deb11u1 amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl         1.1.1w-0+deb11u1 amd64        Secure Sockets Layer toolkit - cryptographic utility

So my guess is Webpack needs to be updated to match whatever new expectations are imposed by NodeJS 18 crypto. That is potentially solveable by upgrading webpack to 5.x? There is eg https://github.com/webpack/webpack/issues/17805

Both Wikibase tainted refs and the Popups extensions have Webpack 4.x as a dependency (for Popups I think that seems to be a transient dependency of Storybook).

If the upgrade is non trivial, I can recreate the NodeJS 16 jobs and rollback the repos from Node 18 to 16.

And here is the webpack issue: nodejs 17: digital envelope routines::unsupported #14532.

Apparently fixed as part of add wasm md4 implementation #14584 which if I get it right was released with webpack v5.61.0.

At least for tainted references, the proper way forward is to update the dependencies, but I'm not sure how much work that would be. I dimly remember that there were some issues with that in the past.

I can't say much about the Popups extension. Though we found in other places that storybook is really only useful during active development and not so much during maintenance phases. So in some cases we have plainly deleted it, because it didn't seem time well spent to keep it updated and unused, and it can always be recreated from Git

The upgrade just for Webpack is pretty easy – https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CodeMirror/+/989887 is the upgrade for MediaWiki-extensions-CodeMirror, where I didn't need to make any config changes; for Page-Previews I've done https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Popups/+/989889 but unfortunately it's compounded by Storybook also being broken by the upgrade, and having seemingly dozens of breaking changes.

I looked into updating webpack (vue-cli) for tainted references, but I ran into a problem with it failing to find some files that we include in scss: Syntax Error: HookWebpackError: Cannot find module '../../../assets/close-icon.svg'. I tried a lot, but couldn't find a solution for this.

Probably, the sustainable way forward is to ditch vue-cli and switch to vite instead. But that will be for Wikidata Dev Team (Quality Tools "Sprint") to decide and to implement.

Change 991062 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Michael Große):

[mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.13] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/991062

Change 991329 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Michael Große):

[mediawiki/extensions/Wikibase@REL1_41] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/991329

Change 991329 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_41] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/991329

Change 991331 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Michael Große):

[mediawiki/extensions/Wikibase@REL1_40] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/991331

Change 991062 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.42.0-wmf.13] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/991062

Mentioned in SAL (#wikimedia-operations) [2024-01-17T14:43:11Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:991062|Skip tainted references test:distnodiff script to fix Wikibase CI (T354881)]], [[gerrit:991060|Only build result entries for used wbsearchentities results (T355053)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-17T14:44:39Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 lucaswerkmeister-wmde: Backport for [[gerrit:991062|Skip tainted references test:distnodiff script to fix Wikibase CI (T354881)]], [[gerrit:991060|Only build result entries for used wbsearchentities results (T355053)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-17T14:51:40Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:991062|Skip tainted references test:distnodiff script to fix Wikibase CI (T354881)]], [[gerrit:991060|Only build result entries for used wbsearchentities results (T355053)]] (duration: 08m 28s)

Change 991331 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_40] Skip tainted references test:distnodiff script to fix Wikibase CI

https://gerrit.wikimedia.org/r/991331

ArthurTaylor renamed this task from Wikibase CI broken due to Tainted Reference Node 18 indirect crypto dependency incompatibility to [T-REF] Wikibase CI broken due to Tainted Reference Node 18 indirect crypto dependency incompatibility.Feb 2 2024, 11:00 AM