Page MenuHomePhabricator

[TECH] Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version - investigate and fix
Open, Needs TriagePublic

Description

Example build:

> test:mwlibs
> ZUUL_BRANCH=${ZUUL_BRANCH:-master} lib-version-check

Error while requiring library from https://raw.githubusercontent.com/wikimedia/mediawiki/master/resources/lib/vue/vue.common.prod.js: href=https://raw.githubusercontent.com/wikimedia/mediawiki/master/resources/lib/vue/vue.common.prod.js; statusCode=404
Build step 'Execute shell' marked build as failure

Affected several Wikibase repositories.

Status as of 2021-12-13: the test has been disabled in all affected repositories; a proper solution (how to make the test work again) needs to be figured out.

Status as of 2024-02-28: we disabled the broken test to unbreak CI, and determined that it would not be so easy to fix this (lib-version-check) check. It might be best to just drop it (see comment T297381#9370727)

Acceptance Criteria

  • Either
    • The check is fixed and renabled
    • The check is removed

Event Timeline

The 404 is because the file was renamed in the change to upgrade Vue to the migration build of Vue 3. However, fixing that isn’t enough:

diff --git a/package.json b/package.json
index ddb8e4ef..8d07b0be 100644
--- a/package.json
+++ b/package.json
@@ -2,7 +2,7 @@
 	"private": true,
 	"config": {
 		"remoteVersion": {
-			"vue": "https://raw.githubusercontent.com/wikimedia/mediawiki/#{ZUUL_BRANCH}/resources/lib/vue/vue.common.prod.js",
+			"vue": "https://raw.githubusercontent.com/wikimedia/mediawiki/#{ZUUL_BRANCH}/resources/lib/vue/vue.global.prod.js",
 			"vuex": "https://raw.githubusercontent.com/wikimedia/mediawiki/#{ZUUL_BRANCH}/resources/lib/vuex/vuex.js"
 		}
 	},
@@ -40,7 +40,7 @@
 		"unexpected": "^10.32.1",
 		"unexpected-dom": "^5.0.1",
 		"unexpected-sinon": "^11.0.1",
-		"vue": "^2.6.11",
+		"vue": "^2.6.11 || ^3.2.23",
 		"vuex": "^3.1.3",
 		"wdio-mediawiki": "^1.1.1",
 		"wdio-wikibase": "^5.2.0",
$ npm run test:mwlibs

> test:mwlibs
> ZUUL_BRANCH=${ZUUL_BRANCH:-master} lib-version-check

vue does not have a version key to check against.

When we require-from-url the fixed file path, it doesn’t actually return anything:

$ node
Welcome to Node.js v16.11.0.
Type ".help" for more information.
> const requireFromUrl = require( 'require-from-url/sync' );
undefined
> let remoteLib = requireFromUrl( 'https://raw.githubusercontent.com/wikimedia/mediawiki/master/resources/lib/vue/vue.global.prod.js' );
undefined
> remoteLib
{}

When we require-from-url the fixed file path, it doesn’t actually return anything:

MediaWiki core works around this in Resources.php:

// The file shipped by Vue does var Vue = ...;, but doesn't export it
// Add module.exports = Vue; programmatically
return file_get_contents( "$IP/$file" ) .
	';module.exports=Vue;';

But we can’t easily do this in test:mwlibs without writing our own require-from-url, I think.

@Catrope would you consider adding this module.exports=Vue; assignment directly to the vue.global(.prod).js files shipped by MediaWiki? That would fix our problem here and also slightly simplify the ResourceLoader setup, at the cost of having to repeat this step each time we want to update the bundled Vue.

Change 745537 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/WikibaseLexeme@master] Temporarily disable test:mwlibs

https://gerrit.wikimedia.org/r/745537

Change 745538 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Temporarily disable test:mwlibs

https://gerrit.wikimedia.org/r/745538

Change 745539 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[wikibase/termbox@master] Temporarily disable test:mwlibs

https://gerrit.wikimedia.org/r/745539

Change 745537 merged by jenkins-bot:

[mediawiki/extensions/WikibaseLexeme@master] Temporarily disable test:mwlibs

https://gerrit.wikimedia.org/r/745537

Change 745539 merged by jenkins-bot:

[wikibase/termbox@master] Temporarily disable test:mwlibs

https://gerrit.wikimedia.org/r/745539

Change 745954 had a related patch set uploaded (by Catrope; author: Catrope):

[mediawiki/core@master] vue: Add vue.cjs.js to make lib-version-check happy

https://gerrit.wikimedia.org/r/745954

@Catrope would you consider adding this module.exports=Vue; assignment directly to the vue.global(.prod).js files shipped by MediaWiki? That would fix our problem here and also slightly simplify the ResourceLoader setup, at the cost of having to repeat this step each time we want to update the bundled Vue.

It's a bit trickier than that, because foreign-resources.yaml currently has no mechanism to automate such a post-processing step, and if we did it manually, we'd 1) forget all the time and 2) break the integrity check, because the post-processed file would have a different sha-512 hash.

We could build post-processing steps into foreign-resources.yaml (ideally not by mangling the files themselves, but by adding new files that are derived from the original ones), and that is probably the best long-term solution, but I uploaded a quick and dirty solution that I hope will work for now. Vue ships a CommonJS version that we can't use in RL (because it tries to load other NPM packages) but that should hopefully work the purposes of lib-version-check. Let me know if that works.

Thanks, I didn’t realize these lib files were built automatically already (though that’s a good thing in general, of course). Unfortunately, it doesn’t look like the CJS build is working with require-from-url either:

$ node_modules/.bin/lib-version-check 
Error while requiring library from https://gerrit.wikimedia.org/r/changes/mediawiki%2Fcore~745954/revisions/1/files/resources%2Flib%2Fvue%2Fvue.cjs.js/download: Cannot find module 'estree-walker'
Require stack:
- /var/www/html/wiki1/extensions/Wikibase/node_modules/require-from-url/6d6608a59d11df077ef47cdbc74c62ea/r/changes/mediawiki%2Fcore~745954/revisions/1/files/resources%2Flib%2Fvue%2Fvue.cjs.js/download
- /var/www/html/wiki1/extensions/Wikibase/node_modules/require-from-url/sync.js
- /var/www/html/wiki1/extensions/Wikibase/node_modules/@wmde/lib-version-check/index.js

I think we (Wikibase devs) should spend a bit more time considering how we could make lib-version-check work with Vue 3 (spitballing: maybe it could be in the browser tests? there we get the file from ResourceLoader anyways…), and if it’s actually worth that investment to us; in the meantime, let’s finish disabling the version check for now (i.e. merge the remaining Wikibase change), and not consider this a blocker for the Vue 3 migration (including the migration away from the compat build).

Change 745538 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Temporarily disable test:mwlibs

https://gerrit.wikimedia.org/r/745538

Lucas_Werkmeister_WMDE renamed this task from WikibaseLexeme CI broken: test:mwlibs can’t find MediaWiki core Vue version to Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version.Dec 13 2021, 1:57 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)

Change 745954 abandoned by Catrope:

[mediawiki/core@master] vue: Add vue.cjs.js to make lib-version-check happy

Reason:

https://gerrit.wikimedia.org/r/745954

Another idea for making lib-version-check work: stop trying to eval anything, and directly get the version from foreign-resources.yaml (either as a new optional field for a dependency, or try to parse the version number out of the src URL). That makes the check more specific to MediaWiki, but I think we can live with that.

Task Review notes:

  • This affects Lexeme, Bridge and Tainted References

Prio Notes:

  • Might affect end users / production (prevents potential version related bugs)
  • Does not affect monitoring
  • Does not affect development efforts
  • Does not affect onboarding efforts
  • Affects additional stakeholders
ItamarWMDE renamed this task from Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version to [TECH] Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version.Sep 20 2023, 2:55 PM

Given how infrequently MediaWiki core updates the Vue version (we’ve just had the first upgrade in over a year – maybe it won’t be quite as long next time, but I think it’s fair to expect it won’t change, like, weekly), and how well such upgrades have been communicated so far (kudos to @Catrope and @egardner), I’m not sure we really need this version check at all: maybe it’s safe enough to assume that we’ll learn about a version upgrade, and file tasks like T352318, without a notification in CI.

Given how infrequently MediaWiki core updates the Vue version (we’ve just had the first upgrade in over a year – maybe it won’t be quite as long next time, but I think it’s fair to expect it won’t change, like, weekly), and how well such upgrades have been communicated so far (kudos to @Catrope and @egardner), I’m not sure we really need this version check at all: maybe it’s safe enough to assume that we’ll learn about a version upgrade, and file tasks like T352318, without a notification in CI.

Seems reasonable to me. I think that ideally we'd upgrade Vue in MW Core (and in Codex) every 6 months.

ArthurTaylor renamed this task from [TECH] Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version to [TECH] [SW] Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version.Feb 26 2024, 10:41 AM

This is an old task and I don't know much about the context here. Is this problem still happening?

The code that is producing this error is trying to request a file called resources/lib/vue/vue.common.prod.js, but this doesn't exist currently (it may have in an older version of Vue, pre v3).

The file you want is either resources/lib/vue/vue.global.js (for dev mode tooling) or resources/lib/vue/vue.global.prod.js (minified production build of the library).

See https://github.com/wikimedia/mediawiki/tree/master/resources/lib/vue for the set of the exact files that are being bundled in MW.

My guess is that somewhere an outdated version of some package is pointing to a Vue file that we're no longer shipping.

This is an old task and I don't know much about the context here. Is this problem still happening?

In a very broad sense, the problem is that we built a system to validate that we’re using the same Vue.js version as MediaWiki core, and that system broke with the move to Vue 3; we’re going to reevaluate (once this task makes it through our backlog) whether we still need that system at all or can just remove it.

The file you want is either resources/lib/vue/vue.global.js (for dev mode tooling) or resources/lib/vue/vue.global.prod.js (minified production build of the library).

Yes, but neither file returns an object with a .version member, so this isn’t as simple as updating the path. We’d have to make @wmde/lib-version-check check the version on the Vue global instead, and the question is whether this is worth the extra complexity or not.

ArthurTaylor renamed this task from [TECH] [SW] Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version to [TECH] Wikibase CI broken: test:mwlibs can’t find MediaWiki core Vue version - investigate and fix.Feb 28 2024, 10:27 AM
ArthurTaylor updated the task description. (Show Details)