On the topic of tracking the size of assets: last week WMDE added some historical tracking of two of their components. The result can be seen at:

https://doc.wikimedia.org/Wikibase/master/js/data-bridge-dist-size/

https://doc.wikimedia.org/Wikibase/master/js/tainted-ref-dist-size/

That uses Github API to retrieve the data and looks very neat. Their task was https://phabricator.wikimedia.org/T253204 and the change https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/594122/

But that might be an entirely different thing ;)

From a quick read at the task, using PHP / MediaWiki one should be able to extract each of the registered RL modules and then run bundlesize against the flat files.

As for which command to run, Quibble has the ability setup the whole environment and runs arbitrary commands. A minimal use case:

quibble --command /bin/bash

Would clone repos, install dependencies, boot mysql/php -s/chromedriver and drop you in a shell. We use that in specific Jenkins jobs for use case we don't have much interest in implementing inside Quibble itself. An example is https://integration.wikimedia.org/ci/job/mediawiki-fresnel-patch-docker which runs some performance tests. The job invokes:

quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip-deps --commands "mediawiki-fresnel-patch"

The mediawiki-fresnel-patch script being provided by the CI team and is maintained along the Docker image.

So plenty of possibilities, we would need to investigate a bit more to have a better understanding of what has to run. The implementation itself should be rather easy :)

Summary from a discussion with Timo. WMDE writes node module and track it after merge for long time perspective. Webteam writes a MediaWiki extension and wants to enforce it on patchsets.

The easiest thing would be to write this as a PHP structure test, but if that can't work for some reason, we'd need to build you a custom job that runs npm test inside quibble (which we don't normally run).

The npm-test job is expected to be standalone, both to maitain consistency for CI management (building and maintaining a custom Jenkins CI image for every product does not scale), as well as for developer ergonomics in the form of reproducibility for local development. E.g. that when you run npm test in an isolated local environment, it Just Works. It does not depend on something else being pre-installed elsewhere locally and connected.

This task is essentially asking for a Quibble job that run shell command to execute a repo-local script (Quibble being our abstraction for setting up a database and web server with MediaWiki). We already have two jobs that do this:

• A Node.js one, which runs selenium-test. This seems to be working today. Is there a problem with this currently?
• A PHP one, which runs PHPUnit.

The way I would recommend implementing this, if asked today without prior art, is as a simple PHPUnit structure test in Vector, which uses ResourceLoader->getModule() to build and measure build sizes. It can then compare them to constant or JSON entry, and fail accordingly as needed. I could work on this with @vas, for example.

As part of T244276 they have added npm -s run test:size to selenium-test in both MinervaNeue and Vector. The selenium-test is meant to run integration tests across multiple repositories and as a result the test:size is run for any change made to a repository that depends on one of those two skins (ie pretty much everything). A change to mediawiki/core thus end up running npm install and npm run test:size for Vector. It is a bit of a waste.

I thought I had vetoed the whole idea of asserting the size of resources by having to spawn a whole MediaWiki setup, and instead suggested to use PHPUnit to generate the assets from MediaWiki ResourceLoader and then asserts from there. Anyway we need:

• a dedicated job for both those repositories
• remove test:size from the selenium-test entry point

Change 639166 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Job to run npm run test:size inside Quibble env

https://gerrit.wikimedia.org/r/639166

Since all repositories depend on Vector, the test:size script is run for any change beside the ones targeting Vector.

Note changes in core can impact the bundle sizes in Vector and it's good if those increase bundle sizes in Vector unexpectedly and the job fails. I agree it's probably useless if run in another repo such as Wikibase though and I guess if run in a dedicated script that could be configured to also run for core changes?

That is a good point, I will look into at adding jobs for mediawiki/core.