This is a general question that should be investigated.
We want to make sure the same processes in CI are setup for the Wikibase extension (& other wikibase related extensions) in terms of mediawiki-vendor checks that are already in place for mediawiki core itself.
The general case that needs to be covered.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| dockerfile for mediawiki-vendor version checker script | integration/config | master | +88 -0 | |
| DNM WIP crappy build checking stuff | mediawiki/vendor | master | +100 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Addshore | T173818 [Epic] Kill the Wikidata build step | |||
| Open | None | T179663 What happens if the Wikibase project specifies a version of a library outside of the range included in mediawiki-vendor? |
If Wikibase uses a feature of ~2.0 that is incompatible with 1.0.0, it'll blow up in production. The CI that currently runs on mediawiki/extensions/Wikidata pulls in only "require-dev" via composer, and uses mediawiki/vendor for the rest, so if there's a mismatch that is executed by tests, it'll get caught.
We probably want to apply the same CI to Wikibase (using mediawiki/vendor instead of composer). Then if you want to update a library, you'll submit a patch to mediawiki/vendor, then a patch to Wikibase with Depends-On so CI is testing against the new library, and then merge both at the same time whenever ready.
@Legoktm nailed it. If wikibase is going to use the same process as the rest of the Foundation production deployed extensions it needs to go all in on that including in the CI chain and the library update process. The current system can probably be improved, but that is a topic for another ticket (like T178137).
@Addshore, is this Resolved (given the title's question is answered) or do you see changes we need to do based on the answer which will now be done under this task?
I'm going to mark this as open as I just carried out a test and, well, I was expecting one of the CI tests to fail
https://gerrit.wikimedia.org/r/#/c/391632/
The patch specifies 7.2.0 of data model, and mediawiki-vendor contains 7.3.0
Just tested with the gate-submit jobs too and they also gave it the green light on https://gerrit.wikimedia.org/r/#/c/391632.
Looks like we have to add or alter a job.
I'm going to link to https://gerrit.wikimedia.org/r/#/c/424278/ here which adds a simple command that would identify differences.
This could be added to Wikibase too (and other extensions that use composer libs).
As part of the branch cut this command could be run for all extensions (where it exists) and package differences between mediawiki-vendor and the extensions could be identified.
This sort of happened a week or so when mediawiki-vendor was actually running versions of libraries that som extensions were not listed as being compatible with.
As a result https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/453581/ downgraded the libraries again, but no element of CI or huma checking caught the issue.
Change 460287 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/vendor@master] DNM WIP crappy build checking stuff
I wrote a crappy script that we can try to modify and work into a daily jenkins job.
The run today after I finished writing it identified 3 differences between the libs (all minor)
> composer-lock-diff --from ./../mw/composer.lock --to ./../current-composer.lock | grep -v "| REMOVED" | grep -v "| NEW" | sed 's/From/Master/g' | sed 's/To/Vendor/g' +---------------------------------------+---------+---------+---------------------------------------------------------------------+ | Production Changes | Master | Vendor | Compare | +---------------------------------------+---------+---------+---------------------------------------------------------------------+ | james-heinrich/getid3 | v1.9.15 | v1.9.14 | https://github.com/JamesHeinrich/getID3/compare/v1.9.15...v1.9.14 | | justinrainbow/json-schema | 5.2.7 | 5.2.6 | https://github.com/justinrainbow/json-schema/compare/5.2.7...5.2.6 | | pear/pear-core-minimal | v1.10.6 | v1.10.3 | https://github.com/pear/pear-core-minimal/compare/v1.10.6...v1.10.3 | +---------------------------------------+---------+---------+---------------------------------------------------------------------+
I made some patches:
remote: https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/460289 Updating james-heinrich/getid3 (v1.9.14 => v1.9.15)
remote: https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/460290 Updating justinrainbow/json-schema (5.2.6 => 5.2.7)
remote: https://gerrit.wikimedia.org/r/#/c/mediawiki/vendor/+/460291 Updating pear/pear-core-minimal (v1.10.3 => v1.10.6)
Change 560413 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] dockerfile for mediawiki-vendor version checker script
Change 560413 abandoned by Hashar:
dockerfile for mediawiki-vendor version checker script
Reason:
Seems that better suited inside mediawiki/vendor or in mediawiki/tools/release :)
I'm not sure if this should be on the external column of the Release-Engineering-Team board.
This is an issue with how mediawiki-vendor and development is tied together that is not owned by us (wmde / wikibase) at all.
Our life would be easier without this repository entirely.
The direct result of not tackling this issue is that production etc will continue to break in unexpected ways due to difference between mediawiki-vendor and dependencies defined in extensions (not just Wikibase).
I moved this to our Release-Engineering-Team (Radar) to indicate we don't have the ability to make forward progress on this so it's not actionable by Release-Engineering-Team (as evidenced by the lack of any movement on this task).
The architectural aspects of our deployments are separate from enabling our deployment. I think that what is needed to make forward progress on issues like this is for there to be an architectural decision made by finding consensus among the impacted technical community. Releng is a portion of the impacted technical community, but can't make the decision unilaterally (I don't know that we have the technical ability to make a good decision).
This work has in the past fallen to a group of folks who think really hard about really hard problems and push through solutions by force of will and dint of hard work. Release-Engineering-Team has been the home of a number of those individuals over time which is why I think we got tagged here. I don't think that's sustainable.