Page MenuHomePhabricator

apitests do not run across repositories
Open, Needs TriagePublic

Description

The REST: Emit swagger spec patch added some validation to the API tests. While doing so, it broke (T352742) CI for a couple of repositories, namely GrowthExperiments, Wikibase and Flow. As far as I understand it, the concept of gated extensions is used to avoid extension CI breaking because of changes in eg. Core.

Since both GrowthExperiments and Wikibase are gated extensions, I'd expect the failure introduced by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/860691 to cause a CI failure right on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/860691. However, that did not happen. The CI failure can be only triggered by uploading an empty DNM CI testing patch, such as https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/980472, and verify the CI passes that way.

Using DNM patches has a couple of downsides: it is expensive in terms of people time, and it also clutters the list of patches. More importantly, I think apitests for gated extensions should run as part of the gate-and-submit tests, so that failures similar to T352742 can be caught by CI (instead of developers trying to figure out why their change does not pass).

Filling this in ci-test-error following an advice given by @hashar.

Event Timeline

@daniel FYI, this is a potential follow-up for the swagger issues on the CI end of things.

Jdforrester-WMF subscribed.

The api-tests were very intentionally only applied to core and not extensions in 26fb84df00 due to the cost of running extra tests for all code and the MW team's plans for this code. Have the team changed their minds on how this is meant to work?

The api-tests were very intentionally only applied to core and not extensions in 26fb84df00 due to the cost of running extra tests for all code and the MW team's plans for this code. Have the team changed their minds on how this is meant to work?

Good to know this is not a bug, but an intention. I think it would be useful to run api tests everywhere at least on-demand, without that, I'm not sure how would we prevent bugs such as T352742: OpenAPI spec validation test fails for SecurePoll and causes CI failures for other repos.