Tagging in Test Platform folks, they may have suggestions on the best way to integrate this with our current system while keeping things snappy.

How long do the test take today to run if you run it on your machine?

What about start adding it to core so we can see how it works? The API test job is really fast today so adding an extra test and npm install should work fine (it's not the job that define the feedback time for devs):

https://integration.wikimedia.org/ci/job/quibble-apitests-only-vendor-php83/20688/console : SUCCESS in 1m 12s

We need to change quibble so it runs npm install then for that job (it do not do that today).

Then extensions can run it themselves when we upgrade to new Zuul (or do we want that to happen automatically?).

Additional discussion notes:

• We may want to make this task an Epic and split it into subtasks
• How the spec is passed to the linter in the context of CI is TBD
• Routes not in modules are available by default (hard-coded in ModuleManager), and might be a good proof-of-concept way to start
• This should initially be non-voting in CI
• Code coverage is a current example of a non-voting CI step

After we complete T425627: Publish the Wikimedia Spectral ruleset to NPM in the next sprint, we'll have an NPM package with our linter rules and should be able to add the linter to CI. I'd like to try and build a common understanding of what needs to happen for us to get there:

We will want to add a test to the API test job. This test will do the following:

1. Download and install the spectral-cli and wikimedia-spectral-ruleset packages from NPM.
2. Run Spectral:
   • Using our custom configuration as a parameter. This configuration must reference our ruleset.
   • On the MediaWiki REST API OAD, currently available in the Special:RestSandbox (e.g. on mediawiki.org) under "MediaWiki REST API (routes not in modules)". The exact URL on mediawiki.org is: https://www.mediawiki.org/w/rest.php/specs/v0/module/-. This OAD (ideally only with the /v1/ paths) should be made available on the wiki used for tests.
   • On every changeset in MediaWiki Core, with test output being extra information rather than a blocker for patch merges (non-voting CI step).

Running the Spectral test, time npx spectral lint mediawiki_restapi_oad.json, on my developer machine consistently returned test times of under 1.5 seconds, but I'd appreciate someone else confirming this as well. The web app version of the linter in dev mode generally takes around 21 seconds to lint the same OAD on the same machine, so 1.5s is surprising.

To add the test, we will use Quibble (https://doc.wikimedia.org/quibble/) and make changes to the https://gerrit.wikimedia.org/g/integration/config repository.

@BPirkle , @Mooeypoo , @HCoplin-WMF , can you please check if this is correct, and add any comments or questions of your own?

@Peter, can you please let us know if this makes sense, and if there are any other tools, repositories, or steps that we need to keep in mind when working on this?

Thank you!

I think it would be easier to just add it as voting from the beginning, that is less job. Is there something special reason to make it non-voting? If we feel we are unsure if it will break things, we could push and then revert if we feel something is broken? I'm also not 100% found of non breaking jobs, I'm thinking does it really adds any value if it is non breaking?

We have the quibble-apitests-only-vendor-php83 job that runs today that runs the npm run api-testing command from the package.json that today run mocha --timeout 0 --recursive --parallel tests/api-testing. It seems adding it to that seems to be the right place. Then you need to make sure the dependencies you need is added a dev dependency and then update the api-testing script in the package.json. Then the change only needs to happen in core and when it is merged, it's done.

The main reason we want to start with "non voting" is taht we're about to add a linter we want to start telling people to adhere to -- but they may not have yet, and may require some work from their side. I am nervous of suddenly adding a helpful tool that basically immediately blocks all merges in their repos 😅

This is especially true if we add this to an already-existing test that we might not know/control *where* it's running.

What we might be able to do is use its command to not output a "fail state" (0) for errors, which will make it non-voting by default until we get people to start adjusting and then get it voting.

What we might be able to do is use its command to not output a "fail state" (0) for errors, which will make it non-voting by default until we get people to start adjusting and then get it voting.

Sounds good. Then you can roll out things when you are ready, I like the idea if we are not blocking you.

+1 to what @Mooeypoo said. I would like to have flexibility around if and where we start throwing CI blocking errors, so that teams can selectively onboard when they're ready. We have plans to universally enforce these standards for all net new APIs too, but we don't want it to be annoying or disruptive for pre-existing APIs that are relatively stable.

To that effect, does this approach also allow us to enable the tests selectively per API? Or is this more of an all or nothing type thing?

I'm thinking either if we just swallows the error code when we run it for now (and we still get the output) or if you can lower the rules to only set warnings for now and then when you feel secure that it can be pushed, set the warnings to errors and it will automatically break builds.

Something like:

"api-testing": "spectral YOUR_SETUP || true && mocha --timeout 0 --recursive --parallel tests/api-testing"

would that work for you?