This seems to be happening again since a yesterday https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/1251102

I wonder if it only affects Math or also other extensions. The error I see

22:22:07   1) Wikibase GraphQL
22:22:07        retains boolean fields:
22:22:07 
22:22:07       AssertionError: expected { data: { searchItems: { …(1) } } } to deeply equal { data: { searchItems: { …(1) } } }
22:22:07       + expected - actual
22:22:07 
22:22:07        {
22:22:07          "data": {
22:22:07            "searchItems": {
22:22:07              "pageInfo": {
22:22:07       -        "hasNextPage": false
22:22:07       +        "hasNextPage": true
22:22:07                "hasPreviousPage": false
22:22:07              }
22:22:07            }
22:22:07          }

might also be caused by some hooks in the math extension that extend wikibase functionality?

This is somehow a blocker and happens reliably on every ci run. This means we can not merge any change. https://integration.wikimedia.org/ci/job/quibble-apitests-only-vendor-php83/14301/console

Change #1253708 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] GQL: Avoid search query in unrelated e2e test

https://gerrit.wikimedia.org/r/1253708

Apologies for not spotting this sooner. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1253708 should fix it.

The error I see might also be caused by some hooks in the math extension that extend wikibase functionality?

No, these errors happen because our search-related tests rely on OpenSearch to be available in the api-testing job, which it isn't (and probably doesn't need to be) for Math. I'm not entirely sure why this only seems to be causing issues for Math. I suspect that other extensions integrating with Wikibase either aren't running api-testing or aren't as active.

We added a guard to check process.env.QUIBBLE_OPENSEARCH for the actual search feature tests recently to avoid this issue, but the test that was added last week wasn't covered by the guard because it wasn't actually testing a search feature per se. I'll see if we can come up with something smarter to prevent this from happening again.

@Jakob_WMDE thank you very much for your fix.

According to https://github.com/wikimedia/integration-config/blob/d99078b266bc6e176b607b8580882c46100daa59/zuul/dependencies.yaml there are just a few extensions that depend on Wikibase. Some of them such as WikibaseManifest, seem to under WMDE maintenance. WikibaseManifest for example also depends on Wikibase and API tests

https://github.com/wikimedia/integration-config/blob/d99078b266bc6e176b607b8580882c46100daa59/zuul/layout.yaml#L5083-L5090

so in theory also WMDE maintained extensions should be affected. So maybe WMDE can establish a way of testing extensions that depend on Wikibase. I was also discussing this a few years ago during a hackathon session, but it seems CI capabilities were not at the level that for every change all the downstream tests could be executed. However, maybe there is a way of running tests daily (for example via github actions) and monitor the test results?

I have developed https://github.com/physikerwelt/mw-math-testing to run tests that are too expensive to run with WMF CI infra. I did also measure the overall carbon footprint of those test, and the good news is executing those tests consumes way less energy than a developer would emit while thinking about a fix.

Change #1253708 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] GQL: Avoid search query in unrelated e2e test

https://gerrit.wikimedia.org/r/1253708

@Physikerwelt thanks, that sounds like a reasonable suggestion. I've created T420318, and running those tests daily as you suggest or as part of Wikibase's existing secondary CI both sound doable. We'll try to get to it soon!