Details

Event Timeline

Change 599963 had a related patch set uploaded (by Physikerwelt; owner: Physikerwelt):
[integration/config@master] Enable API tests for extension Math

https://gerrit.wikimedia.org/r/599963

Restricted Application added a subscriber: RhinosF1. May 29 2020, 6:59 PM

Change 599963 merged by jenkins-bot:
[integration/config@master] Enable API tests for extension Math

https://gerrit.wikimedia.org/r/599963

Reedy closed this task as Resolved.May 30 2020, 4:54 PM

Deployed

Change 601427 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/extensions/Math] Disable API tests, not passing yet

https://gerrit.wikimedia.org/r/601427

Change 601427 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/extensions/Math] Disable API tests, not passing yet

https://gerrit.wikimedia.org/r/601427

Mentioned in SAL (#wikimedia-releng) [2020-06-01T20:49:46Z] <James_F> Zuul: Disabled new mediawiki-quibble-apitests-vendor-docker for Math as it doesn't pass T254031

Jdforrester-WMF reopened this task as Stalled.Jun 1 2020, 8:49 PM

Undeployed. This job isn't passing yet. Please un-mark as stalled if and when the experimental job passes so we can re-enable it.

The search tests are failing because Math depends on Wikibase in CI, which in turn brings in CirrusSearch as a dependency.

Running api-testing search tests with CirrusSearch installed fails for some reason. I couldn't really find anything very relevant in the logs to explain why exactly is it failing.

We think this is big enough that it should be its own small project.

Physikerwelt removed Physikerwelt as the assignee of this task.Jun 3 2020, 5:50 AM

We think this is big enough that it should be its own small project.

How is project different from a subtask, such as enable API tests for CirrusSearch?

Currently, we see the following error.

17:49:25   5 failing
17:49:25
17:49:25   1) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return array of pages when there is only a text match:
17:49:25
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25
17:49:25       -0
17:49:25       +1
17:49:25
17:49:25       at Context.it (tests/api-testing/REST/Search.js:31:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25
17:49:25   2) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return array of pages when there is only title match:
17:49:25
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25
17:49:25       -0
17:49:25       +1
17:49:25
17:49:25       at Context.it (tests/api-testing/REST/Search.js:50:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25
17:49:25   3) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return a single page when there is a title and text match on the same page:
17:49:25
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25
17:49:25       -0
17:49:25       +1
17:49:25
17:49:25       at Context.it (tests/api-testing/REST/Search.js:61:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25
17:49:25   4) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return two pages when both pages match:
17:49:25
17:49:25       AssertionError: expected [] to have a length of 2 but got 0
17:49:25       + expected - actual
17:49:25
17:49:25       -0
17:49:25       +2
17:49:25
17:49:25       at Context.it (tests/api-testing/REST/Search.js:72:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25
17:49:25   5) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return only one page when two pages match but limit is 1:
17:49:25
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25
17:49:25       -0
17:49:25       +1
17:49:25
17:49:25       at Context.it (tests/api-testing/REST/Search.js:79:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)

from https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-docker/10701/console
I mean it is not really surprising that the search results differ based on the search engine. And from the test results, it seems that the data is not (yet?) ready to be queried by CS. To me, there is a pressing need for a CI test that runs all tests with all WMF deployed extensions enabled that is triggered for every core change... Maybe it consumes too much energy, but it would reduce the debugging effort after the fact.