Page MenuHomePhabricator
Create Task
Maniphest T254031

Enable mediawiki-quibble-apitests-vendor-docker for extension Math
Closed, ResolvedPublic
Actions

Description

Follow this guide for Math.

Details

SubjectRepoBranchLines +/-
Revert "layout: [mediawiki/extensions/Math] Disable API tests, not passing yet"integration/configmaster+3 -6
layout: [mediawiki/extensions/Math] Disable API tests, not passing yetintegration/configmaster+6 -3
Enable API tests for extension Mathintegration/configmaster+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Physikerwelt created this task.May 29 2020, 6:45 PM
gerritbot added a comment.May 29 2020, 6:51 PM

Change 599963 had a related patch set uploaded (by Physikerwelt; owner: Physikerwelt):
[integration/config@master] Enable API tests for extension Math

https://gerrit.wikimedia.org/r/599963

gerritbot added a project: Patch-For-Review.May 29 2020, 6:51 PM
RhinosF1 added projects: Continuous-Integration-Config, Math.May 29 2020, 6:59 PM
Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptMay 29 2020, 6:59 PM
gerritbot added a comment.May 30 2020, 4:53 PM

Change 599963 merged by jenkins-bot:
[integration/config@master] Enable API tests for extension Math

https://gerrit.wikimedia.org/r/599963

Reedy closed this task as Resolved.May 30 2020, 4:54 PM
Reedy subscribed.

Deployed

Maintenance_bot removed a project: Patch-For-Review.May 30 2020, 5:10 PM
gerritbot added a comment.Jun 1 2020, 8:46 PM

Change 601427 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [mediawiki/extensions/Math] Disable API tests, not passing yet

https://gerrit.wikimedia.org/r/601427

gerritbot added a project: Patch-For-Review.Jun 1 2020, 8:46 PM
gerritbot added a comment.Jun 1 2020, 8:49 PM

Change 601427 merged by jenkins-bot:
[integration/config@master] layout: [mediawiki/extensions/Math] Disable API tests, not passing yet

https://gerrit.wikimedia.org/r/601427

Stashbot added a comment.Jun 1 2020, 8:49 PM

Mentioned in SAL (#wikimedia-releng) [2020-06-01T20:49:46Z] <James_F> Zuul: Disabled new mediawiki-quibble-apitests-vendor-docker for Math as it doesn't pass T254031

Jdforrester-WMF reopened this task as Stalled.Jun 1 2020, 8:49 PM
Jdforrester-WMF subscribed.

Undeployed. This job isn't passing yet. Please un-mark as stalled if and when the experimental job passes so we can re-enable it.

• Pchelolo edited projects, added Platform Team Workboards (Clinic Duty Team); removed Patch-For-Review.Jun 1 2020, 9:00 PM
• Pchelolo added a comment.Jun 1 2020, 9:18 PM

The search tests are failing because Math depends on Wikibase in CI, which in turn brings in CirrusSearch as a dependency.

Running api-testing search tests with CirrusSearch installed fails for some reason. I couldn't really find anything very relevant in the logs to explain why exactly is it failing.

• eprodromou edited projects, added Platform Engineering; removed Platform Team Workboards (Clinic Duty Team).Jun 2 2020, 8:27 PM
• eprodromou subscribed.

We think this is big enough that it should be its own small project.

• eprodromou moved this task from Inbox to Future Initiatives/Small Projects on the Platform Engineering board.Jun 2 2020, 8:27 PM
Physikerwelt removed Physikerwelt as the assignee of this task.Jun 3 2020, 5:50 AM
In T254031#6186517, @eprodromou wrote:

We think this is big enough that it should be its own small project.

How is project different from a subtask, such as enable API tests for CirrusSearch?

Currently, we see the following error.

17:49:25   5 failing
17:49:25 
17:49:25   1) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return array of pages when there is only a text match:
17:49:25 
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25 
17:49:25       -0
17:49:25       +1
17:49:25       
17:49:25       at Context.it (tests/api-testing/REST/Search.js:31:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25 
17:49:25   2) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return array of pages when there is only title match:
17:49:25 
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25 
17:49:25       -0
17:49:25       +1
17:49:25       
17:49:25       at Context.it (tests/api-testing/REST/Search.js:50:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25 
17:49:25   3) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return a single page when there is a title and text match on the same page:
17:49:25 
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25 
17:49:25       -0
17:49:25       +1
17:49:25       
17:49:25       at Context.it (tests/api-testing/REST/Search.js:61:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25 
17:49:25   4) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return two pages when both pages match:
17:49:25 
17:49:25       AssertionError: expected [] to have a length of 2 but got 0
17:49:25       + expected - actual
17:49:25 
17:49:25       -0
17:49:25       +2
17:49:25       
17:49:25       at Context.it (tests/api-testing/REST/Search.js:72:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)
17:49:25 
17:49:25   5) Search
17:49:25        GET /search/page?q={term}
17:49:25          should return only one page when two pages match but limit is 1:
17:49:25 
17:49:25       AssertionError: expected [] to have a length of 1 but got 0
17:49:25       + expected - actual
17:49:25 
17:49:25       -0
17:49:25       +1
17:49:25       
17:49:25       at Context.it (tests/api-testing/REST/Search.js:79:11)
17:49:25       at process._tickCallback (internal/process/next_tick.js:68:7)

from https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-docker/10701/console
I mean it is not really surprising that the search results differ based on the search engine. And from the test results, it seems that the data is not (yet?) ready to be queried by CS. To me, there is a pressing need for a CI test that runs all tests with all WMF deployed extensions enabled that is triggered for every core change... Maybe it consumes too much energy, but it would reduce the debugging effort after the fact.

Physikerwelt moved this task from Inbox to Blocked: needs help on the Math board.Jun 19 2020, 7:48 PM
Naike added a project: Platform Engineering Roadmap Decision Making.Oct 8 2020, 8:55 PM
daniel subscribed.Nov 2 2020, 5:59 PM
In T254031#6186517, @eprodromou wrote:

We think this is big enough that it should be its own small project.

After talking to @Physikerwelt, this seems like a simple CI config change to me. I don't think it has anything to do with us. I'll untag PET for now It's already on the CI config board.

In T254031#6183274, @Pchelolo wrote:

Running api-testing search tests with CirrusSearch installed fails for some reason. I couldn't really find anything very relevant in the logs to explain why exactly is it failing.

End-to-end tests for search will fail with CirrusSearch installed because CIrrusSearch needs an Elastic service to back it up. We don't have that in CI. The API tests for core are explicitly written to work with a vanilla install only.

For the math extension, I see two solutions: either run only the API tests for Math, not the ones for core. Or only enable the Math extension for Jenkins job that runs the API tests, not other extensions. I'm not sure how simple or hard either one is to do in quibble, but that would make it work. I hope we don't need T243976 for this.

In T254031#6187554, @Physikerwelt wrote:

To me, there is a pressing need for a CI test that runs all tests with all WMF deployed extensions enabled that is triggered for every core change... Maybe it consumes too much energy, but it would reduce the debugging effort after the fact.

Yes, sure, in some bright an shiny future. See T248683: Create and run a suite of end-to-end tests for the Wikimedia environment

Physikerwelt added a comment.Nov 3 2020, 6:14 AM

@Jdforrester-WMF do you see a way forward here?

Physikerwelt changed the task status from Stalled to Open.Nov 6 2020, 6:30 AM
Physikerwelt assigned this task to Jdforrester-WMF.

The API tests do no longer fail on an experimental run (https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-docker/23533/console). See https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Math/+/639630
However, there are two new problems both seem to be related to the test DB config.

I suspect that those errors are due to other experimental run features that would not occur if we would switch on the API tests for the normal test run.

gerritbot added a comment.Nov 6 2020, 6:52 PM

Change 639651 had a related patch set uploaded (by Physikerwelt; owner: Physikerwelt):
[integration/config@master] Revert "layout: [mediawiki/extensions/Math] Disable API tests, not passing yet"

https://gerrit.wikimedia.org/r/639651

gerritbot added a project: Patch-For-Review.Nov 6 2020, 6:52 PM
hashar subscribed.Nov 7 2020, 8:43 AM

However, there are two new problems both seem to be related to the test DB config.

That job uses SQLite as a database backend? We used to only test with SQLite until in 2015 we switched everything to use MySQL (T37912 / 2957704054eabc1ef2ffc59b2cb62754a14b0714). Nowadays SQLite is only enforced for mediawiki/core.

I suspect that those errors are due to other experimental run features that would not occur if we would switch on the API tests for the normal test run.

That job uses PostgreSQL. GeoData has been written with MySQL in mind and support for SQLite was added in 2012 by a8cb327bf078cb7b653dff18e04464beceeca5b8 which also reject any other database backend. The change passed with MySQL though.

What is important in the experimental result is that the api testing job (mediawiki-quibble-apitests-vendor-docker) did pass. One would still want to ensure it ran the expected tests

gerritbot added a comment.Nov 7 2020, 8:45 AM

Change 639651 merged by jenkins-bot:
[integration/config@master] Revert "layout: [mediawiki/extensions/Math] Disable API tests, not passing yet"

https://gerrit.wikimedia.org/r/639651

Naike moved this task from Untriaged to Icebox on the Platform Engineering Roadmap Decision Making board.Nov 16 2020, 3:24 PM
Physikerwelt closed this task as Resolved.Dec 4 2020, 3:03 PM
Physikerwelt added a subscriber: Naike.

@Naike, this task was already resolved by @hashar on Nov 7th

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL