Page MenuHomePhabricator

Set up CI for mediawiki/tools/api-testing
Closed, ResolvedPublic5 Story Points

Description

The mediawiki/tools/api-testing repo contains integration tests for the MediaWiki action API. It should have a CI job set up that ensures that all the tests in it pass against the current master version of MediaWiki.

The job needs to do the following:

  • spin up an empty MediaWiki instance
  • run npm install to pull in any dependencies (from a local mirror?)
  • adjust config.json for the target wiki
  • run npm test to run all tests

A similar CI job could later be added to the core repo, to ensure that any patches submitted for MediaWiki core don't break the API.

Details

Related Gerrit Patches:

Event Timeline

daniel created this task.Aug 12 2019, 12:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2019, 12:04 PM
daniel updated the task description. (Show Details)Aug 12 2019, 12:05 PM
WDoranWMF triaged this task as Normal priority.Aug 12 2019, 3:24 PM
WDoranWMF added a project: Core Platform Team.
WDoranWMF added subscribers: Pchelolo, mobrovac, WDoranWMF.

@mobrovac / @Pchelolo do either of you have experience setting this up that you'd be able to advise @daniel

@mobrovac / @Pchelolo do either of you have experience setting this up that you'd be able to advise @daniel

Since this invloves setting up MW in a containerised environment, it would be good if Release-Engineering-Team could help us out.

Cool, there is no rush on this, just wanted to check if it was something we had past experience on wiring. I can loop in RelEng. Thanks!

It doesn't seem to hav anything to do with the clinic duty. Untagging.

hashar added a subscriber: hashar.Sep 3 2019, 3:20 PM

I would guess the easiest is to add some kind of entry point in the mediawiki/tools/api-testing repository, then we can use Quibble to setup MediaWiki and expose it over a webserver to run commands against.

The command would then be something like that:

quibble --commands './mediawiki/tools/api-testing/ci.sh' mediawiki/tools/api-testing

(last argument is an extra repository to clone)

Quibble spawns the webserver on http://127.0.0.1:9412/ so that can probably be hardcoded in the ci.sh script.

From there, a job would just use a Docker container ( eg docker-registry.wikimedia.org/releng/quibble-stretch-php72 ) and pass the parameters. Then the job get added to mediawiki/tools/api-testing and magic!

I will be happy to demo the usage of Quibble.

daniel added a comment.EditedSep 3 2019, 3:23 PM

I would guess the easiest is to add some kind of entry point in the mediawiki/tools/api-testing repository, then we can use Quibble to setup MediaWiki and expose it over a webserver to run commands against.

The entry point is just npm test.
Can we tell quibble to run npm install first?

The address of that web server would need to be injected into config.json.

WDoranWMF set the point value for this task to 5.Sep 4 2019, 6:40 PM
hashar added a comment.Sep 6 2019, 2:45 PM

Gave it a try and I have something working!

I first bootstrapped it with:

mkdir workspace && cd workspace
quibble --git-cache /home/hashar/projects --db sqlite -c 'cd mediawiki/tools/api-testing && npm install' mediawiki/tools/api-testing

Then I have replaced src/mediawiki/tools/api-testing/config.json using settings hardcoded in Quibble (url, username and password):

src/mediawiki/tools/api-testing/config.json
{
  "base_uri": "http://127.0.0.1:9412/",
  "main_page": "Main_Page",
  "root_user": {
    "name": "WikiAdmin",
    "password": "testwikijenkinspass"
  }
}

The for subsequent runs I have to:

  • delete the generated LocalSettings.php from the previous run (rm src/LocalSettings.php)
  • do not git checkout anything (pass --skip-zuul to Quibble)
  • do not reinstall mediawiki dependencies (pass --skip-deps to Quibble)
  • stop running npm install everytime

So after the bootstrap commands above, I ran:

rm -f src/LocalSettings.php
quibble \
    --skip-zuul --skip-deps \
    --git-cache /home/hashar/projects --db sqlite \
    -c 'cd mediawiki/tools/api-testing && npm test'

Which ultimately yield:

85 passing (37s)
2 failing

The whole console output: P9050

hashar added a comment.Sep 6 2019, 2:46 PM

The two tests failing are:

1) Move action
     should move a page without a redirect or its subpages and talkpages:
   AssertionError: expected '' to not exist
    at Context.it (test/Move.js:43:16)
    at process._tickCallback (internal/process/next_tick.js:68:7)
2) Move action
     should move a page with a redirect and its subpages and talkpages:
   TypeError: Cannot read property 'from' of undefined
    at Context.it (test/Move.js:73:47)
    at process._tickCallback (internal/process/next_tick.js:68:7)
daniel added a comment.Sep 6 2019, 3:00 PM

@hashar this is excellent progress, thank you!

The Move test seems to be broken. I'm getting the same result locally. I'll check it out.

Change 535712 had a related patch set uploaded (by Holger Knust; owner: Holger Knust):
[mediawiki/tools/api-testing@master] api-testing: Preparation for CI integration

https://gerrit.wikimedia.org/r/535712

Change 535747 had a related patch set uploaded (by Holger Knust; owner: Holger Knust):
[integration/config@master] WIP: api-testing: Add job template and job for api-testing

https://gerrit.wikimedia.org/r/535747

Change 535747 merged by jenkins-bot:
[integration/config@master] api-testing: Add job template and non voting jobs

https://gerrit.wikimedia.org/r/535747

Mentioned in SAL (#wikimedia-releng) [2019-09-13T16:19:24Z] <James_F> Zuul: Add initial jobs for mediawiki/tools/api-testing T230340

OK, the tests are installed in non-voting mode, but https://gerrit.wikimedia.org/r/c/mediawiki/tools/api-testing/+/535712 doesn't pass yet (trivial coding style issue means it doesn't get to the real tests yet).

Change 537065 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/tools/api-testing@master] Make Move.js test less brittle.

https://gerrit.wikimedia.org/r/537065

Change 537118 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[integration/quibble@master] Force value of $wgSecretKey.

https://gerrit.wikimedia.org/r/537118

Change 537065 merged by jenkins-bot:
[mediawiki/tools/api-testing@master] Make Move.js test less brittle.

https://gerrit.wikimedia.org/r/537065

Change 537118 merged by jenkins-bot:
[integration/quibble@master] Allow appending values to LocalSettings.php, and set value of $wgSecretKey

https://gerrit.wikimedia.org/r/537118

Change 535712 merged by jenkins-bot:
[mediawiki/tools/api-testing@master] api-testing: Preparation for CI integration

https://gerrit.wikimedia.org/r/535712

Change 542210 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use $wgSecretKey in api-testing job

https://gerrit.wikimedia.org/r/542210

Adam Wight eventually found out a creative and nice way to solve the $wgSecretKey injection to LocalSettings.php. I have updated the CI jobs to switch to Quibble 0.0.38 and the suite almost pass:

96 passing (16s)
1 failing

1) Reparse of dependent pages
     should get page with updated template and link:
   AssertionError: expected '<div class="mw-parser-output"><p><a href="/index.php?title=Template:Templatep3IwPjAgci&amp;action=edit&amp;redlink=1" class="new" title="Template:Templatep3IwPjAgci (page does not exist)">Template:Templatep3IwPjAgci</a> <a href="/index.php?title=Link3igPVD7X7U&amp;action=edit&amp;redlink=1" class="new" title="Link3igPVD7X7U (page does not exist)">Link3igPVD7X7U</a>\n</p>\n<!-- \nNewPP limit report\nParsed by a05419648e24\nCached time: 20191010195025\nCache expiry: 86400\nDynamic content: false\nComplications: []\nCPU time usage: 0.004 seconds\nReal time usage: 0.004 seconds\nPreprocessor visited node count: 2/1000000\nPreprocessor generated node count: 0/1000000\nPost‐expand include size: 32/2097152 bytes\nTemplate argument size: 0/2097152 bytes\nHighest expansion depth: 2/40\nExpensive parser function count: 0/100\nUnstrip recursion depth: 0/20\nUnstrip post‐expand size: 0/5000000 bytes\n-->\n<!--\nTransclusion expansion time report (%,ms,calls,template)\n100.00%    0.678      1 Template:Templatep3IwPjAgci\n100.00%    0.678      1 -total\n-->\n\n<!-- Saved in parser cache with key wikidb:pcache:idhash:30-0!canonical and timestamp 20191010195025 and revision id 54\n -->\n</div>' to not include 'redlink=1'
    at Context.<anonymous> (test/ReparseDependentPages.js:33:16)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Change 542210 merged by jenkins-bot:
[integration/config@master] Use $wgSecretKey in api-testing job

https://gerrit.wikimedia.org/r/542210

Change 542223 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: Make mediawiki-quibble-api-testing* voting

https://gerrit.wikimedia.org/r/542223

Change 542223 merged by jenkins-bot:
[integration/config@master] layout: Make mediawiki-quibble-api-testing* voting

https://gerrit.wikimedia.org/r/542223

Mentioned in SAL (#wikimedia-releng) [2019-10-10T20:52:57Z] <James_F> Zuul: Make mediawiki-quibble-api-testing* voting T230340