Run WikibaseLexeme tests with Wikibase CI
Closed, ResolvedPublic

Description

When a change happens in Wikibase that breaks something in Lexeme we don't know until it has already been merged.

This is a problem because we could plausibly have a release cut at this stage. AFAICT no additional CI jobs are run for Lexeme when the release branch is cut so I think these breaks could get as far as wikidata.org

An example of a (minor) breaking change is: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/448005/
Which caused breaks in some of the Lexeme tests such as: https://integration.wikimedia.org/ci/job/mwext-testextension-php70-composer-jessie/1083/console
We fixed this in https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikibaseLexeme/+/448016/

In order to prevent this happening we may want to run the Lexeme test suite on changes to Wikibase. One question is should this test be voting? If so does it put us in catch-22 and stop any breaking changes (even desired ones) in Wikibase.

We may also want to run a nightly test to give us a warning.

Tarrow created this task.Jul 27 2018, 10:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 27 2018, 10:54 AM
Tarrow updated the task description. (Show Details)Jul 27 2018, 10:54 AM

Change 448468 had a related patch set uploaded (by Tarrow; owner: Tarrow):
[integration/config@master] Run WikibaseLexeme Jobs on Wikibase

https://gerrit.wikimedia.org/r/448468

I've uploaded a "strawman" patch to be clear what I'm suggesting and give a place to track the discussion.

Change 448468 merged by jenkins-bot:
[integration/config@master] Run WikibaseLexeme Jobs on Wikibase

https://gerrit.wikimedia.org/r/448468

Not very helpful comment: AFAIK (but can be wrong, so pinging @Addshore so an expert) there is no CI job which is run when deployment branches are cut. Such thing could also be an option for this particular problem, and for developing extension to be deployed on WMF wikis in general. But I imagine this would come with the own list of challenges (e.g. I assume no single wiki has ALL the extensions, there is N combinations instead, etc).

oh, that was fast :)

Validating CI for Wikibase using https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/447080/ (which I just rebased).
If PS2 there passes then we can close this

Restricted Application added a project: User-Addshore. · View Herald TranscriptJul 27 2018, 11:42 AM
Addshore moved this task from Backlog to In progress on the Continuous-Integration-Config board.
Addshore added a comment.EditedJul 27 2018, 12:50 PM

https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/1899/console

11:49:20 FILE: .../phpunit/includes/Search/Elastic/DispatchingQueryBuilderTest.php
11:49:20 ----------------------------------------------------------------------
11:49:20 FOUND 0 ERRORS AND 1 WARNING AFFECTING 1 LINE
11:49:20 ----------------------------------------------------------------------
11:49:20  12 | WARNING | Duplicate interface name
11:49:20     |         | "CirrusSearch\Query\FullTextQueryBuilder" found; first
11:49:20     |         | defined in
11:49:20     |         | /workspace/src/extensions/Wikibase/tests/phan/stubs/cirrussearch.php
11:49:20     |         | on line 193 (Generic.Classes.DuplicateClassName.Found)
11:49:20 ----------------------------------------------------------------------
11:49:20 
11:49:20 Time: 2 mins, 29.15 secs; Memory: 17.52Mb

sounds like phpcs should ignore the phan stubs?

Change 448511 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] phpcs ignore phan stubs

https://gerrit.wikimedia.org/r/448511

Change 448511 abandoned by Addshore:
phpcs ignore phan stubs

https://gerrit.wikimedia.org/r/448511

Addshore closed this task as Resolved.Jul 27 2018, 1:45 PM

The patch linked above fixes the 1 issue

Addshore reopened this task as Open.EditedAug 6 2018, 4:27 PM

Re opening as this needs to be re solved as we had to stop WikibaseLexeme from running with Wikibase due to all of the recent CI issues.

My recent test with Lexeme running with Wikibase with the ContentTranslation CI produced the following: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/9972/console

16:09:26 There were 2 errors:
16:09:26 
16:09:26 1) Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest::testAddedStatementsWithLexemesAsTargetDisplayLemma
16:09:26 Hamcrest\AssertionError: Expected: is valid html piece having child (with tag name "ins" and having child (with tag name "a" and having text contents "foo"))
16:09:26      but: valid html piece having no children (with tag name "ins" and having child (with tag name "a" and having text contents "foo"))
16:09:26 Actual html:
16:09:26 <tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">property / <a title="Property:P1" href="/index.php/Property:P1">P1</a></td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span><a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span></span></ins></div></td></tr><tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">property / <a title="Property:P1" href="/index.php/Property:P1">P1</a>: <a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span> / rank</td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span>Normal rank</span></ins></div></td></tr>
16:09:26 
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:115
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:52
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php:29
16:09:26 /workspace/src/tests/phpunit/HamcrestPHPUnitIntegration.php:31
16:09:26 /workspace/src/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Diff/LexemeDiffVisualizerIntegrationTest.php:114
16:09:26 /workspace/src/tests/phpunit/MediaWikiTestCase.php:475
16:09:26 /workspace/src/maintenance/doMaintenance.php:94
16:09:26 
16:09:26 2) Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest::testAddedStatementsOnFormsTargettingFormsAreDisplayedAsLinks
16:09:26 Hamcrest\AssertionError: Expected: is valid html piece having child (with tag name "ins" and having child (with tag name "a" and having text contents "baz"))
16:09:26      but: valid html piece having no children (with tag name "ins" and having child (with tag name "a" and having text contents "baz"))
16:09:26 Actual html:
16:09:26 <tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">Form / property / <a title="Property:P1" href="/index.php/Property:P1">P1</a></td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span><a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1-F1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span></span></ins></div></td></tr><tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">Form / property / <a title="Property:P1" href="/index.php/Property:P1">P1</a>: <a title="Lexeme:L1" href="/index.php/Lexeme:L1">L1-F1</a> <span class="error wb-format-error">The value's type "wikibase-entityid" does not match property's type "string".</span> / rank</td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span>Normal rank</span></ins></div></td></tr>
16:09:26 
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:115
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest/MatcherAssert.php:52
16:09:26 /workspace/src/vendor/hamcrest/hamcrest-php/hamcrest/Hamcrest.php:29
16:09:26 /workspace/src/tests/phpunit/HamcrestPHPUnitIntegration.php:31
16:09:26 /workspace/src/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Diff/LexemeDiffVisualizerIntegrationTest.php:448
16:09:26 /workspace/src/tests/phpunit/MediaWikiTestCase.php:475
16:09:26 /workspace/src/maintenance/doMaintenance.php:94
16:09:26 
16:09:26 FAILURES!
16:09:26 Tests: 3789, Assertions: 13750, Errors: 2, Skipped: 25.

This matches up to T197771

Addshore renamed this task from Changes in Wikibase that break WikibaseLexeme are not visible before merge to Run WikibaseLexeme tests with Wikibase CI.Aug 6 2018, 5:57 PM
Addshore claimed this task.Aug 13 2018, 1:34 PM

Change 452361 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Revert "Temporarily exclude WikibaseLexeme from jobs including Wikibase"

https://gerrit.wikimedia.org/r/452361

Change 452361 merged by jenkins-bot:
[integration/config@master] Revert "Temporarily exclude WikibaseLexeme from jobs including Wikibase"

https://gerrit.wikimedia.org/r/452361

Addshore reopened this task as Open.Aug 17 2018, 8:41 AM

Even though the patch is merged and apparently the Lexeme tests get run with the ContentTranslation CI due to their including through Wikibase apparently they don't actually get run during the Wikibase CI itself.

Looking at the CLI debug log for https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-hhvm-docker/3199/ which has phpunit debugging turned on so it would output all tests run I do not see Lexeme tests, for example SpecialnewLexemeTest...

I discovered this while figuring out how T202114 got through our CI.....

addshore@deployment-deploy01:~$ cat test-cli-log.txt| grep "Start test" | grep SpecialNew
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Hooks\HtmlPageLinkRendererBeginHookHandlerTest::testDoHtmlPageLinkRendererBegin_overrideSpecialNewEntityLink with data set #0
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Hooks\HtmlPageLinkRendererBeginHookHandlerTest::testDoHtmlPageLinkRendererBegin_overrideSpecialNewEntityLink with data set #1
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testAllNecessaryFormFieldsArePresent_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testSiteAndPageInputFieldsWithPredefinedValuesPresent_WhenRenderedWithGetParametersPassed
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testLabelAndDescriptionValuesAreSetAccordingToSubpagePath_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenItemWithTheSameLabelAndDescriptionInThisLanguageAlreadyExists
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorAboutNonExistentPageIsDisplayed_WhenSiteExistsButPageDoesNot
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenLabelIsInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenDescriptionIsInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenAliasIsInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testWhenAliasesAreInvalid_ThenHtmlContainsErrorMessage
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only label is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "another language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only description is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "single alias"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "multiple aliases"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "nontrimmed label"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "nontrimmed description"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "all input is present"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown site identifier"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "all fields are empty"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "empty label and description, aliases contain only spaces and pipe symbols"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testMediaWikiTestCaseParentSetupCalled
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewItemTest::testValidCovers
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testAllNecessaryFormFieldsArePresent_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testLabelAndDescriptionAndDataTypeValuesAreSetAccordingToSubpagePath_WhenRendered
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testFailsAndDisplaysAnError_WhenTryToCreateSecondPropertyWithTheSameLabel
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only label is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "another language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "only description is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "single alias"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "multiple aliases"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "another datatype is set"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testEntityIsBeingCreated_WhenValidInputIsGiven with data set "all input is present"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown language"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "unknown datatype"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "all fingerprint fields are empty"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "empty label and description, aliases contain only spaces and pipe symbols"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testErrorBeingDisplayed_WhenInvalidInputIsGiven with data set "empty datatype"
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testMediaWikiTestCaseParentSetupCalled
[PHPUnitCommand] Start test Wikibase\Repo\Tests\Specials\SpecialNewPropertyTest::testValidCovers

Even though WikibaseLexme is listed to be loaded with Wikibase in parameter_functions.py it doesn't seem to actually be loading that, even after reloading zuul again just now.

Addshore moved this task from Done ✔️ to In Progress on the User-Addshore board.EditedAug 17 2018, 12:07 PM

so Wikibase doesn't run the Lexeme tests
it is only loaded in the "old" wikibase jobs, mwext-Wikibase-{kind}-tests-{dbflavor}-{phpflavor}-{image}
but that runs with either --group Wikibase,WikibaseAPI,Purtle or --group Wikibase,WikibaseClient
so lexeme jobs will never run there
and the wmf- prefixed quibble doesn't actually load Wikibase lexeme
ContentTranslation that keeps seeming to break (as in T202114) is running quibble-vendor-mysql-hhvm-docker (without the wmf- prefix) that actually loads Wikibase and Lexeme together and runs the tests...

Change 453385 had a related patch set uploaded (by Addshore; owner: Addshore):
[integration/config@master] Add extension-quibble template to Wikibase

https://gerrit.wikimedia.org/r/453385

Change 453385 merged by jenkins-bot:
[integration/config@master] Add extension-quibble template to Wikibase

https://gerrit.wikimedia.org/r/453385

Mentioned in SAL (#wikimedia-releng) [2018-08-17T12:32:21Z] <addshore> reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)

Addshore closed this task as Resolved.Aug 17 2018, 12:35 PM

Resulting job list now looks like this:

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/11201/console is the first example run and Lexeme and other Wikibase extensions can be seen loaded...