I gave this task a bit of love as the description was out of date, although the task itself is still relevant.

Given that Wikibase is now running the quibble jobs, I think we should just drop the old mwext-repo/client jobs. If sqlite testing is still wanted, we could add the generic quibble job that runs sqlite.

In T188717#4541170, @Legoktm wrote:

Given that Wikibase is now running the quibble jobs, I think we should just drop the old mwext-repo/client jobs. If sqlite testing is still wanted, we could add the generic quibble job that runs sqlite.

They still run different different stuff, mainly with different configuration.

- builder:
    name: wd-wikibase-apply-settings
    builders:
     - shell: "$WORKSPACE/src/extensions/Wikibase/build/jenkins/mw-apply-wb-settings.sh -r {repoorclient} -e {experimental} -b false"

experimental can probably die, but the repoorclient element may still be needed, will have to check

Isn't it that quibble job runs repo and client enabled variant only? We still should have a job that has only client enabled (Wikipedia scenario). Whether done using quibble or not, is a technical detail (if done via quibble, might be worth considering excluding duplicate stuff, like browser tests, as wikibase client does not add its browser tests afair, at least not now).

Well the goal of this ticket is to get rid of all the special casing for Wikibase :)

In T188717#4541180, @Legoktm wrote:

Well the goal of this ticket is to get rid of all the special casing for Wikibase :)

I think we always want to run our tests with just client loaded, Quibble could provide a way to inject some settings before running the tests?
Another route might be to make it load a different entry point for the extension instead of Wikibase/Wikibase.php which is currently still a special case entrypoint for Jenkins only.

I guess it is kind of going to be solved once we've managed to split WikibaseClient off WikibaseRepo, and making it fully stand-alone extension, or so. The work on making WikibaseRepo and WikibaseClient use extension.json is related to this I guess.

In T188717#4541195, @WMDE-leszek wrote:

I guess it is kind of going to be solved once we've managed to split WikibaseClient off WikibaseRepo, and making it fully stand-alone extension, or so. The work on making WikibaseRepo and WikibaseClient use extension.json is related to this I guess.

Is there a ticket to link to for this split yet?

In T188717#4541201, @Addshore wrote:

In T188717#4541195, @WMDE-leszek wrote:

I guess it is kind of going to be solved once we've managed to split WikibaseClient off WikibaseRepo, and making it fully stand-alone extension, or so. The work on making WikibaseRepo and WikibaseClient use extension.json is related to this I guess.

Is there a ticket to link to for this split yet?

I guess that is T88258: Convert WikibaseRepository, WikibaseClient, WikibaseLib and WikibaseView to use extension registration

In T188717#4669782, @hashar wrote:

In T188717#4541201, @Addshore wrote:

Is there a ticket to link to for this split yet?

I guess that is T88258: Convert WikibaseRepository, WikibaseClient, WikibaseLib and WikibaseView to use extension registration

No, the extensions & entry points are already split in that way.
If you want to load client you load 1 entry point, if you want to load repo you load another.
That ticket just talks about converting from php entry points to json extension registration entry points.

Ok sorry for the confusion.

Meanwhile, I have Quibble running Wikibase PHPUnit tests just fine with:

ZUUL_PROJECT=mediawiki/extensions/Wikibase quibble --run=phpunit

Thought there are 200 skipped tests for Database less tests, related to lack of CLDR/CirrusSearch/and 79 skipped ones for Database tests. Seems to be integration tests that rely on CirrusSearch/CLDR/Scribunto/Echo etc.

So maybe we can drop the old Client vs Repo jobs? They use a script in Wikibase:

#!/bin/bash -xe

function usage {
  echo "usage: $0 -r <repo|client> -e <true|false> -b <true|false>"
  echo "       -r specify if the settings are for repo or client"
  echo "       -b specify if the settings are for a build or not"
  exit 1
}

while getopts r:e:b: opt
do
   case $opt in
       r) REPO="$OPTARG";;
       b) BUILD=$OPTARG;;
   esac
done

if [ $BUILD = true ]; then
  echo "-b true is not supported by this script anymore."
  exit 1
fi

function apply_client_settings {
  echo "client"
  echo '$wgEnableWikibaseRepo = false;' >> LocalSettings.php
  echo '$wgEnableWikibaseClient = true;' >> LocalSettings.php
  # $wgWikimediaJenkinsCI is only set later, so need to set it here, too
  echo '$wgWikimediaJenkinsCI = true;' >> LocalSettings.php
  echo '$wmgUseWikibaseRepo = false;' >> LocalSettings.php
  echo '$wmgUseWikibaseClient = true;' >> LocalSettings.php
  echo 'require_once __DIR__ . "/extensions/Wikibase/Wikibase.php";' >> LocalSettings.php
}

function apply_repo_settings {
  echo '$wgEnableWikibaseRepo = true;' >> LocalSettings.php
  echo '$wgEnableWikibaseClient = true;' >> LocalSettings.php
  # $wgWikimediaJenkinsCI is only set later, so need to set it here, too
  echo '$wgWikimediaJenkinsCI = true;' >> LocalSettings.php
  echo '$wmgUseWikibaseRepo = true;' >> LocalSettings.php
  echo '$wmgUseWikibaseClient = true;' >> LocalSettings.php
  echo 'require_once __DIR__ . "/extensions/Wikibase/Wikibase.php";' >> LocalSettings.php
}

cd $WORKSPACE/src

if [ "$(tail -n1 LocalSettings.php)" = "?>" ]
then
  PHPTAGS=true
fi
if [ -v PHPTAGS ]
then
  echo '<?php' >> LocalSettings.php
fi

if [ "$REPO" = "repo" ]
then
  apply_repo_settings
elif [ "$REPO" = "client" ]
then
  apply_client_settings
else
  usage
fi

if [ -v PHPTAGS ]
then
  echo '?>' >> LocalSettings.php
fi

Change 467675 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove obsolete Wikibase settings

https://gerrit.wikimedia.org/r/467675

Change 467675 merged by jenkins-bot:
[integration/config@master] Remove obsolete Wikibase settings

https://gerrit.wikimedia.org/r/467675

In T188717#4541189, @Addshore wrote:

In T188717#4541180, @Legoktm wrote:

Well the goal of this ticket is to get rid of all the special casing for Wikibase :)

I think we always want to run our tests with just client loaded, Quibble could provide a way to inject some settings before running the tests?
Another route might be to make it load a different entry point for the extension instead of Wikibase/Wikibase.php which is currently still a special case entrypoint for Jenkins only.

^^ I think we still want a job that only loads Client.

The Quibble jobs now report again junit xml files (T207841) which offers a way to compare the tests that have been run.

I went with a recheck of a dummy Wikibase change (Gerrit #70395) and compared the testcase names for three jobs:

The Quibble jobs runs everything, when the Wikibase jobs run only a subset of tests based on groups.

I then downloaded the Junit XML files then extracted the test names with grep -P -o 'name=".*?"' to compare the tests being run.

All tests from the client job are run.

For the repo flavor, I have the following extra tests:

-name="testErrorBeingDisplayed_WhenItemWithTheSameLabelAndDescriptionInThisLanguageAlreadyExists"
-name="testGetLabelWithDescriptionConflicts_NoUseSearchFields with data set "by label and description, different description capitalization""
-name="testGetLabelWithDescriptionConflicts_NoUseSearchFields with data set "by label and description, different label capitalization""
-name="testGetLabelWithDescriptionConflicts_NoUseSearchFields with data set "by label and description""
-name="testGetLabelWithDescriptionConflicts_NoUseSearchFields with data set "by label, empty descriptions""
-name="testGetLabelWithDescriptionConflicts_NoUseSearchFields with data set "by label, mismatching description""
-name="testGetLabelWithDescriptionConflicts_NoUseSearchFields with data set "two languages for label and description""
-name="testGetLabelWithDescriptionConflicts with data set "by label and description, different description capitalization""
-name="testGetLabelWithDescriptionConflicts with data set "by label and description, different label capitalization""
-name="testGetLabelWithDescriptionConflicts with data set "by label and description""
-name="testGetLabelWithDescriptionConflicts with data set "by label, empty descriptions""
-name="testGetLabelWithDescriptionConflicts with data set "by label, mismatching description""
-name="testGetLabelWithDescriptionConflicts with data set "two languages for label and description""
-name="testItemLabelDescriptionConflict"
-name="testReallyDoQuery"

The repo job runs with MySQL, all five tests are explicitly skipped when running on MySQL, notably because it does not allow self-joins on temporary tables.

So I think we can safely remove the legacy jobs mwext-Wikibase-repo-tests-sqlite-hhvm-jessie and mwext-Wikibase-client-tests-mysql-hhvm-jessie .

Thanks @hashar for looking into this.
A quick question checking that I understood what you wrote right.

All tests from the client job are run.

The point of having "client" job is to have tests from group WikibaseClient run AND no repo part of Wikibase enabled. Is this what could be somehow set up in quibble jobs?

The sqlite job looks cool, and seem that indeed that custom job can be removed, if we now have all those sqlite-only tests are run with quibble.

The point of having "client" job is to have tests from group WikibaseClient run AND no repo part of Wikibase enabled. Is this what could be somehow set up in quibble jobs?

We can surely create a dedicated job for that purpose. I guess the job would use the extensions:

Can probably be done by hardcoding the list of dependencies somehow.

And then just run the group WikibaseClient, which is doable in Quibble with something like --commands 'php tests/phpunit/phpunit.php --group WikibaseClient'.

The sqlite job looks cool, and seem that indeed that custom job can be removed, if we now have all those sqlite-only tests are run with quibble.

I don't think there is a Quibble job with SQLite as a backend, but that is fairly trivial to add.

I will look at adding both jobs.

And then just run the group WikibaseClient, which is doable in Quibble with something like --commands 'php tests/phpunit/phpunit.php --group WikibaseClient'.

This is different to not loading repo / not loading client.
The config var currently has to be set, using @group essentially ends up in slightly different code being run, this is not what we want to do, we want to set the config vars.

Change 470060 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Wikibase client/repo tests with Quibble

https://gerrit.wikimedia.org/r/470060

I have crafted two jobs that use quibble to run commands.

They should have the proper dependencies injected (I have triggered them manually and filled EXT_DEPENDENCIES myself).

https://integration.wikimedia.org/ci/job/wikibase-client-docker/3/consoleFull
https://integration.wikimedia.org/ci/job/wikibase-repo-docker/6/consoleFull

phpunit is invoked with --verbose to show the skipped tests.

Probably we now want to audit the test reports and ensure the proper tests are being run:
https://integration.wikimedia.org/ci/job/wikibase-client-docker/3/testReport/
https://integration.wikimedia.org/ci/job/wikibase-repo-docker/6/testReport/

(we have a nodejs app in integration/junitdiff.git which might help to compare the junit results)

Change 470060 merged by jenkins-bot:
[integration/config@master] Wikibase client/repo tests with Quibble

https://gerrit.wikimedia.org/r/470060

Change 470594 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Make Wikibase client/repo phpunit tests verbose

https://gerrit.wikimedia.org/r/470594

Change 470594 merged by jenkins-bot:
[integration/config@master] Make Wikibase client/repo phpunit tests verbose

https://gerrit.wikimedia.org/r/470594

Change 470598 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Fix phpunit --verbose for Wikibase jobs

https://gerrit.wikimedia.org/r/470598

Change 470598 merged by jenkins-bot:
[integration/config@master] Fix phpunit --verbose for Wikibase jobs

https://gerrit.wikimedia.org/r/470598

Change 470604 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove Wikibase sqlite client jobs

https://gerrit.wikimedia.org/r/470604

Change 470605 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Wikibase nodepool jobs with php7

https://gerrit.wikimedia.org/r/470605

Change 470604 merged by jenkins-bot:
[integration/config@master] Remove Wikibase sqlite client jobs

https://gerrit.wikimedia.org/r/470604

Change 470605 merged by jenkins-bot:
[integration/config@master] Wikibase nodepool jobs with php7

https://gerrit.wikimedia.org/r/470605

I wrote a quick script to list testsuite in a junit file ( P7742 ).

mwext-Wikibase-repo-tests-sqlite-php70-jessie and wikibase-repo-docker are identical.

For the client, there is a lot more differences I have to investigate. The docker / quibble based jobs runs way more tests, apparently including Repo tests.

Change 470791 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Wikibase client/repo copy LocalSettings.php

https://gerrit.wikimedia.org/r/470791

I fixed the Docker jobs so they capture LocalSettings.php AFTER build/jenkins/mw-apply-wb-settings.sh has been run. For the repo variant the tail of the file looks like:

...
wfLoadExtension( 'Elastica' );
wfLoadExtension( 'GeoData' );
require_once "$IP/extensions/Wikibase/Wikibase.php";


# End of automatically generated settings.
# Add more configuration options below.

$wgEnableWikibaseRepo = true;
$wgEnableWikibaseClient = true;
$wgWikimediaJenkinsCI = true;
$wmgUseWikibaseRepo = true;
$wmgUseWikibaseClient = true;
require_once __DIR__ . "/extensions/Wikibase/Wikibase.php";

The first require_once "$IP/extensions/Wikibase/Wikibase.php"; is done by the MediaWiki installer which is autodetecting extensions (via install.php --with-extensions).

There is no good way to inject configuration files before extensions get loaded. The way I have solved that in Quibble is to inject custom settings at the top of the file, instead of at the tail of it.

Change 470791 merged by jenkins-bot:
[integration/config@master] Wikibase client/repo copy LocalSettings.php

https://gerrit.wikimedia.org/r/470791

There is no good way to inject configuration files before extensions get loaded.

We could make mediawiki add a comment before it starts loading the extensions that it is installed with to the file?

Then in our jenkins job we could split split the file at that point and shove our config in?

Change 470798 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/Wikibase@master] build: prepend Wikibase client/repo settings

https://gerrit.wikimedia.org/r/470798

Status: I need a few patches for Wikibase to be merged in in order to have the custom configuration to be prepended in LocalSettings.php, in effect before doing require_once( __DIR__ . 'mediawiki/extensions/Wikibase.php' ). That is https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/470798/ which has two parent changes.

On that change I ran the experimental pipeline, retrieved the Junit XML files for each of the four builds:

I compared the testcases and testsuite being run using a couple scripts: P7796 P7797. They are noop:

$ colordiff -u <(./parse.py mwext-Wikibase-client-tests-mysql-php70-jessie.xml) <(./parse.py wikibase-client-docker.xml)
$ colordiff -u <(./parse.py mwext-Wikibase-repo-tests-sqlite-php70-jessie.xml) <(./parse.py wikibase-repo-docker.xml)
$

So we have a success. Just have to merge the three Wikibase patches below and we can phase out the legacy Nodepool jobs.

• https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/470798/
• https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/470796/
• https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/470788/

:)

Change 473197 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate Wikibase client/repo to Docker + cleanup

https://gerrit.wikimedia.org/r/473197

Change 470798 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] build: prepend Wikibase client/repo settings

https://gerrit.wikimedia.org/r/470798

Change 473197 merged by jenkins-bot:
[integration/config@master] Migrate Wikibase client/repo to Docker + cleanup

https://gerrit.wikimedia.org/r/473197