Drop the Zend PHP 5.5 jobs entirely
Wikimedia is on HHVM and we don't really release Wikibase for outside world reuse. Would let us drop three jobs:

Wikibase is used outside of Wikimedia and dropping all testing for zend PHP would probably be a bad idea.

In T160989#3118490, @Addshore wrote:

Drop the Zend PHP 5.5 jobs entirely
Wikimedia is on HHVM and we don't really release Wikibase for outside world reuse. Would let us drop three jobs:

Wikibase is used outside of Wikimedia and dropping all testing for zend PHP would probably be a bad idea.

What about moving it to Travis? It would help because the number of triggered jobs will be limited to the ones that get merged. I'm not sure if anyone follows travis reports in github though.

In T160989#3121754, @Ladsgroup wrote:

What about moving it to Travis? It would help because the number of triggered jobs will be limited to the ones that get merged. I'm not sure if anyone follows travis reports in github though.

Personally I would prefer to move it to a post merge job than move it to travis.

Moving some jobs to postmerge would at least prevent them from running on every patchsets. The postmerge jobs reports on the Gerrit change that triggered them, so most probably developers will pay attention to the eventual failure message.

Eg move out of test / gate-and-submit to postmerge the three jobs:

• mwext-Wikibase-repo-tests-sqlite-php55-trusty
• mwext-Wikibase-client-tests-sqlite-php55-trusty
• mwext-Wikibase-client-tests-sqlite-hhvm-jessie

Current

For gate-and-submit we end up triggering:

Job name	Description
npm-node-6-jessie	Javascript lint/style and others
composer-php55-trusty	lint/style
composer-hhvm-jessie	lint/style
php55lint	Lint on permanent slave. Composer already run parallel-lint
mwext-testextension-php55-composer-trusty	PHPUnit
mwext-testextension-hhvm-composer-jessie	PHPUnit
mwext-qunit-composer-jessie	QUnit tests with Karma
mwext-mw-selenium-composer-jessie	Browser tests based on ruby/selenium

Job name	App	DB	PHP
mwext-Wikibase-repo-tests-sqlite-php55-trusty	Repo	sqlite	php55
mwext-Wikibase-repo-tests-sqlite-hhvm-jessie	Repo	sqlite	HHVM
mwext-Wikibase-client-tests-mysql-php55-trusty	Client	MySQL	php55
mwext-Wikibase-client-tests-sqlite-php55-trusty	Client	sqlite	php55
mwext-Wikibase-client-tests-mysql-hhvm-jessie	Client	Mysql	HHVM
mwext-Wikibase-client-tests-sqlite-hhvm-jessie	Client	sqlite	HHVM

Questions

What is the difference between mwext-Wikibase-*-tests-* and mwext-testextension-*-composer-*? It seems like the latter already runs all PHPUnit tests (both client and repo). It also varies by PHP flavour already, but not by RDBMS. It would make sense to add a third variation to sqlite with one of the PHP flavours.

Proposed

Change:

• mwext-testextension-php55-composer-trusty include composer test (for lint/style)
• mwext-testextension-hhvm-composer-trusty include composer test (for lint/style)

Remove:

• php55lint (redundant with composer-php55-trusty)
• composer-php55-trusty (redundant with mwext-testextension-php55-composer-trusty)
• composer-hhvm-jessie (redundant with mwext-testextension-hhvm-composer-trusty)
• all 6 mwext-Wikibase-*-tests-* jobs. Redunant with mwext-testextension-*-composer-*.

Add:

• mwext-testextension-hhvm-sqlite-composer-trusty to ensure sqlite is covered in addition to mysql. No need to cover it with both hhvm and php55.

Overview:

This means a small amount is duplicated (2x lint/style for HHVM) but that seems marginal and unimportant, given they run in parallel anyway.

I wanted to check the overlaps between three different tests jobs, generic, client and repo:

• mwext-Wikibase-repo-tests-sqlite-hhvm-jessie
• mwext-Wikibase-client-tests-mysql-hhvm-jessie
• mwext-testextension-hhvm-composer-jessie

None of them seems to rely on mediawiki/vendor.

Config vary

The jobs invoke the Wikibase script build/jenkins/mw-apply-wb-settings.sh which to vary $wg settings:

$wgEnableWikibaseRepo = false;                                                                                                
$wgEnableWikibaseClient = true;                                                                                               
$wgWikimediaJenkinsCI = true;                                                                                                 
$wmgUseWikibaseRepo = false;                                                                                                  
$wmgUseWikibaseClient = true;

$wgEnableWikibaseRepo = true;                                                                                                        
$wgEnableWikibaseClient = true;                                                                                                      
$wgWikimediaJenkinsCI = true;                                                                                                        
$wmgUseWikibaseRepo = true;                                                                                                          
$wmgUseWikibaseClient = true;

Extensions dependencies:

PHPUnit is invoked with --group filters:

Tests list analysis

I have picked the JUnit result files from each three jobs, parsed them to render a text only tree of tests then used vimdiff to compare the three files. The scripts are P5173 and P5174.

client vs repo

They both run Wikibase group albeit with different LocalSettings.

Some Wikibase\Client\Tests are also run in repo.

In both jobs, some tests ends up with 0 assertions. Maybe due to different tags.

client does not run any Wikibase\Repo\Tests (good).

generic vs client

It is almost a full overlap, but some tests in client run extra tests and assertions for some reason. Probably because they have a slightly different set of MediaWiki extensions or some tests are skipped based on the database backend.

Utilities:

Tests trees for diff:

Tests are indented by two spaces, each line show the testsuite name, the number of tests and number of assertions:

Change:

• mwext-testextension-php55-composer-trusty include composer test (for lint/style)
• mwext-testextension-hhvm-composer-trusty include composer test (for lint/style)

That is definitely a good point. We should do that for all MediaWiki repositories, I have filled T161895 for that.

What @Krinkle has suggested sounds like a good plan at glance!
We just need to make sure that all tests are run in such set up. I will be looking at differences between jobs, and keep spamming here if I find something relevant.

Wikimedia solely uses MySQL, my understanding is that the sqlite jobs were kept around since developers tend to use sqlite on their development box.

This in general sounds like a good explanation of why sqlite jobs are around. As such it makes them quite good candidates for jobs to be kicked out.
However, as far as I understand, there are few tests that are skipped in mysql jobs and only run as part of sqlite jobs. Those are test of functionality where there some more complex DB queries are done. PHPUnit test use temporary tables and MySQL does not allow self-joins on temporary tables, so those test must be skipped. We still want to have those test run, so that seems like a good reason to
That said, there is only a handful of such tests. Maybe it makes sense to have a sqlite job running only those tests that are skipped in mysql jobs? That would should be quite performant, I assume. We could use a separate @group tag for those or so.
As far as I know, for such a sqlite job it would not be needed to have repo/client split (just run test with both components enabled), same for php55/hhvm (hhvm should be enough I guess).

With https://phabricator.wikimedia.org/T161895 in, I've had another look at Wikibase Jenkins jobs. Current jobs running PHPUnit jobs look as follows:

test jobs

gate-and-submit jobs

Possible improvements:

• It seems we don't need jobs running tests against sqlite with only Client enabled. We should be OK with only running does test in the Repo+Client set up
• Similar thing regarding php55 job - it should be enough to only run sqlite job with hhvm php flavour
• We actually might not even need mwext-testextension-php55-composer-trusty but dropping this would mean Wikibase is configured different than all other extensions, so we don't want to follow this path :)
• Further optimisation could be to make sqlite jobs only run PHPUnit tests that require sqlite, not all tests. There is only handful of those tests in Wikibase, so this would make this job/jobs faster. But this is something to do as a next step.

So what I believe would make sense from the Wikibase/Wikidata point of view would be the following set of jobs:
test

gate and submit

Together with @Addshore we've tinkered a patch dropping a few jobs as describe above: https://gerrit.wikimedia.org/r/354725.
@hashar: if you think changes like this make sense for you, we would be happy to try it out!

Change 354725 merged by jenkins-bot:
[integration/config@master] Reduce the number of Wikibase jobs

https://gerrit.wikimedia.org/r/354725

Thanks @hashar! In the next days I'll try to make the change to the sqlite job so it is smaller and faster (we will still need the instance, though).