It looks like there is already some infrastructure set up for this in jjb/wikidata.yaml added as part of T285049: Run CI tests daily on master for ungated extensions

- project:
    name: wikidata-quibble-daily-notgated
    # Wikidata extensions that are not part of gate
    project:
        # The commented out ones are because of T285049#7311916
        # - ArticlePlaceholder
        - Cognate
        - InterwikiSorting
        # - WikibaseLexeme
        # - WikibaseManifest
        # - WikibaseQualityConstraints
        # - Wikidata.org
    jobs:
        - quibble-daily-{project}-{packages-source}-{database}-{php}:
            database: mysql
            packages-source: vendor
            php:
                - php74:
                    docker_image: docker-registry.wikimedia.org/releng/quibble-buster-php74:1.10.0
                    entrypoint: quibble-with-supervisord

So for a very easy hack, we could just add MediaWiki core to the list, and then we should probably also rename the file and document its purpose (not-gated Wikidata extensions, and checking core tests when run serially).

What we may want, though, is a job that runs core + all extensions (in same job would be fine), and afaict that wouldn't fit well into this sectin of jjb config. cc @ArthurTaylor @hashar.

Where this job could log in when it fails? Maybe it could push messages to mediawiki-core-bots channel for better visibility?

In T372618#10164215, @pmiazga wrote:

Where this job could log in when it fails? Maybe it could push messages to mediawiki-core-bots channel for better visibility?

I'd suggest we add a few individual email addresses to be alerted (I'd volunteer mine). If it's easy to push to mediawiki-core-bots, that sounds fine. We could also make an email alert that routes to a Slack channel.

Sounds good. Also happy to be on that e-mail list if it turns into emails.

We already have mwcore-phpunit-coverage-master which runs mediawiki/core tests serially twice per day and alerts on IRC whenever it fails (it is not ideal but that is how I notice it fails).

The wmf-quibble* jobs receive the list of extensions/skins from zuul/parameter_functions.py and each of the participating repositories must trigger the jobs via zuul/layout.yaml. I'd rather avoid having to maintain another list in a Jenkins job, then I imagine potentially it could clone integration/config.git to retrieve the set of repositories to test. That needs some code to clone the repo, extract the list and somehow pass it as arguments to the Quibble command.

An alternative is to create a new pipeline in Zuul that triggers on a daily basis. It would inject the same list of repositories and email whenever there is a failure. We don't have any such pipeline though.

For notification, Phabricator supports creating a task from a received email though I don't know whether we ever used that. Maybe there could be a mediawiki-qa@phabricator.wikimedia.org list that would generate a task with the appropriate tags / subscribers.

Note: the wmf-quibble* jobs do not run the PHPUnit tests from mediawiki/core but the extensions (or skins) testsuites. I am pretty sure mediawiki/core integration tests fail with extensions/skins loaded due to hooks interfering.

In T372618#10170375, @hashar wrote:

Note: the wmf-quibble* jobs do not run the PHPUnit tests from mediawiki/core but the extensions (or skins) testsuites. I am pretty sure mediawiki/core integration tests fail with extensions/skins loaded due to hooks interfering.

So perhaps we would need two, daily serial jobs: 1) the mediawiki/core one 2) the extensions testsuite one.

Change #1087442 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[integration/config@master] Add docker image for quibble linear test runner

https://gerrit.wikimedia.org/r/1087442

Change #1087458 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[integration/config@master] Add daily job for quibble linear phpunit testing

https://gerrit.wikimedia.org/r/1087458

Change #1088284 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[integration/config@master] Export list of gated skins and extensions as environment variables

https://gerrit.wikimedia.org/r/1088284

@hashar and Release-Engineering-Team, @ArthurTaylor and I could use some guidance on how to move forward with this (please see discussions in attached patches).

In my experience the troubles is with parallelizing tests which cause them to hit various kind of race conditions or fail cause they put to much pressure. The parallelism has been rolled out and most of the issues were caught with little troubles, so I think that is good as is.

As found in the series of patches, it is not entirely straightforward to run a job on a daily basis from Jenkins and having the context injected from Zuul parameters function. I don't think there is much value in running them serially and I'd like to not introduce the extra complexity that will have to be migrated (the current Zuul/Jenkins will eventually be replaced).

Change #1088284 abandoned by Hashar:

[integration/config@master] Export list of gated skins and extensions as environment variables

Reason:

After a discussion with Kosta I am declining this and T372618 (see task for details)

https://gerrit.wikimedia.org/r/1088284

Change #1087458 abandoned by Hashar:

[integration/config@master] Add daily job for quibble linear phpunit testing

Reason:

After a discussion with Kosta I am declining this and T372618 (see task for details)

https://gerrit.wikimedia.org/r/1087458

Change #1087442 abandoned by Hashar:

[integration/config@master] Add docker image for quibble linear test runner

Reason:

After a discussion with Kosta I am declining this and T372618 (see task for details)

https://gerrit.wikimedia.org/r/1087442