It looks like there is already some infrastructure set up for this in jjb/wikidata.yaml added as part of T285049: Run CI tests daily on master for ungated extensions

- project:
    name: wikidata-quibble-daily-notgated
    # Wikidata extensions that are not part of gate
    project:
        # The commented out ones are because of T285049#7311916
        # - ArticlePlaceholder
        - Cognate
        - InterwikiSorting
        # - WikibaseLexeme
        # - WikibaseManifest
        # - WikibaseQualityConstraints
        # - Wikidata.org
    jobs:
        - quibble-daily-{project}-{packages-source}-{database}-{php}:
            database: mysql
            packages-source: vendor
            php:
                - php74:
                    docker_image: docker-registry.wikimedia.org/releng/quibble-buster-php74:1.10.0
                    entrypoint: quibble-with-supervisord

So for a very easy hack, we could just add MediaWiki core to the list, and then we should probably also rename the file and document its purpose (not-gated Wikidata extensions, and checking core tests when run serially).

What we may want, though, is a job that runs core + all extensions (in same job would be fine), and afaict that wouldn't fit well into this sectin of jjb config. cc @ArthurTaylor @hashar.

Where this job could log in when it fails? Maybe it could push messages to mediawiki-core-bots channel for better visibility?

In T372618#10164215, @pmiazga wrote:

Where this job could log in when it fails? Maybe it could push messages to mediawiki-core-bots channel for better visibility?

I'd suggest we add a few individual email addresses to be alerted (I'd volunteer mine). If it's easy to push to mediawiki-core-bots, that sounds fine. We could also make an email alert that routes to a Slack channel.

Sounds good. Also happy to be on that e-mail list if it turns into emails.

We already have mwcore-phpunit-coverage-master which runs mediawiki/core tests serially twice per day and alerts on IRC whenever it fails (it is not ideal but that is how I notice it fails).

The wmf-quibble* jobs receive the list of extensions/skins from zuul/parameter_functions.py and each of the participating repositories must trigger the jobs via zuul/layout.yaml. I'd rather avoid having to maintain another list in a Jenkins job, then I imagine potentially it could clone integration/config.git to retrieve the set of repositories to test. That needs some code to clone the repo, extract the list and somehow pass it as arguments to the Quibble command.

An alternative is to create a new pipeline in Zuul that triggers on a daily basis. It would inject the same list of repositories and email whenever there is a failure. We don't have any such pipeline though.

For notification, Phabricator supports creating a task from a received email though I don't know whether we ever used that. Maybe there could be a mediawiki-qa@phabricator.wikimedia.org list that would generate a task with the appropriate tags / subscribers.

Note: the wmf-quibble* jobs do not run the PHPUnit tests from mediawiki/core but the extensions (or skins) testsuites. I am pretty sure mediawiki/core integration tests fail with extensions/skins loaded due to hooks interfering.

In T372618#10170375, @hashar wrote:

Note: the wmf-quibble* jobs do not run the PHPUnit tests from mediawiki/core but the extensions (or skins) testsuites. I am pretty sure mediawiki/core integration tests fail with extensions/skins loaded due to hooks interfering.

So perhaps we would need two, daily serial jobs: 1) the mediawiki/core one 2) the extensions testsuite one.