Regularly run mwext-{name}-testextension-* jobs to make sure they are still passing after core or dependency changes
Open, LowPublic

Description

In #mediawiki-i18n Nemo said that l10n-bot should not be a first line regression tester now that jenkins is checking l10n-bot commits.

hashar said it is possible for zuul to trigger jobs daily, except it has no idea who to report failures to. So as a first step I propose that we run these jobs daily, and create a dashboard of some sort that lists whether extensions are passing or failing. If it is useful, we can configure notifications for those who need it, either auto-filing bug reports, email, whatever.

Legoktm created this task.Mar 20 2015, 7:47 PM
Legoktm updated the task description. (Show Details)
Legoktm raised the priority of this task from to Needs Triage.
Legoktm added subscribers: Legoktm, hashar, Nemo_bis.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2015, 7:47 PM
Krinkle added a subscriber: Krinkle.EditedMar 21 2015, 6:41 AM

Extensions should encounter these naturally if they're being maintained. As a last resort, the commits creating the release branches can be used as a last defense to ensure extensions keep up at least once per release cycle.

Also note that many extensions deployed at Wikimedia are part of the combo-group mediawiki-extensions-hhvm, which is triggered by MediaWiki core after every commit.

Krinkle triaged this task as Low priority.Mar 21 2015, 6:41 AM
Krinkle set Security to None.
greg added a subscriber: greg.

This may be addressed in the future by the very preliminary project code-named quibble.

Krinkle removed a subscriber: Krinkle.May 25 2017, 2:36 PM

Change 376739 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] zuul: allow email connection

https://gerrit.wikimedia.org/r/376739

Change 376740 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] zuul: periodic pipeline

https://gerrit.wikimedia.org/r/376740

hashar added a subscriber: Reedy.Sep 8 2017, 8:31 PM

I am pretty sure @Reedy also asked to run the PHPUnit tests for all extensions/skins.

Legoktm updated the task description. (Show Details)

Change 376739 merged by Dzahn:
[operations/puppet@production] zuul: allow email connection

https://gerrit.wikimedia.org/r/376739

From Gerrit #376739, puppet restarted the zuul-merger. Gotta restart Zuul scheduler to have the mail/smtp configuration to be taken in account.

Then it is all about adding the smtp reporter in zuul/layout.yaml and we can start getting emails for timed jobs.

Mentioned in SAL (#wikimedia-operations) [2019-01-09T07:43:52Z] <hashar> contint1001: restarted Zuul to take in account SMTP configuration | https://gerrit.wikimedia.org/r/376739 | T93414

hashar added a comment.Wed, Jan 9, 7:47 AM

The Zuul scheduler now has a SMTP connection which defaults to send emails to qa-alerts@lists.wikimedia.org. We can thus have a pipeline that emits email, with something like:

zuul/layout.yaml
- name: daily
  failure:
    smtp
  success:
    smtp:
       from: jenkins@example.org
       to: someone@example.org
       subject: Change {change} failed