Create a CI task for MCS periodic tests
Closed, ResolvedPublic

Description

Create a CI task for MCS live system tests which runs on a predetermined schedule (at least workdaily) and/or on demand when a Parsoid deploy is done against the main WMF projects.
Bonus points if this could be also triggered automatically after a Parsoid deployment.

It would be great if test runs are logged on IRC, similarly to what is done for Android CI (channel \#wikimedia-android-ci).

Examples for tests that should ran against live data:

  • a modified version of featured-image-large (-/+ 30 days of current day)
  • diff-test (we could have another version of it run against sepia fixtures as part of the medium tests)
  • news test but against live data instead of fixture. The current news test could probably be rewritten to use the sepia fixtures as well.
  • tests running against Beta Cluster (if we still want to keep them) but they should be easily identifiable in the test report

Once set this up we might be able to move more system tests under test/features there.

Current status

A mobileapps-periodic-test job has been created. It runs hourly, immediately failing each time. It needs an environment with npm and currently doesn't have one. Test results should be sent to #wikimedia-ri-ci but aren't.

The config is in integration-config/jjb/mediawiki-services.yaml:

- job-template:
    name: 'mobileapps-periodic-test'
    node: DebianJessieDocker
    concurrent: true
    triggers:
     - timed: '@hourly'
    scm:
     - git:
        url: https://phabricator.wikimedia.org/diffusion/GMOA
        branches:
          - master
    builders:
     - shell: npm run -s periodic
    publishers:
     - irc-wikimedia-ri-ci

- publisher:
    name: irc-wikimedia-ri-ci
    publishers:
     - ircbot:
         message-type: summary-scm-fail
         channels:
             - name: '#wikimedia-ri-ci'
               notify-only: true

Next steps

  • Get the job running in the correct environment
  • Get the job passing
  • Fix the irc-wikimedia-ri-ci publisher definition so results are sent to #wikimedia-ri-ci
bearND created this task.Oct 10 2017, 11:14 PM
bearND renamed this task from Create a CI task for MCS live integration tests to Create a CI task for MCS large tests.Jan 17 2018, 10:11 PM
bearND updated the task description. (Show Details)
Mholloway added a subscriber: Mholloway.

I am glad to see this idea make it to Phabricator!

It would be great if test runs are logged on IRC, similarly to what is done for Android CI (channel \#wikimedia-android-ci).

Where should we log them? Is it time to create a #wikimedia-ri channel?

  • tests running against Beta Cluster (if we still want to keep them) but they should be easily identifiable in the test report

Isn't there only one of these, for mobile-sections?

bearND added a comment.EditedJan 18 2018, 12:58 AM

I was thinking an #wikimedia-ri-ci channel, similar to #wikimedia-android-ci. Basically, a channel just for CI. But if you think this wouldn't be too much noise then I guess we could do it in #wikimedia-ri.

Yes, there is currently only one beta cluster test left, and we skip it since yesterday. I'm just saying IF we need to keep Beta Cluster tests they should go to the large tests, too. Ok, I guess that was implied. Just being explicit.

bearND renamed this task from Create a CI task for MCS large tests to Create a CI task for MCS periodic tests.Mar 13 2018, 10:58 PM
bearND moved this task from Backlog to Kanban on the Reading-Infrastructure-Team-Backlog board.
bearND moved this task from To Do to Doing on the Reading-Infrastructure-Team-Backlog (Kanban) board.
bearND triaged this task as Normal priority.
bearND assigned this task to Mholloway.

Change 419326 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[mediawiki/services/mobileapps@master] Add 'periodic' script for periodic testing

https://gerrit.wikimedia.org/r/419326

Change 415886 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[integration/config@master] Add mobileapps-diff-test periodic job

https://gerrit.wikimedia.org/r/415886

Change 419326 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Add 'periodic' script for periodic testing

https://gerrit.wikimedia.org/r/419326

bearND added a subscriber: hashar.Apr 17 2018, 7:09 PM
Mholloway raised the priority of this task from Normal to High.Apr 19 2018, 7:37 PM

The instability of these tests is a large and ongoing burden.

It occurs to me that this could run on a separate Cloud VPS instance and not necessarily as part of the CI infrastructure.

Change 415886 merged by jenkins-bot:
[integration/config@master] Add mobileapps-diff-test periodic job

https://gerrit.wikimedia.org/r/415886

Change 432310 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[integration/config@master] Run mobileapps-periodic-test hourly regardless of repo changes

https://gerrit.wikimedia.org/r/432310

Change 432310 merged by jenkins-bot:
[integration/config@master] Run mobileapps-periodic-test hourly regardless of repo changes

https://gerrit.wikimedia.org/r/432310

Mentioned in SAL (#wikimedia-releng) [2018-05-16T17:51:53Z] <twentyafterfour> deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs T177896

Probably need to set the correct Docker image somewhere, see https://integration.wikimedia.org/ci/job/mobileapps-periodic-test/167/console:

npm: command not found

Strangely enough there are not notifications on IRC even though we get this in the console log:

IRC notifier plugin: Sending notification to: #wikimedia-ri-ci

@Jdforrester-WMF If you know your way around the CI infrastructure, assistance here would be most welcome.

Mholloway updated the task description. (Show Details)Jun 26 2018, 2:47 PM
Mholloway removed a project: Patch-For-Review.
Mholloway removed Mholloway as the assignee of this task.
Mholloway updated the task description. (Show Details)Jun 26 2018, 2:50 PM

Change 442126 had a related patch set uploaded (by Mholloway; owner: Mholloway):
[integration/config@master] Run mobileapps-periodic-test in npm-test docker image

https://gerrit.wikimedia.org/r/442126

Mholloway updated the task description. (Show Details)Jun 26 2018, 3:11 PM

Change 442126 merged by jenkins-bot:
[integration/config@master] Run mobileapps-periodic-test in npm-test docker image

https://gerrit.wikimedia.org/r/442126

Change 446372 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] mobileapps-periodic-test: add ZUUL_BRANCH

https://gerrit.wikimedia.org/r/446372

Change 446372 merged by jenkins-bot:
[integration/config@master] Make mobileapps-periodic-test work

https://gerrit.wikimedia.org/r/446372

bearND awarded a token.
bearND closed this task as Resolved.

Change 446615 had a related patch set uploaded (by BearND; owner: BearND):
[integration/config@master] Only report on IRC when periodic test failed or fixed

https://gerrit.wikimedia.org/r/446615

Change 446615 merged by jenkins-bot:
[integration/config@master] Only report on IRC when periodic test failed or fixed

https://gerrit.wikimedia.org/r/446615