Page MenuHomePhabricator

CI service-pipeline-test-and-publish job assumes blubber config has a single production image
Open, NormalPublic

Description

The CI job for publishing blindly assumes a single image called production.

For the Kartotherian project, which has 2 services, this is not true.

CI publication thus fails, so changes merge but no images are published.

Event Timeline

MSantos created this task.Sep 3 2019, 6:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 3 2019, 6:20 PM
Jdforrester-WMF renamed this task from CI assumes blubber config to have a single production image to CI service-pipeline-test-and-publish job assumes blubber config has a single production image.Sep 3 2019, 6:52 PM
Jdforrester-WMF updated the task description. (Show Details)

Not sure but seems we are missing some configs in our config.yaml patch

Change 534419 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[mediawiki/services/kartotherian@master] Define config pipeline for kartotherian and tilerator

https://gerrit.wikimedia.org/r/534419

thcipriani triaged this task as Normal priority.Sep 18 2019, 2:56 PM
thcipriani moved this task from INBOX to Watching/External on the Release-Engineering-Team-TODO board.
thcipriani added subscribers: dduvall, thcipriani.

Not sure but seems we are missing some configs in our config.yaml patch

To use config.yaml there are a few tweaks needed in integration/config (I should update the docs on wikitech). Also, pinging @dduvall to tell me if any of what I've written here is totally wrong :)


Currently, kartotherian is setup to use the default pipeline trigger job: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/zuul/layout.yaml#7215

That job runs through the variants test and production and ignores other variants.

To use custom pipelines you need to:

  1. define the jobs in jjb/project-pipelines.yaml
  2. trigger jobs on postmerge of mediawiki/services/kartotherian

Define jobs

It looks like you have 2 pipelines that you want to use in place of the test-and-publish job that happens on postmerge and publish. To achieve that, you'd update jjb/project-pipelines.yaml to create a job for each pipeline and a trigger for each job:

jjb/project-pipelines.yaml
- project:
    name: kartotherian
    pipeline:
      - kartotherian
      - tilerator
    jobs:
      - 'trigger-{name}-pipeline-{pipeline}'
      - '{name}-pipeline-{pipeline}'

This will create the jobs to trigger and run the pipelines in jenkins.

Run jobs on postmerge

Now that you have jobs to run, you can trigger them on postmerge of your repository by updating the record for mediawiki/services/kartotherian:

zuul/layout.yaml
- name: mediawiki/services/kartotherian
  template:
    - name: service-pipeline-test
  test:
    - kartotherian-node10-browser-docker
  gate-and-submit:
    - kartotherian-node10-browser-docker
  postmerge:
    - trigger-kartotherian-pipeline-kartotherian
    - trigger-kartotherian-pipeline-tilerator
  publish:
    - trigger-kartotherian-pipeline-kartotherian
    - trigger-kartotherian-pipeline-tilerator

Change 537689 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Define pipeline jobs for kartotherian and tilerator

https://gerrit.wikimedia.org/r/537689

Change 537690 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: Use trigger-kartotherian-pipeline-kartotherian and trigger-kartotherian-pipeline-tilerator

https://gerrit.wikimedia.org/r/537690

Change 537689 merged by jenkins-bot:
[integration/config@master] jjb: Define pipeline jobs for kartotherian and tilerator

https://gerrit.wikimedia.org/r/537689

Change 537690 merged by jenkins-bot:
[integration/config@master] layout: Use trigger-kartotherian-pipeline-kartotherian and trigger-kartotherian-pipeline-tilerator

https://gerrit.wikimedia.org/r/537690

Mentioned in SAL (#wikimedia-releng) [2019-09-18T17:19:41Z] <James_F> Zuul: T231928 Migrate kartotherian to custom pipeline jobs

Change 534419 merged by jenkins-bot:
[mediawiki/services/kartotherian@master] Define config pipeline for kartotherian and tilerator

https://gerrit.wikimedia.org/r/534419

Mentioned in SAL (#wikimedia-releng) [2019-09-18T18:47:45Z] <James_F> T231928 terminating endlessly-running attempts to build kartotherian and tilerator