Maniphest T231928

CI service-pipeline-test-and-publish job assumes blubber config has a single production image
Open, MediumPublic
Actions

Assigned To

None

Authored By

	MSantos
	Sep 3 2019, 6:20 PM

Description

The CI job for publishing blindly assumes a single image called production.

For the Kartotherian project, which has 2 services, this is not true.

CI publication thus fails, so changes merge but no images are published.

Details

Subject	Repo	Branch	Lines +/-
Define config pipeline for kartotherian and tilerator	mediawiki/services/kartotherian	master	+20 -0
layout: Use trigger-kartotherian-pipeline-kartotherian and trigger-kartotherian-pipeline-tilerator	integration/config	master	+6 -1
jjb: Define pipeline jobs for kartotherian and tilerator	integration/config	master	+25 -6

Customize query in gerrit

Event Timeline

MSantos created this task.Sep 3 2019, 6:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 3 2019, 6:20 PM

Jdforrester-WMF renamed this task from CI assumes blubber config to have a single production image to CI service-pipeline-test-and-publish job assumes blubber config has a single production image.Sep 3 2019, 6:52 PM

Jdforrester-WMF added projects: Release-Engineering-Team-TODO, Release-Engineering-Team (Pipeline).

Jdforrester-WMF updated the task description. (Show Details)

MSantos moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.Sep 3 2019, 7:50 PM

Not sure but seems we are missing some configs in our config.yaml patch

Change 534419 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[mediawiki/services/kartotherian@master] Define config pipeline for kartotherian and tilerator

https://gerrit.wikimedia.org/r/534419

gerritbot added a project: Patch-For-Review.Sep 4 2019, 12:10 PM

In T231928#5464522, @Mathew.onipe wrote:

Not sure but seems we are missing some configs in our config.yaml patch

To use config.yaml there are a few tweaks needed in integration/config (I should update the docs on wikitech). Also, pinging @dduvall to tell me if any of what I've written here is totally wrong :)

Currently, kartotherian is setup to use the default pipeline trigger job: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/master/zuul/layout.yaml#7215

That job runs through the variants test and production and ignores other variants.

To use custom pipelines you need to:

define the jobs in jjb/project-pipelines.yaml
trigger jobs on postmerge of mediawiki/services/kartotherian

Define jobs

It looks like you have 2 pipelines that you want to use in place of the test-and-publish job that happens on postmerge and publish. To achieve that, you'd update jjb/project-pipelines.yaml to create a job for each pipeline and a trigger for each job:

jjb/project-pipelines.yaml

- project:
    name: kartotherian
    pipeline:
      - kartotherian
      - tilerator
    jobs:
      - 'trigger-{name}-pipeline-{pipeline}'
      - '{name}-pipeline-{pipeline}'

This will create the jobs to trigger and run the pipelines in jenkins.

Run jobs on postmerge

Now that you have jobs to run, you can trigger them on postmerge of your repository by updating the record for mediawiki/services/kartotherian:

zuul/layout.yaml

- name: mediawiki/services/kartotherian
  template:
    - name: service-pipeline-test
  test:
    - kartotherian-node10-browser-docker
  gate-and-submit:
    - kartotherian-node10-browser-docker
  postmerge:
    - trigger-kartotherian-pipeline-kartotherian
    - trigger-kartotherian-pipeline-tilerator
  publish:
    - trigger-kartotherian-pipeline-kartotherian
    - trigger-kartotherian-pipeline-tilerator

Change 537689 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] jjb: Define pipeline jobs for kartotherian and tilerator

https://gerrit.wikimedia.org/r/537689

Change 537690 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: Use trigger-kartotherian-pipeline-kartotherian and trigger-kartotherian-pipeline-tilerator

https://gerrit.wikimedia.org/r/537690

Change 537689 merged by jenkins-bot:
[integration/config@master] jjb: Define pipeline jobs for kartotherian and tilerator

https://gerrit.wikimedia.org/r/537689

Change 537690 merged by jenkins-bot:
[integration/config@master] layout: Use trigger-kartotherian-pipeline-kartotherian and trigger-kartotherian-pipeline-tilerator

https://gerrit.wikimedia.org/r/537690

Mentioned in SAL (#wikimedia-releng) [2019-09-18T17:19:41Z] <James_F> Zuul: T231928 Migrate kartotherian to custom pipeline jobs

Change 534419 merged by jenkins-bot:
[mediawiki/services/kartotherian@master] Define config pipeline for kartotherian and tilerator

https://gerrit.wikimedia.org/r/534419

Maintenance_bot removed a project: Patch-For-Review.Sep 18 2019, 6:10 PM

Mentioned in SAL (#wikimedia-releng) [2019-09-18T18:47:45Z] <James_F> T231928 terminating endlessly-running attempts to build kartotherian and tilerator

MSantos moved this task from Kartotherian to Tracking on the Maps board.Nov 25 2020, 4:08 PM

MSantos edited projects, added Maps; removed Maps (Kartotherian).

thcipriani removed a project: Release-Engineering-Team (Pipeline).Apr 20 2021, 1:25 AM

thcipriani edited projects, added Release-Engineering-Team (Radar); removed Release-Engineering-Team-TODO.Apr 20 2021, 3:33 AM

thcipriani moved this task from Limbo to Watching/External on the Release-Engineering-Team (Radar) board.Apr 20 2021, 3:34 AM

CI service-pipeline-test-and-publish job assumes blubber config has a single production imageOpen, MediumPublicActions