Fix pipeline image publishing workflow
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ACraze
	Dec 15 2021, 7:33 PM

Description

Currently our model deployment pipelines are composed of two stages:

run-test
run-production

The tests are simple formatting/syntax checks and the production stage builds the production image and if successful, pushes it to the WMF docker registry. One issue with this is that for each patchset in a CR, the image is pushed to the registry, which is confusing and wastes space/resources because after the CR is merged, the image gets built and published again.

We need to split the image publishing section to it's own stage that only runs during the gate-and-submit jobs.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	pipeline: add separate publish pipelines	machinelearning/liftwing/inference-services	main	+52 -0
	inference: add publishing pipelines for postmerge	integration/config	master	+80 -42

Customize query in gerrit

Related Objects

Mentioned In: rMLISc3af6dbe9e22: pipeline: add separate publish pipelines

Event Timeline

• ACraze created this task.Dec 15 2021, 7:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 15 2021, 7:33 PM

Looking at the PipelineLib and Deployment pipeline docs, it seems we will need to define an additional stage (i.e. publish) that pushes the production image to the registry during the gate-and-submit jobs. Right now, we are including this inside of the production stage which is not great.

https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#Publishing_Docker_Images

• ACraze updated the task description. (Show Details)Dec 16 2021, 10:14 PM

Ok, it seems we may need a separate "publish" pipeline defined in .pipeline/config.yaml (inference-services repo) for each image that will get called during postmerge:

editquality-publish:
  blubberfile: editquality/blubber.yaml
  stages:
    - name: production
      build: production
      publish:
        image:
          tags: [stable]

In the integrations/config repo, we add the new pipeline in jjb/project-pipelines.yaml and then in zuul/layouts.yaml we add the new job to the postmerge (under gate-and-submit) like this:

...
    postmerge:
      - trigger-inference-services-pipeline-editquality-publish

I will try to get a CR up on gerrit later today with a publish pipeline for each image. It might be best to include all of the new publish pipelines in one CR since all images get rebuilt when we edit the config.yaml. After that we can work on the integrations/config CRs.

Change 751777 had a related patch set uploaded (by Accraze; author: Accraze):

[machinelearning/liftwing/inference-services@main] pipeline: add separate publish pipelines

https://gerrit.wikimedia.org/r/751777

gerritbot added a project: Patch-For-Review.Jan 5 2022, 5:34 PM

Change 751777 merged by Accraze:

[machinelearning/liftwing/inference-services@main] pipeline: add separate publish pipelines

https://gerrit.wikimedia.org/r/751777

• ACraze mentioned this in rMLISc3af6dbe9e22: pipeline: add separate publish pipelines.Jan 7 2022, 5:00 PM

Maintenance_bot removed a project: Patch-For-Review.Jan 7 2022, 5:10 PM

Change 752714 had a related patch set uploaded (by Accraze; author: Accraze):

[integration/config@master] inference: add publishing pipelines for postmerge

https://gerrit.wikimedia.org/r/752714