Configure draftquality deployment pipeline
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ACraze
	Jul 30 2021, 6:32 PM

Project Tags

Referenced Files

None

Subscribers

• ACraze

Aklapper

kevinbazira

Description

We want to deploy our production image to the WMF docker registry by using the deployment pipeline. We should also run a test before publishing the image.

This will require us to update multiple repos:

Configure the pipeline in the inference-services repo: ./pipeline/config.yaml
Update the integrations/config repo:
- create pipelines & define jobs: jjb/project-pipelines.yaml file
- assign job triggers: zuul/layouts.yaml
  - also make sure the pipeline only runs when the config, blubberfile or model-server is changed.

This has already been done for editquality (see: T279004) which can be used as an example.

Pipeline tutorial: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#Publishing_Docker_Images

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
add explicit versions for draftquality deps	machinelearning/liftwing/inference-services	main	+84 -3
inference-services: added draftquality pipeline	integration/config	master	+12 -2
add draftquality pipeline config	machinelearning/liftwing/inference-services	main	+12 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	kevinbazira	T301409 Migrate editquality models
Resolved	• ACraze	T284689 Create migration plan for editquality models from ORES to Lift Wing
Resolved	calbon	T272874 Prepare 4 ORES English models for Lift Wing
Resolved	kevinbazira	T279004 Production images for ORES/revscoring models
Resolved	• ACraze	T287787 Configure draftquality deployment pipeline
Resolved	• ACraze	T287783 Create blubberfile for draftquality model server

Event Timeline

• ACraze created this task.Jul 30 2021, 6:32 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 30 2021, 6:32 PM

• ACraze added a parent task: T279004: Production images for ORES/revscoring models.Jul 30 2021, 6:35 PM

• ACraze added a subtask: T287783: Create blubberfile for draftquality model server.

Maintenance_bot added a project: artificial-intelligence.Jul 30 2021, 6:45 PM

• ACraze claimed this task.Aug 10 2021, 11:01 PM

Change 711690 had a related patch set uploaded (by Accraze; author: Accraze):

[machinelearning/liftwing/inference-services@main] add draftquality pipeline config

https://gerrit.wikimedia.org/r/711690

gerritbot added a project: Patch-For-Review.Aug 11 2021, 8:53 PM

• ACraze closed subtask T287783: Create blubberfile for draftquality model server as Resolved.Aug 11 2021, 9:18 PM

Change 711731 had a related patch set uploaded (by Accraze; author: Accraze):

[integration/config@master] inference-services: added draftquality pipeline

https://gerrit.wikimedia.org/r/711731

Change 711690 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] add draftquality pipeline config

https://gerrit.wikimedia.org/r/711690

• ACraze mentioned this in rMLIS2b705b62f48f: add draftquality pipeline config.Aug 12 2021, 4:43 AM

Change 711731 merged by jenkins-bot:

[integration/config@master] inference-services: added draftquality pipeline

https://gerrit.wikimedia.org/r/711731

Maintenance_bot removed a project: Patch-For-Review.Aug 16 2021, 11:10 PM

It seems our production image build for the draftquality model server is unable to build. It ran all night and could not resolve all the dependencies between the draftquality repo and revscoring package, getting the error: ResolutionTooDeep. The image seems to build fine for me locally though, so I will continue investigating this issue.

https://integration.wikimedia.org/ci/job/inference-services-pipeline-draftquality/

After digging into the issue a bit more, it seems that pip is unable to resolve the correct version of the tqdm library. The version ranges defined for revscoring, nltk and kfserving seem to conflict using our requirements.txt included with the draftquality model-server production image.

After checking the ores-deploy repo, it seems we should be able to pin tqdm==4.15.0, nltk==3.5 and revscoring==2.8.2, however we will need to figure out which version of kfserving will run with these deps.