Page MenuHomePhabricator

Configure revscoring topic deployment pipeline
Closed, ResolvedPublic

Description

We want to deploy our production image to the WMF docker registry by using the deployment pipeline. We should also run a test before publishing the image.

This will require us to update multiple repos:

  • Configure the pipeline in the inference-services repo: ./pipeline/config.yaml
  • Update the integrations/config repo:
    • create pipelines & define jobs: jjb/project-pipelines.yaml file
    • assign job triggers: zuul/layouts.yaml
      • also make sure the pipeline only runs when the config, blubberfile or model-server is changed.

This has already been done for editquality (see: T279004) which can be used as an example.

Pipeline tutorial: https://wikitech.wikimedia.org/wiki/Deployment_pipeline/Migration/Tutorial#Publishing_Docker_Images

Event Timeline

Change 713279 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] add topic pipeline config

https://gerrit.wikimedia.org/r/713279

Change 713279 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] add topic pipeline config

https://gerrit.wikimedia.org/r/713279

Change 713439 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: add topic pipeline

https://gerrit.wikimedia.org/r/713439

Change 713439 merged by jenkins-bot:

[integration/config@master] inference-services: add topic pipeline

https://gerrit.wikimedia.org/r/713439

It looks like the pipeline was created, however there are some errors when creating the production image:
https://integration.wikimedia.org/ci/job/inference-services-pipeline-topic/

/usr/bin/python3: Error while finding module specification for 'nltk.downloader' (ModuleNotFoundError: No module named 'nltk')
model-server/topic-prerequisites.sh: 6: model-server/topic-prerequisites.sh: wget: not found
grep: Makefile: No such file or directory
make: *** No targets specified and no makefile found.  Stop.

Change 714699 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] fix topic model-server production image

https://gerrit.wikimedia.org/r/714699

Change 714699 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] fix topic model-server production image

https://gerrit.wikimedia.org/r/714699

Nice one @kevinbazira! Looks like the pipeline is working well now:
https://integration.wikimedia.org/ci/job/inference-services-pipeline-topic/

Also the topic model-server production image is now available in the WMF docker registry:
https://docker-registry.wikimedia.org/wikimedia/machinelearning-liftwing-inference-services-topic/tags/

Marking this as RESOLVED