Page MenuHomePhabricator

Add editquality isvc configurations to ml-services helmfile
Closed, ResolvedPublic

Description

Once we have published our updated "predictor-only" editquality model-server image to the WMF Docker Registry and have uploaded all editquality model files to storage, we can add the Inference Service configurations to the ml-services helmfile.

We can follow the Deployment guide that @elukey put together to do this:
https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Deploy#Add_a_model_to_an_existing_helmfile_config

We mostly need to update the editquality image version to the new predictor-only image and then add an entry under inference_services for each model we want to deploy.

Details

SubjectRepoBranchLines +/-
operations/deployment-chartsmaster+97 -194
operations/deployment-chartsmaster+0 -106
operations/deployment-chartsmaster+4 -4
operations/deployment-chartsmaster+8 -0
operations/deployment-chartsmaster+7 -1
operations/deployment-chartsmaster+9 -1
operations/deployment-chartsmaster+12 -0
operations/deployment-chartsmaster+12 -0
operations/deployment-chartsmaster+12 -0
operations/deployment-chartsmaster+12 -0
operations/deployment-chartsmaster+12 -0
operations/deployment-chartsmaster+11 -1
operations/deployment-chartsmaster+9 -43
operations/puppetproduction+0 -6
labs/privatemaster+0 -9
operations/deployment-chartsmaster+14 -27
operations/deployment-chartsmaster+396 -0
operations/deployment-chartsmaster+9 -0
operations/puppetproduction+18 -0
labs/privatemaster+27 -0
operations/deployment-chartsmaster+30 -0
operations/deployment-chartsmaster+24 -0
operations/deployment-chartsmaster+24 -0
operations/deployment-chartsmaster+26 -0
operations/deployment-chartsmaster+27 -0
operations/deployment-chartsmaster+25 -1
operations/deployment-chartsmaster+19 -1
operations/deployment-chartsmaster+24 -0
operations/deployment-chartsmaster+1 -1
Show related patches Customize query in gerrit

Event Timeline

ACraze changed the task status from Open to In Progress.Feb 14 2022, 8:15 PM

Change 762532 had a related patch set uploaded (by Accraze; author: Accraze):

[operations/deployment-charts@master] ml-services: update editquality predictor image

https://gerrit.wikimedia.org/r/762532

Change 762533 had a related patch set uploaded (by Accraze; author: Accraze):

[operations/deployment-charts@master] ml-services: add arwiki & bnwiki editquality isvcs

https://gerrit.wikimedia.org/r/762533

Change 762532 merged by Elukey:

[operations/deployment-charts@master] ml-services: update editquality predictor image

https://gerrit.wikimedia.org/r/762532

Change 762777 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add bswiki & cawiki editquality isvcs

https://gerrit.wikimedia.org/r/762777

Change 762777 merged by Elukey:

[operations/deployment-charts@master] ml-services: add bswiki & cawiki editquality isvcs

https://gerrit.wikimedia.org/r/762777

Change 762533 merged by Elukey:

[operations/deployment-charts@master] ml-services: add arwiki & bnwiki editquality isvcs

https://gerrit.wikimedia.org/r/762533

Change 763480 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add cswiki & dewiki editquality isvcs

https://gerrit.wikimedia.org/r/763480

Change 763480 merged by Elukey:

[operations/deployment-charts@master] ml-services: add cswiki & dewiki editquality isvcs

https://gerrit.wikimedia.org/r/763480

Change 763556 had a related patch set uploaded (by Accraze; author: Accraze):

[operations/deployment-charts@master] ml-services: add elwiki, enwiktionary, eswikibooks

https://gerrit.wikimedia.org/r/763556

Change 763556 merged by Elukey:

[operations/deployment-charts@master] ml-services: add elwiki, enwiktionary, eswikibooks

https://gerrit.wikimedia.org/r/763556

Change 763647 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add eswiki & eswikiquote editquality isvcs

https://gerrit.wikimedia.org/r/763647

Change 763647 merged by Elukey:

[operations/deployment-charts@master] ml-services: add eswiki & eswikiquote editquality isvcs

https://gerrit.wikimedia.org/r/763647

Change 763773 had a related patch set uploaded (by Accraze; author: Accraze):

[operations/deployment-charts@master] ml-services: add etwiki and fawiki editquality

https://gerrit.wikimedia.org/r/763773

Change 763773 merged by Elukey:

[operations/deployment-charts@master] ml-services: add etwiki and fawiki editquality

https://gerrit.wikimedia.org/r/763773

Change 764295 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add fiwiki & frwiki editquality isvcs

https://gerrit.wikimedia.org/r/764295

Change 764295 merged by Elukey:

[operations/deployment-charts@master] ml-services: add fiwiki & frwiki editquality isvcs

https://gerrit.wikimedia.org/r/764295

Change 764623 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add glwiki, hewiki & hiwiki editquality isvcs

https://gerrit.wikimedia.org/r/764623

Change 764623 merged by Elukey:

[operations/deployment-charts@master] ml-services: add glwiki, hewiki & hiwiki editquality isvcs

https://gerrit.wikimedia.org/r/764623

After talking with @elukey and @kevinbazira on IRC, it seems that we may need to split up the revscoring-editquality namespace as some nodes are filling up with pods.

@elukey had the idea to split the namespace into something revscoring-editquality-damaging and revscoring-editquality-goodfaith, which I think is a good way to go for now.

Also there are the reverted editquality models (~10 of them) should we do a separate namespace for those as well?

Change 765193 had a related patch set uploaded (by Elukey; author: Elukey):

[labs/private@master] Split the revscoring-editquality ml-serve settings in three

https://gerrit.wikimedia.org/r/765193

Change 765193 merged by Elukey:

[labs/private@master] Split the revscoring-editquality ml-serve settings in three

https://gerrit.wikimedia.org/r/765193

Change 765196 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kubernetes::deployment_server: split revscoring-ediquality

https://gerrit.wikimedia.org/r/765196

Change 765196 merged by Elukey:

[operations/puppet@production] profile::kubernetes::deployment_server: split revscoring-ediquality

https://gerrit.wikimedia.org/r/765196

Change 765198 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: add new namespaces for revscoring-editquality

https://gerrit.wikimedia.org/r/765198

Change 765199 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: add helmfile config for the new revscoring-editquality ns

https://gerrit.wikimedia.org/r/765199

Change 765198 merged by Elukey:

[operations/deployment-charts@master] admin_ng: add new namespaces for revscoring-editquality

https://gerrit.wikimedia.org/r/765198

Change 765199 merged by Elukey:

[operations/deployment-charts@master] ml-services: add helmfile config for the new revscoring-editquality ns

https://gerrit.wikimedia.org/r/765199

Change 765228 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: move reverted models to their new namespace

https://gerrit.wikimedia.org/r/765228

Change 765228 merged by Elukey:

[operations/deployment-charts@master] ml-services: move reverted models to their new namespace

https://gerrit.wikimedia.org/r/765228

Change 765235 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: move goodfaith/damaging models to the new ns

https://gerrit.wikimedia.org/r/765235

Change 765235 merged by Elukey:

[operations/deployment-charts@master] ml-services: move goodfaith/damaging models to the new ns

https://gerrit.wikimedia.org/r/765235

Change 765242 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] ml-services: deprecate the revscoring-editquality ns

https://gerrit.wikimedia.org/r/765242

Change 765243 had a related patch set uploaded (by Elukey; author: Elukey):

[labs/private@master] Remove references to revscoring-editquality

https://gerrit.wikimedia.org/r/765243

Change 765244 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Remove references of revscoring-editquality for ml-serve

https://gerrit.wikimedia.org/r/765244

Change 765242 merged by Elukey:

[operations/deployment-charts@master] ml-services: deprecate the revscoring-editquality ns

https://gerrit.wikimedia.org/r/765242

Change 765243 merged by Elukey:

[labs/private@master] Remove references to revscoring-editquality

https://gerrit.wikimedia.org/r/765243

Change 765244 merged by Elukey:

[operations/puppet@production] Remove references of revscoring-editquality for ml-serve

https://gerrit.wikimedia.org/r/765244

After talking with @elukey and @kevinbazira on IRC, it seems that we may need to split up the revscoring-editquality namespace as some nodes are filling up with pods.

@elukey had the idea to split the namespace into something revscoring-editquality-damaging and revscoring-editquality-goodfaith, which I think is a good way to go for now.

Also there are the reverted editquality models (~10 of them) should we do a separate namespace for those as well?

Refactoring done, we now have three namespaces (and related pods): revscoring-editquality-{reverted,goodfaith,damaging}

Change 765254 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add hrwiki, huwiki, idwiki & iswiki editquality isvcs

https://gerrit.wikimedia.org/r/765254

Change 765260 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] kserve-inference: dry model config for revscoring_inference_services

https://gerrit.wikimedia.org/r/765260

Change 765260 merged by Elukey:

[operations/deployment-charts@master] kserve-inference: dry model config for revscoring_inference_services

https://gerrit.wikimedia.org/r/765260

Change 765254 merged by Elukey:

[operations/deployment-charts@master] ml-services: add hrwiki, huwiki, idwiki & iswiki editquality isvcs

https://gerrit.wikimedia.org/r/765254

Change 765501 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add itwiki, jawiki & kowiki editquality isvcs

https://gerrit.wikimedia.org/r/765501

Change 765501 merged by Elukey:

[operations/deployment-charts@master] ml-services: add itwiki, jawiki & kowiki editquality isvcs

https://gerrit.wikimedia.org/r/765501

Change 766565 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add lvwiki, nlwiki & nowiki editquality isvcs

https://gerrit.wikimedia.org/r/766565

Change 766566 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add lvwiki, nlwiki & nowiki editquality isvcs

https://gerrit.wikimedia.org/r/766566

Change 766566 abandoned by Kevin Bazira:

[operations/deployment-charts@master] ml-services: add lvwiki, nlwiki & nowiki editquality isvcs

Reason:

duplicate to https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/766565/

https://gerrit.wikimedia.org/r/766566

This task has been paused because if we continue loading models we'll saturate svc ips.

We'll proceed soon as #T302701 has been figured out.

Change 766565 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add lvwiki, nlwiki & nowiki editquality isvcs

https://gerrit.wikimedia.org/r/766565

Change 778251 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add plwiki, ptwiki & rowiki editquality isvcs

https://gerrit.wikimedia.org/r/778251

Change 778251 merged by Klausman:

[operations/deployment-charts@master] ml-services: add plwiki, ptwiki & rowiki editquality isvcs

https://gerrit.wikimedia.org/r/778251

Change 779438 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add ruwiki, sqwiki & srwiki editquality isvcs

https://gerrit.wikimedia.org/r/779438

Change 779438 merged by Elukey:

[operations/deployment-charts@master] ml-services: add ruwiki, sqwiki & srwiki editquality isvcs

https://gerrit.wikimedia.org/r/779438

Change 784620 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add svwiki, tawiki & translatewiki editquality isvcs

https://gerrit.wikimedia.org/r/784620

Change 784620 merged by Elukey:

[operations/deployment-charts@master] ml-services: add svwiki, tawiki & translatewiki editquality isvcs

https://gerrit.wikimedia.org/r/784620

Change 786276 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add ukwiki & viwiki editquality isvcs

https://gerrit.wikimedia.org/r/786276

Change 786276 merged by Elukey:

[operations/deployment-charts@master] ml-services: add ukwiki & viwiki editquality isvcs

https://gerrit.wikimedia.org/r/786276

Change 786924 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add wikidatawiki & zhwiki editquality isvcs

https://gerrit.wikimedia.org/r/786924

Change 786924 merged by Klausman:

[operations/deployment-charts@master] ml-services: add wikidatawiki & zhwiki editquality isvcs

https://gerrit.wikimedia.org/r/786924

Change 786982 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: fix wikidatawiki & zhwiki editquality isvcs

https://gerrit.wikimedia.org/r/786982

Change 786982 merged by Klausman:

[operations/deployment-charts@master] ml-services: fix wikidatawiki & zhwiki editquality isvcs

https://gerrit.wikimedia.org/r/786982

Inference services were created for all the 76 editquality models and they are all up and running in KServe on both eqiad and codfw.