Page MenuHomePhabricator

Reorganize LiftWing isvcs repo structure to improve maintainability
Closed, ResolvedPublic5 Estimated Story Points

Description

To address the growing complexity of our LiftWing isvcs project directory, @isarantopoulos proposed reorganizing the directory structure to improve navigation and maintainability. The new structure introduces a src/ folder to contain model-servers for all models, thereby decluttering the top-level directory. Specifically:

Current Structure
.
├── README.md
├── requirements-test.txt
├── article_descriptions
│   .
│   .
│   .
├── revert_risk_model
├── revscoring_model
├── test
└── tox.ini
Proposed Structure
.
├── README.md
├── requirements-test.txt
├── src
│   └── models
│       ├── article_descriptions
│       .
│       .
│       .
│       └── revert_risk_model
├── revscoring_model
├── test
└── tox.ini

Note: The ores-legacy/ and revscoring_model/ directories will remain at the top level due to their impending deprecation, avoiding unnecessary migration effort.

This reorganization will result in a cleaner and more intuitive project layout, facilitating easier navigation, maintainability, and future scalability.

Migration Progress

isvcCImodel-serverREADMEUnit TestsBlubberMakefileDeploy
article_descriptions🚫✅ (P66378)
huggingface_modelserver🚫🚫🚫✅ (P67379)
langid🚫✅ (P67272)
llm🚫🚫🚫 (T369344#10060436)
logo_detection🚫✅ (P66111)
nsfw-model🚫🚫🚫🚫
outlink_topic_model✅ (P67281)
readability_model🚫✅ (P66582)
revert_risk_model✅ (P68590)

Details

SubjectRepoBranchLines +/-
machinelearning/liftwing/inference-servicesmain+4 -4
integration/configmaster+0 -16
machinelearning/liftwing/inference-servicesmain+1 -493
operations/deployment-chartsmaster+4 -4
integration/configmaster+0 -4
machinelearning/liftwing/inference-servicesmain+15 -15
operations/deployment-chartsmaster+1 -1
machinelearning/liftwing/inference-servicesmain+6 -6
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+1 -1
machinelearning/liftwing/inference-servicesmain+1 -1
operations/deployment-chartsmaster+13 -0
operations/deployment-chartsmaster+2 -2
integration/configmaster+5 -0
integration/configmaster+1 -1
machinelearning/liftwing/inference-servicesmain+7 -7
machinelearning/liftwing/inference-servicesmain+13 -13
integration/configmaster+2 -2
operations/deployment-chartsmaster+1 -1
machinelearning/liftwing/inference-servicesmain+3 -3
integration/configmaster+1 -1
operations/deployment-chartsmaster+1 -1
integration/configmaster+1 -1
machinelearning/liftwing/inference-servicesmain+6 -6
integration/configmaster+1 -1
machinelearning/liftwing/inference-servicesmain+9 -9
operations/deployment-chartsmaster+1 -1
machinelearning/liftwing/inference-servicesmain+74 -74
integration/configmaster+1 -1
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1053207 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: logo_detection from src dir

https://gerrit.wikimedia.org/r/1053207

Change #1053241 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: update article_descriptions src path

https://gerrit.wikimedia.org/r/1053241

Change #1053242 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] article_descriptions: migrate to src dir

https://gerrit.wikimedia.org/r/1053242

Change #1053207 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: logo_detection from src dir

https://gerrit.wikimedia.org/r/1053207

Change #1053241 merged by jenkins-bot:

[integration/config@master] inference-services: update article_descriptions src path

https://gerrit.wikimedia.org/r/1053241

Mentioned in SAL (#wikimedia-releng) [2024-07-10T10:39:57Z] <James_F> Zuul: [mediawiki/services/inference-services]: update article_descriptions src path, for T369344

Change #1053242 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] article_descriptions: migrate to src dir

https://gerrit.wikimedia.org/r/1053242

Change #1053536 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: article_descriptions from src dir

https://gerrit.wikimedia.org/r/1053536

Change #1053569 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: update readability_model src path

https://gerrit.wikimedia.org/r/1053569

Change #1053608 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] readability_model: migrate to src dir

https://gerrit.wikimedia.org/r/1053608

Change #1053569 merged by jenkins-bot:

[integration/config@master] inference-services: update readability_model src path

https://gerrit.wikimedia.org/r/1053569

Change #1053608 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] readability_model: migrate to src dir

https://gerrit.wikimedia.org/r/1053608

Change #1053536 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: article_descriptions from src dir

https://gerrit.wikimedia.org/r/1053536

Change #1054080 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: readability_model from src dir

https://gerrit.wikimedia.org/r/1054080

Change #1054085 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: update nsfw-model src path

https://gerrit.wikimedia.org/r/1054085

Change #1054087 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] nsfw-model: migrate to src dir

https://gerrit.wikimedia.org/r/1054087

Change #1054085 merged by jenkins-bot:

[integration/config@master] inference-services: update nsfw-model src path

https://gerrit.wikimedia.org/r/1054085

Change #1054087 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] nsfw-model: migrate to src dir

https://gerrit.wikimedia.org/r/1054087

Change #1054080 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: readability_model from src dir

https://gerrit.wikimedia.org/r/1054080

Change #1054525 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: update outlink_topic_model src path

https://gerrit.wikimedia.org/r/1054525

Change #1054526 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] outlink_topic_model: migrate to src dir

https://gerrit.wikimedia.org/r/1054526

Change #1054525 merged by jenkins-bot:

[integration/config@master] inference-services: update outlink_topic_model src path

https://gerrit.wikimedia.org/r/1054525

Change #1054800 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: update langid src path

https://gerrit.wikimedia.org/r/1054800

Change #1054801 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] langid: migrate to src dir

https://gerrit.wikimedia.org/r/1054801

Change #1054800 merged by jenkins-bot:

[integration/config@master] inference-services: update langid src path

https://gerrit.wikimedia.org/r/1054800

Change #1054526 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] outlink_topic_model: migrate to src dir

https://gerrit.wikimedia.org/r/1054526

Change #1054801 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] langid: migrate to src dir

https://gerrit.wikimedia.org/r/1054801

Change #1055085 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: outlink_topic_model from src dir

https://gerrit.wikimedia.org/r/1055085

Change #1055145 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: langid from src dir

https://gerrit.wikimedia.org/r/1055145

Change #1055151 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: add filters for new model-server paths

https://gerrit.wikimedia.org/r/1055151

Change #1055151 merged by jenkins-bot:

[integration/config@master] inference-services: add filters for new model-server paths

https://gerrit.wikimedia.org/r/1055151

Change #1055085 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: outlink_topic_model from src dir

https://gerrit.wikimedia.org/r/1055085

Change #1060378 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: langid from src dir

https://gerrit.wikimedia.org/r/1060378

Change #1060378 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: langid from src dir

https://gerrit.wikimedia.org/r/1060378

I've deployed langid in staging, but the pod is running into a CrashLoopBackOff issue. This is similar to the issue we faced with outlink caused by previously undeployed changes. The solution we used then was to match the python module usage with other isvcs. I will rebuild the langid model-server image with this fix.

Change #1060471 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] langid: match python module usage with other isvcs

https://gerrit.wikimedia.org/r/1060471

Change #1060471 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] langid: match python module usage with other isvcs

https://gerrit.wikimedia.org/r/1060471

Change #1060790 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: fix langid python module usage

https://gerrit.wikimedia.org/r/1060790

Change #1060790 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: fix langid python module usage

https://gerrit.wikimedia.org/r/1060790

Change #1055145 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: langid from src dir

https://gerrit.wikimedia.org/r/1055145

Following the conversation shown in the screenshot, we are going to remove the llm directory from the LiftWing isvc repo and its corresponding CI pipeline configurations:

Screenshot from 2024-07-18 12-12-46.png (1×1 px, 491 KB)

Change #1062700 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] llm: remove modle-server

https://gerrit.wikimedia.org/r/1062700

Change #1062701 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: remove llm CI pipeline jobs

https://gerrit.wikimedia.org/r/1062701

Change #1063152 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] huggingface: migrate to src dir

https://gerrit.wikimedia.org/r/1063152

Change #1063152 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] huggingface: migrate to src dir

https://gerrit.wikimedia.org/r/1063152

Change #1062701 merged by jenkins-bot:

[integration/config@master] inference-services: remove llm CI pipeline jobs

https://gerrit.wikimedia.org/r/1062701

Change #1063704 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: huggingface from src dir

https://gerrit.wikimedia.org/r/1063704

Change #1063704 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: huggingface from src dir

https://gerrit.wikimedia.org/r/1063704

Change #1062700 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] llm: remove model-server

https://gerrit.wikimedia.org/r/1062700

Change #1067216 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] revert_risk_model: migrate to src dir

https://gerrit.wikimedia.org/r/1067216

Change #1067216 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] revert_risk_model: migrate to src dir

https://gerrit.wikimedia.org/r/1067216

Change #1067933 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: revert_risk_model from src dir

https://gerrit.wikimedia.org/r/1067933

Change #1069029 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[integration/config@master] inference-services: remove filters for old model-server paths

https://gerrit.wikimedia.org/r/1069029

Change #1069029 merged by jenkins-bot:

[integration/config@master] inference-services: remove filters for old model-server paths

https://gerrit.wikimedia.org/r/1069029

Change #1067933 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: revert_risk_model from src dir

https://gerrit.wikimedia.org/r/1067933

Change #1099166 had a related patch set uploaded (by Ilias Sarantopoulos; author: Ilias Sarantopoulos):

[machinelearning/liftwing/inference-services@main] llm: move dir under src/models

https://gerrit.wikimedia.org/r/1099166

Change #1099166 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] llm: move dir under src/models

https://gerrit.wikimedia.org/r/1099166