>>FILES<<

On the KFv1.1 sandbox, I have been able to create a generic revscoring inference service using the files below;

1. custom.yaml inference service configuration file.

apiVersion: serving.kubeflow.org/v1alpha2
kind: InferenceService
metadata:
  name: enwiki-goodfaith
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  default:
    predictor:
      custom:
        container:
          name: kfserving-container
          image: kevinbazira/kfserving-revscoring-model:v14
          env:
            - name: STORAGE_URI
              value: "s3://wmf-ml-models/goodfaith/enwiki/202105132212/"
            - name: INFERENCE_NAME
              value: "enwiki-goodfaith"

2. model.py python service in docker image that is called by custom.yaml.

import kfserving
import mwapi
import os
from revscoring import Model
from revscoring.extractors import api
from typing import Dict

class RevscoringModel(kfserving.KFModel):
    def __init__(self, name: str):
        super().__init__(name)
        self.name = name
        self.ready = False

    def load(self):
        model_uri = os.path.join("/mnt/models/", "model.bin")
        with open(model_uri) as f:
            self.model = Model.load(f)
        self.extractor = api.Extractor(mwapi.Session(
            "https://en.wikipedia.org",
            user_agent="KFServing revscoring demo"))
        self.ready = True

    def predict(self, request: Dict) -> Dict:
        inputs = request["rev_id"]
        feature_values = list(
            self.extractor.extract(inputs, self.model.features))
        results = self.model.score(feature_values)
        return {"predictions": results}


if __name__ == "__main__":
    inference_name = os.environ.get("INFERENCE_NAME")
    model = RevscoringModel(inference_name)
    model.load()
    kfserving.KFServer(workers=1).start([model])

>>INSIDE THE FILES<<

As explained in this task's description, the two major things that got this to work are;

1. Inject the model binary into the container from storage. This happens by:

setting STORAGE_URI environment variable in custom.yaml, the value can mainly point to s3://,gs://,pvc://, file:// and https://(.+?).blob.core.windows.net/(.+) as these are the URIs supported by KFServing at the moment.

- name: STORAGE_URI
  value: "s3://wmf-ml-models/goodfaith/enwiki/202105132212/"

receiving and reading the model binary in model.py

model_uri = os.path.join("/mnt/models/", "model.bin")
with open(model_uri) as f:

for more on how this works, please see @ACraze's explanation here T282802#7109731

2. Pass in model metadata to the container.

Used metadata.name as the value to the INFERENCE_NAME envrionment variable in custom.yaml.

- name: INFERENCE_NAME
  value: "enwiki-goodfaith"

Then retrieved and used this inference_name in model.py

inference_name = os.environ.get("INFERENCE_NAME")
model = RevscoringModel(inference_name)

>>REQUIRED RESULT<<

When creating a new revscoring inference service, I no longer have to update the docker image. I just have to change metadata.name, INFERENCE_NAME value and STORAGE_URI value in custom.yaml. Then in infer.sh change MODEL_NAME to match the metadata.name in custom.yaml.

@kevinbazira this is great news! glad to hear the generic container approach is working so far. I have uploaded the enwiki-damaging model to our public wmf-ml-models bucket, so you should be able to inject that model into a separate container now. Let me know if you run into any issues. uri is : s3://wmf-ml-models/damaging/enwiki/202105260914/model.bin

Also, one thing I noticed, in your model.py file, in the RevscoringModel.load method, when we initialize mw.api.Session, we hardcode to en.wikipedia.org. We might want to add another env var for the language code, so can specify the different wikis needed for each model.

All in all, this looks great so far! Feel free to make a patch set in gerrit once you are ready :)

@ACraze, thank you for sharing the public URI for the enwiki.damaging model. I have been able to create two inference services for two models (enwiki.goodfaith and enwiki.damaging) using one generic container.

Prediction from enwiki-goodfaith inference service:

$ ./infer-enwiki-goodfaith.sh
...
*   Trying 10.97.188.113...
* Connected to 10.97.188.113 (10.97.188.113) port 80 (#0)
> POST /v1/models/enwiki-goodfaith:predict HTTP/1.1
> Host: enwiki-goodfaith.kubeflow-user.example.com
> User-Agent: curl/7.47.0
> Accept: */*
> Cookie: authservice_session=<auth_cookie>
> Content-Length: 21
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 21 out of 21 bytes
< HTTP/1.1 200 OK
< content-length: 112
< content-type: application/json; charset=UTF-8
< date: Thu, 27 May 2021 07:19:06 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 596
< 
* Connection #0 to host 10.97.188.113 left intact
{"predictions": {"prediction": true, "probability": {"false": 0.02523431512745833, "true": 0.9747656848725417}}}

Prediction from enwiki-damaging inference service:

$ ./infer-enwiki-damaging.sh
...
*   Trying 10.97.188.113...
* Connected to 10.97.188.113 (10.97.188.113) port 80 (#0)
> POST /v1/models/enwiki-damaging:predict HTTP/1.1
> Host: enwiki-damaging.kubeflow-user.example.com
> User-Agent: curl/7.47.0
> Accept: */*
> Cookie: authservice_session=<auth_cookie>
> Content-Length: 21
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 21 out of 21 bytes
< HTTP/1.1 200 OK
< content-length: 113
< content-type: application/json; charset=UTF-8
< date: Thu, 27 May 2021 07:19:45 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 488
< 
* Connection #0 to host 10.97.188.113 left intact
{"predictions": {"prediction": false, "probability": {"false": 0.9225825879090204, "true": 0.07741741209097959}}}

In T283526#7117095, @ACraze wrote:

Also, one thing I noticed, in your model.py file, in the RevscoringModel.load method, when we initialize mw.api.Session, we hardcode to en.wikipedia.org. We might want to add another env var for the language code, so can specify the different wikis needed for each model.

Thank you for catching this. I have added a WIKI_URL environment variable in custom.yaml and referenced it in model.py.

I have been able to create two inference services for two models (enwiki.goodfaith and enwiki.damaging) using one generic container.

@kevinbazira this is very exciting! It seems like the generic revscoring image is working well. I have fixed the permissions issue on the gerrit repo and now you should be able to push your code up.

I will try adding some of the other models mentioned in T272874: Prepare 4 ORES English models for Lift Wing (e.g. en.articlequality, en.draftquality) later today and we can try running those sometime next week.

I uploaded two more models to the public bucket for testing:

enwiki.drafttopic: s3://wmf-ml-models/drafttopic/enwiki/202105271548/model.bin
enwiki.articlequality: s3://wmf-ml-models/articlequality/enwiki/wp10/202105271538/model.bin

Change 697577 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] update inference service to work with generic image

https://gerrit.wikimedia.org/r/697577

Change 697577 merged by Accraze:

[machinelearning/liftwing/inference-services@main] update inference service to work with generic image

https://gerrit.wikimedia.org/r/697577

@kevinbazira I've been thinking about how to structure our repo with the generic revscoring image and all the service config files. There are still some unknowns around deployment and following the SRE guidelines with respect to our inference services, so it is highly likely the codebase structure will change a bit in the near future.

For now, while we are testing multiple models, here's what I think we should do:

1. Rename the enwiki-goodfaith/ directory to revscoring/ (which will better reflect the name of the class in model-server/model.py)
2. Make a subdirectory inside of revscoring/ for each inference-service that will run a revscoring model.
3. Inside this directory will be the yaml config/input.json/readme/ anything else that the model might need.

So the directory might look something like this:

➜  inference-services/revscoring
├── enwiki-damaging
│   ├── input.json
│   └── service.yaml
├── enwiki-goodfaith
│   ├── input.json
│   └── service.yaml
├── model-server
│   ├── Dockerfile
│   ├── model.py
│   └── requirements.txt
└── README.md

Does this make any sense? It could be overkill for all 110 ores models, but let's see how it works for a handful of models and then discuss the pros & cons.

Yes, this makes sense. Thank you for thinking through the repo structure @ACraze. The previous structure was getting a bit confusing with the duplication but this new structure is much better and coherent. Going to push patches for it today.

Change 697920 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] update repo structure for generic image

https://gerrit.wikimedia.org/r/697920

Change 697920 merged by Accraze:

[machinelearning/liftwing/inference-services@main] update repo structure for generic image

https://gerrit.wikimedia.org/r/697920

Change 698176 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[machinelearning/liftwing/inference-services@main] add enwiki damaging inference service config

https://gerrit.wikimedia.org/r/698176

Change 698176 merged by Accraze:

[machinelearning/liftwing/inference-services@main] add enwiki damaging inference service config

https://gerrit.wikimedia.org/r/698176

@kevinbazira Just adding notes here after digging into all the different model classes (articlequality, drafttopic, etc..) today:

1. As you mentioned earlier today, articlequality models only support up to Python3.7 and also use an older version of revscoring, which means we will need a separate model-server image to load those types of revscoring models.
2. The editquality models (damaging/goodfaith/reverted/etc.) seem to run well with our current image, although we should think about loading them all into inference services to see if there are any issues like old revscoring dependencies.
3. The drafttopic/articletopic models use additional word embeddings that we would either need to package inside a container, or inject via storage. Let's hold off on migrating these to KFServing for now, as there is talk about the language-agnostic Outlink topic model replacing these types of models.

Should we close this task and make new tasks (one for developing a model-server for articlequality models and one for preparing a full editquality migration)?

Thank you for the breakdown @ACraze. Yes, we can close this task.