I quickly checked the two problematic revisions (ruwiki:133170407 and nlwiki:66027941) reported in the doc. Both of them are new pages, meaning they are the first revision of the page. This can cause errors because revert risk model requires the parent revision to calculate the difference between revisions, but in this case, there is no parent revision available.

The new knative autoscaling settings have been applied to outlink and revert risk.

Hi @lbowmaker, thanks for the heads up. Unfortunately I won't be able to attend the meeting today as it conflicts with my other meetings. However, I would like to follow up on the discussion that you and the Enterprise team will have. Could you please share any documents or notes from the meeting afterwards? I would greatly appreciate it! :)

I conducted some load tests on the readability model in staging using the same input and script as we did for revert-risk (code), as they share the same input parameters. The results can be found here: P52406.

Hi @kostajh, there is an updated URL as the model server has been moved to Lift Wing production.

curl "https://inference.discovery.wmnet:30443/v1/models/revertrisk-language-agnostic:predict" -d '{"rev_id": 123456, "lang": "en"}' -H "Host: revertrisk-language-agnostic.revertrisk.wikimedia.org" --http1.1 -k

For more information, please refer to the Lift Wing documentation: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage
Thanks! :)

The new model binary has been uploaded and deployed. Thank you all for working on this! The task is resolved. :)

@klausman Last Friday I deployed new changes to the revert-risk models. In ml-serve-codfw, all pods were updated and running without any issues. But in ml-serve-eqiad, not all pods for revertrisk-multilingual were updated. Specifically, there are some remaining revertrisk-multilingual-predictor-default-00006-* pods, and some of the older pods for revertrisk-language-agnostic were not deleted and crashed. I'm not sure why. Could you take a look? :)

@elukey Yes I agree, we should return http 400 for those domains! I will also send a patch for that.

@elukey Should we add sourceswiki to the changeprop config to filter out edits for wikisource.org? as the outline model is not recommended for use in projects outside of Wikipedia.

Wikiloop is using the mediawiki.revision-score stream in https://github.com/google/wikiloop-doublecheck/blob/master/server/ingest/ores-stream.ts.

@Ladsgroup What I don't understand is how the order for the ORES API response is defined. It seems that for fiwiki and cswiki, the order is ["very likely bad faith", "maybe bad faith", "likely bad faith"]. Does this same order apply to all other wikis as well?

@elukey Research team's plan for the RevertRisk Wikidata model is to evaluate it in Q1, and then improve and deploy it in Q2.

@elukey In Ilias's comment https://phabricator.wikimedia.org/T319170#8807964, the example url is querying thresholds for frwiki damaging model.

I have an assumption about what is causing this problem, but I need to verify it by examining the code further. My assumption is that the new setting has somehow messed up the threshold configs for "maybe bad faith" and "very likely bad faith" (should be null)

The RC filters now only show "maybe malicious" and "probably malicious" which is correct. Don't see "very likely malicious" anymore.

The thresholds used in ores-extension:

@achou, reading the Model Card Data section it looks like your model will not use external links for T328899 outlink, right?

@elukey The redirect_page_link field was added to mediawiki/page/change schema in T325315: Add support for redirects in CirrusSearch. Therefore, the version of /fragment/mediawiki/state/change/page has been updated to 1.1.0. In the prediction_classification_change, we still use 1.0.0.

After investigation, we found that using batching for revscoring models does not provide much improvement in latency. Therefore, I'm going to close this task, but we might explore this approach for other models such as revert-risk.

The model_info query parameter is also available in the endpoints /v3/scores/ in ORES (see https://ores.wikimedia.org/v3#!/scoring/get_v3_scores)

As per the discussion on https://github.com/tiangolo/fastapi/issues/610, it seems that FastAPI does not support query string flags ?features.

I conducted a small experiment using mwapi to obtain features for multiple revision ids. I used a list of 50 different revision ids (same as the request in T341479) and ran the function extractor.extract(revids[:n], scorer_model.features).

The readability model has been deployed to LiftWing staging. It is available via an internal endpoint.

is it possible to get an overview of the model's response time (not only for the errors, but in general?)

I reviewed kserve's logs from June 30 to July 3 and collected statistics on model response times. Out of 58549 requests, 58140 were successful (http_status: 200) and 409 resulted in errors (http_status: 500). Please note that these statistics do not include requests that ended with timeouts/504, as there are no logs on kserve's side.

Timeout issue for be-x-old has been fixed :)

Meanwhile, I have added the list of currently supported languages to the model card: https://meta.wikimedia.org/wiki/Machine_learning_models/Proposed/Language-agnostic_revert_risk#Motivation

I investigated all the Wikipedia projects listed in https://en.wikipedia.org/w/api.php?action=sitematrix&formatversion=2. This source was helpful for checking the metadata for Wiki projects. I initially thought that other languages may have the same issue with redirections, but it turns out that only be-x-old is affected.

We now can see some traffic hitting the outlink model server on LiftWing!
https://grafana.wikimedia.org/d/zsdYRV7Vk/istio-sidecar?orgId=1&var-cluster=eqiad%20prometheus%2Fk8s-mlserve&var-namespace=articletopic-outlink&var-backend=All&var-response_code=All&var-quantile=0.5&var-quantile=0.95&var-quantile=0.99

Added return object https://api.wikimedia.org/wiki/API_reference/Service/Lift_Wing/Revert_risk_score_object

All the model binaries in LW production have been mirrored to /srv/published/wmf-ml-models:

aikochou@stat1008:~$ ls -al /srv/published/wmf-ml-models
total 48
drwxrwxr-x 12 aikochou wikidev 4096 Jun  8 09:28 .
drwxrwxr-x  5 root     wikidev 4096 Jun  7 14:57 ..
drwxrwxr-x 13 aikochou wikidev 4096 Jun  7 17:01 articlequality
drwxrwxr-x 13 aikochou wikidev 4096 Jun  7 17:15 articletopic
drwxrwxr-x 35 aikochou wikidev 4096 Jun  7 17:06 damaging
drwxrwxr-x  4 aikochou wikidev 4096 Jun  7 17:07 draftquality
drwxrwxr-x  3 aikochou wikidev 4096 Jun  8 09:28 drafttopic
drwxrwxr-x 35 aikochou wikidev 4096 Jun  7 17:08 goodfaith
drwxrwxr-x  3 aikochou wikidev 4096 Jun  7 17:08 itemquality
drwxrwxr-x  3 aikochou wikidev 4096 Jun  7 17:08 itemtopic
drwxrwxr-x 11 aikochou wikidev 4096 Jun  7 17:09 reverted
drwxrwxr-x  4 dsaez    wikidev 4096 Jun  6 21:17 revertrisk

I found two problems while testing the following Change-Prop staging config:

outlink-topic-model:
  concurrency: 2
  match_config_need_quotes: ['page_change_kind', 'wiki_id']
  match_config:
    wiki_id: 'enwiki'
    page_change_kind: '/^(edit|create)$/'
    page:
      is_redirect: false
      namespace_id: 0
  namespace: articletopic-outlink
  kafka_topic: 'liftwing.test-outlink-events'

I used a test event https://phabricator.wikimedia.org/P48594 from eqiad.rc1.mediawiki.page_change.

removed ~155G for aikochou

The model inference server has been deployed to ml-staging:

Both models (Language-Agnostic and Multilingual) have been deployed to Lift Wing production. (T332998, T333124) The next step is to work on the public endpoints.

Test the internal endpoint and it works correctly:

aikochou@deploy1002:~$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/revertrisk-multilingual:predict" -d @input.json -H "Host: revertrisk-multilingual.revertrisk.wikimedia.org" --http1.1
{"lang":"en","rev_id":1096086751,"score":{"prediction":false,"probability":{"true":0.3807116729020513,"false":0.6192883270979487}}}
real	0m5.907s
user	0m0.014s
sys	0m0.000s

Next step is to config API gateway to enable the public endpoint.

This model has been deployed to Lift Wing production. Note that the isvc/model has been renamed to revertrisk-language-agnostic

The model has been moved to a new bucket:

The model has been moved to a new bucket: