User Details
- User Since
- Feb 15 2022, 2:51 PM (83 w, 3 d)
- Availability
- Available
- IRC Nick
- aiko
- LDAP User
- Unknown
- MediaWiki User
- AChou-WMF [ Global Accounts ]
Today
I quickly checked the two problematic revisions (ruwiki:133170407 and nlwiki:66027941) reported in the doc. Both of them are new pages, meaning they are the first revision of the page. This can cause errors because revert risk model requires the parent revision to calculate the difference between revisions, but in this case, there is no parent revision available.
Thu, Sep 14
The new knative autoscaling settings have been applied to outlink and revert risk.
Tue, Sep 12
Hi @lbowmaker, thanks for the heads up. Unfortunately I won't be able to attend the meeting today as it conflicts with my other meetings. However, I would like to follow up on the discussion that you and the Enterprise team will have. Could you please share any documents or notes from the meeting afterwards? I would greatly appreciate it! :)
Mon, Sep 11
Fri, Sep 8
Tue, Sep 5
Mon, Sep 4
Tue, Aug 29
Hi @kostajh, there is an updated URL as the model server has been moved to Lift Wing production.
curl "https://inference.discovery.wmnet:30443/v1/models/revertrisk-language-agnostic:predict" -d '{"rev_id": 123456, "lang": "en"}' -H "Host: revertrisk-language-agnostic.revertrisk.wikimedia.org" --http1.1 -k
For more information, please refer to the Lift Wing documentation: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Usage
Thanks! :)
Thu, Aug 24
Aug 14 2023
The new model binary has been uploaded and deployed. Thank you all for working on this! The task is resolved. :)
@klausman Last Friday I deployed new changes to the revert-risk models. In ml-serve-codfw, all pods were updated and running without any issues. But in ml-serve-eqiad, not all pods for revertrisk-multilingual were updated. Specifically, there are some remaining revertrisk-multilingual-predictor-default-00006-* pods, and some of the older pods for revertrisk-language-agnostic were not deleted and crashed. I'm not sure why. Could you take a look? :)
Aug 11 2023
Aug 9 2023
@elukey Yes I agree, we should return http 400 for those domains! I will also send a patch for that.
@elukey Should we add sourceswiki to the changeprop config to filter out edits for wikisource.org? as the outline model is not recommended for use in projects outside of Wikipedia.
Aug 8 2023
Wikiloop is using the mediawiki.revision-score stream in https://github.com/google/wikiloop-doublecheck/blob/master/server/ingest/ores-stream.ts.
Aug 7 2023
@Ladsgroup What I don't understand is how the order for the ORES API response is defined. It seems that for fiwiki and cswiki, the order is ["very likely bad faith", "maybe bad faith", "likely bad faith"]. Does this same order apply to all other wikis as well?
Aug 4 2023
@elukey Research team's plan for the RevertRisk Wikidata model is to evaluate it in Q1, and then improve and deploy it in Q2.
Aug 3 2023
@elukey In Ilias's comment https://phabricator.wikimedia.org/T319170#8807964, the example url is querying thresholds for frwiki damaging model.
Aug 2 2023
I have an assumption about what is causing this problem, but I need to verify it by examining the code further. My assumption is that the new setting has somehow messed up the threshold configs for "maybe bad faith" and "very likely bad faith" (should be null)
The RC filters now only show "maybe malicious" and "probably malicious" which is correct. Don't see "very likely malicious" anymore.
The thresholds used in ores-extension:
Aug 1 2023
Jul 28 2023
Jul 27 2023
After investigation, we found that using batching for revscoring models does not provide much improvement in latency. Therefore, I'm going to close this task, but we might explore this approach for other models such as revert-risk.
The model_info query parameter is also available in the endpoints /v3/scores/ in ORES (see https://ores.wikimedia.org/v3#!/scoring/get_v3_scores)
As per the discussion on https://github.com/tiangolo/fastapi/issues/610, it seems that FastAPI does not support query string flags ?features.
Jul 26 2023
I conducted a small experiment using mwapi to obtain features for multiple revision ids. I used a list of 50 different revision ids (same as the request in T341479) and ran the function extractor.extract(revids[:n], scorer_model.features).
Jul 12 2023
Jul 6 2023
Jul 5 2023
The readability model has been deployed to LiftWing staging. It is available via an internal endpoint.
is it possible to get an overview of the model's response time (not only for the errors, but in general?)
I reviewed kserve's logs from June 30 to July 3 and collected statistics on model response times. Out of 58549 requests, 58140 were successful (http_status: 200) and 409 resulted in errors (http_status: 500). Please note that these statistics do not include requests that ended with timeouts/504, as there are no logs on kserve's side.
Jul 3 2023
Jun 30 2023
Timeout issue for be-x-old has been fixed :)
Meanwhile, I have added the list of currently supported languages to the model card: https://meta.wikimedia.org/wiki/Machine_learning_models/Proposed/Language-agnostic_revert_risk#Motivation
I investigated all the Wikipedia projects listed in https://en.wikipedia.org/w/api.php?action=sitematrix&formatversion=2. This source was helpful for checking the metadata for Wiki projects. I initially thought that other languages may have the same issue with redirections, but it turns out that only be-x-old is affected.
Jun 29 2023
Jun 26 2023
Jun 22 2023
Jun 21 2023
Jun 15 2023
We now can see some traffic hitting the outlink model server on LiftWing!
https://grafana.wikimedia.org/d/zsdYRV7Vk/istio-sidecar?orgId=1&var-cluster=eqiad%20prometheus%2Fk8s-mlserve&var-namespace=articletopic-outlink&var-backend=All&var-response_code=All&var-quantile=0.5&var-quantile=0.95&var-quantile=0.99
Jun 14 2023
Jun 12 2023
Jun 8 2023
All the model binaries in LW production have been mirrored to /srv/published/wmf-ml-models:
aikochou@stat1008:~$ ls -al /srv/published/wmf-ml-models total 48 drwxrwxr-x 12 aikochou wikidev 4096 Jun 8 09:28 . drwxrwxr-x 5 root wikidev 4096 Jun 7 14:57 .. drwxrwxr-x 13 aikochou wikidev 4096 Jun 7 17:01 articlequality drwxrwxr-x 13 aikochou wikidev 4096 Jun 7 17:15 articletopic drwxrwxr-x 35 aikochou wikidev 4096 Jun 7 17:06 damaging drwxrwxr-x 4 aikochou wikidev 4096 Jun 7 17:07 draftquality drwxrwxr-x 3 aikochou wikidev 4096 Jun 8 09:28 drafttopic drwxrwxr-x 35 aikochou wikidev 4096 Jun 7 17:08 goodfaith drwxrwxr-x 3 aikochou wikidev 4096 Jun 7 17:08 itemquality drwxrwxr-x 3 aikochou wikidev 4096 Jun 7 17:08 itemtopic drwxrwxr-x 11 aikochou wikidev 4096 Jun 7 17:09 reverted drwxrwxr-x 4 dsaez wikidev 4096 Jun 6 21:17 revertrisk
Jun 7 2023
May 30 2023
I found two problems while testing the following Change-Prop staging config:
outlink-topic-model: concurrency: 2 match_config_need_quotes: ['page_change_kind', 'wiki_id'] match_config: wiki_id: 'enwiki' page_change_kind: '/^(edit|create)$/' page: is_redirect: false namespace_id: 0 namespace: articletopic-outlink kafka_topic: 'liftwing.test-outlink-events'
I used a test event https://phabricator.wikimedia.org/P48594 from eqiad.rc1.mediawiki.page_change.
May 26 2023
May 24 2023
removed ~155G for aikochou
May 23 2023
The model inference server has been deployed to ml-staging:
May 19 2023
Test the internal endpoint and it works correctly:
aikochou@deploy1002:~$ time curl "https://inference.svc.eqiad.wmnet:30443/v1/models/revertrisk-multilingual:predict" -d @input.json -H "Host: revertrisk-multilingual.revertrisk.wikimedia.org" --http1.1 {"lang":"en","rev_id":1096086751,"score":{"prediction":false,"probability":{"true":0.3807116729020513,"false":0.6192883270979487}}} real 0m5.907s user 0m0.014s sys 0m0.000s
Next step is to config API gateway to enable the public endpoint.
This model has been deployed to Lift Wing production. Note that the isvc/model has been renamed to revertrisk-language-agnostic
May 17 2023
May 16 2023
May 12 2023
The model has been moved to a new bucket:
The model has been moved to a new bucket: