User Details
- User Since
- Feb 15 2022, 2:51 PM (117 w, 12 h)
- Availability
- Available
- IRC Nick
- aiko
- LDAP User
- Unknown
- MediaWiki User
- AChou-WMF [ Global Accounts ]
Yesterday
Thanks for sharing the use case!
Potentially called on all edit attempts by not-yet-logged-in users.
One thing to note is that for edits by not-yet-logged-in users, the revert risk multilingual (RRML) model might be more suitable than revertrisk language agnostic (RRLA) model as it handles bias better. But RRML requires more resources and is much slower, with prediction latency ranging from hundreds of ms to a few seconds.
Mon, May 6
Can't believe I missed this :(
Fri, Apr 26
I got an error when testing the batch model after deploying the new image of kserve 0.12.1 for revert risk models
aikochou@deploy1002:~$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/revertrisk-language-agnostic:predict" -d@./input_some_succeed.json -H "Host: revertrisk-language-agnostic-batcher.revertrisk.wikimedia.org" --http1.1 -k | jq '.' { "error": "AttributeError : 'JSONResponse' object has no attribute 'encode'" }
It worked before. There may be a change in kserve 0.12.1 that's causing the problem. I'll debug this.
Hi @kostajh, yes, this is something we can work on this quarter. I am wondering if there's an ongoing project or product in development that needs this feature. If so, could you provide the links? Also, do you have an estimate of the expected traffic for this feature? I'm assuming it will be requested via the external endpoint, correct?
Tue, Apr 23
Thanks, that is what I am proposing as well. @achou, how feasible do you think this is from your side? It would involve accepting a POST with all the features (https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/blob/main/knowledge_integrity/featureset.py?ref_type=heads) needed.
Fri, Apr 19
Tue, Apr 16
@kostajh @XiaoXiao-WMF thanks for tagging. Sorry I was unaware of the discussion here. The ML team is currently in the middle of quarterly planning. I will bring up the proposal during our planning and get back to you shortly!
Apr 12 2024
Apr 11 2024
Apr 9 2024
I built a RRML image locally using the Pytorch 2.2.x base image from T360638.
Apr 8 2024
Apr 5 2024
We have deployed the new RRLA model server to production.
Apr 4 2024
This task is complete. I've created T361881 to follow up on the above test results issue.
FYI @MunizaA :)
The new RRLA model server featuring KI v.0.6 has been deployed to ML-staging. I used wrk to conduct load testing and compare the performance between the old and new versions. The results for the previous version are under P59447, and the results for the new version are under P59464. From these results, it's clear that the new KI version does not affect the performance metrics, such as average latency and RPS.
I repost what I previously wrote here as the issue is more related to deployment.
This task is complete. Check out these examples:
This task is complete. Check out these examples of new error messages:
$ curl "https://inference-staging.svc.codfw.wmnet:30443/v1/models/revertrisk-language-agnostic:predict" -d '{"rev_id": 15925124, "lang": "ro"}' -H "Host: revertrisk-language-agnostic.revertrisk.wikimedia.org" --http1.1 -k | jq '.' { "detail": "Could not make prediction for revision 15925124 (ro). Reason: revision_missing" }
Apr 3 2024
@kevinbazira posed a question - how can end users switch between batch and non-batch requests?
Apr 2 2024
Mar 28 2024
@isarantopoulos do you remember the config values in locust.conf when you ran the revertrisk tests? I can't reproduce the result in revertrisk_stats.csv. I haven't deployed RRLA to staging yet, so it's the same model you tested.
Mar 27 2024
Mar 26 2024
Mar 25 2024
Mar 22 2024
Mar 20 2024
Mar 19 2024
It would be nice to wait for an additional patch (improving error messages) to be merged.
Mar 15 2024
Mar 14 2024
Mar 13 2024
Mar 12 2024
Mar 5 2024
The PR for pydantic v2 in kserve has been merged! We can use this commit https://github.com/kserve/kserve/commit/426fe21da0612ea6ef4a116b5114270313e02bbb to test the RRLA model-server :)
Mar 1 2024
Feb 29 2024
Knowledge Integrity v0.6.0 improved error representations by introducing an Error data class and different error codes for various situations when fetching MediaWiki API for revisions. (See https://gitlab.wikimedia.org/repos/research/knowledge_integrity/-/blob/main/knowledge_integrity/mediawiki.py?ref_type=heads#L14-26)