A WME customer is working on an application that will be a high volume consumer of RevertRisk data. The customer is reporting an unexpected number of null scores being returned by the model. Our goal is to (1) investigate and identify the reason for these null scores, then (2) fix any issues that might be causing these null scores to be returned. The customer is also seeing events with a score, so it's not the case that the JSON path is wrong and all events are returning null scores.
Definition of score:
score = (
data.get("version", {})
.get("scores", {})
.get("revertrisk", {})
.get("probability", {})
.get("true", None)
)Examples of events producing null scores:
It has reported by WME that there are errors with revertrisk scores showing nulls when making requests to the public API on LiftWing.
The following revisions have been reported (the language wasn't mentioned -- they seem to be for multiple languages).
revision_id,page_id 83683012,342014 84340166,2455844 70356784,9946471 35809399,232548 76777654,1090533 70356748,9946465 81987290,4188256 53202907,8759009 54192350,8807323 82488525,7302042 166871684,11176613 25238086,7872922 68882343,7411503 38027525,881178 254642515,476795 54539135,1637895 176976718,4026126 54917510,550785 29771319,4493955 1241413531,33663997 81483207,10470054 1277757288,12619081 29972557,386158 251855396,13145699 1037954403,55780478 72003196,7651031 166012213,11067926 37838936,3417252 82762131,1125446 71982980,1438281 36540223,2379991 21723694,2708922 51582186,175611 70356874,9946493 166871533,11176608 183714576,4919327 70356852,9946488 1279260079,76376185 84177170,5853372 51397937,6564642 1257934825,50687758 34654257,2950783 26201482,1728232 1285807239,14986513
In some cases, when we attempted to replicate the issue for the events listed above, we found that these events sometimes do produce scores. As such, we don't believe that these null scores are caused by a bug in the model code or a pattern about the types of events that product null scores.
$ curl https://api.wikimedia.org/service/lw/inference/v1/models/revertrisk-language-agnostic:predict -X POST -d '{"lang": "en", "rev_id": 1285807239}' -H "Content-type: application/json"