The srwiki advanced edit quality models are deployed and ready for RC integration.
|Resolved||Catrope||T197012 Enable srwiki edit quality filters in RecentChanges|
|Resolved||awight||T194745 Train / test reverted model for srwiki|
|Resolved||Ladsgroup||T174687 Add language support for Serbian|
|Resolved||Halfak||T199355 Investigate srwiki goodfaith model, why is it so bad?|
|Resolved||None||T220556 New labeling campaign for srwiki|
Sorry for the delay, this caught me right in the middle of a busy time.
I looked at the properties of the srwiki model, and while the damaging model is usable, the goodfaith model is not. The highest precision for bad faith that this model can achieve is 23.1% (see queries for >=0.23 and >=0.24), which means we could implement a "may be bad faith" filter (which would have 16.8% precision at 62.5% recall) but not a "likely bad faith" or "very likely bad faith" model, because we want those to have a precision of at least 45% and 60% respectively, and ideally 60% and 90%.
The damaging model is adequate though; it's not the best model we have, but it's workable. We could configure the following filters:
- Very likely good: 99.5% precision at 100% (?!) recall, or alternatively 100% precision at 90.7% recall
- May be bad: 15.5% precision at 90.1% recall (we aim for 90% recall or 15% precision, so this fits that perfectly)
- Likely bad: 45.7% precision at 39.9% recall (normally we aim for 60% precision, but 45% is fine for lower-fit models)
- Very likely bad: 75% precision at 17.5% recall (normally we aim for 90% precision, but that would lead to 5.7% recall which I think is too low)
@awight The last time we ran into this situation, on T192498: Deploy ORES advanced editquality models to arwiki, I ended up deploying only the damaging model but not the goodfaith model, and you said that we should consider not deploying (or undeploying) low quality models. How do you feel about this case?
I just wanted to post on this task to clarify its status. Given that the Growth team has enabled the filters, and the remainder of the conversation is about improvements to the models, I'm going to resolve this ticket. @Halfak, is there a separate task where it could be good to have that conversation?