Page MenuHomePhabricator

Deploy ORES advanced editquality models to cawiki
Closed, ResolvedPublic

Description

  • Deploy models to beta and smoke test.
  • Deploy to production.
  • Enable new models via configuration.
  • Announce on-wiki.

Event Timeline

awight created this task.Apr 18 2018, 9:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 18 2018, 9:20 PM
awight updated the task description. (Show Details)Apr 18 2018, 9:26 PM
awight added a subscriber: Townie.

@Townie Would you be willing to draft a release announcement for these upcoming models? We expect to have the deployment ready by the end of the month.

Here's an example announcement: https://tr.wikipedia.org/w/index.php?title=Vikipedi:K%C3%B6y_%C3%A7e%C5%9Fmesi_(ilginize)&oldid=17339200#ORES_review_tool_deployment_in_Turkish_Wikipedia

Wonderful, thank you! We'll post more information here as the deployment goes forward.

@Trizek-WMF, heads-up that this announcement has just been published on cawiki, see the previous comment.

Change 431036 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] Enable ORES on cawiki

https://gerrit.wikimedia.org/r/431036

Catrope added a subscriber: Catrope.EditedMay 4 2018, 10:49 PM

Note that the goodfaith model for cawiki isn't super helpful. There's a quite good "may be bad faith" filter with 32% precision and 89.1% recall, but the threshold range for that is 0.000 through 0.999. So the only way to make "very likely good faith" not overlap with "may be bad faith" was to set its threshold range to be 1 through 1 (i.e. just the value 1), which has 99.8% precision and 97.2% recall. This seems to indicate that a very large proportion of all inputs get score 1.

Using the default settings would have pegged "very likely good faith" at 99.5% precision, which has 99.5% recall and corresponds to a threshold range of 0.883 through 1. This would have overlapped both with "may be bad faith" (0 - 0.999 at 32/89) and "likely bad faith" (0 - 0.926 at 60/70). On the other end of the spectrum, the "very likely bad faith" filter gets 90.4% precision with 29.2% recall, but its threshold range is 0 - 0.008.

Maybe this is all fine and the model is just very confident, but I felt a bit dirty setting this up.

Deployment scheduled for Wednesday May 9th at 23:00 UTC.

Change 431036 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ORES on cawiki

https://gerrit.wikimedia.org/r/431036

Mentioned in SAL (#wikimedia-operations) [2018-05-09T23:53:18Z] <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Enable ORES on cawiki, lvwiki, huwiki (T192501, T192499, T192496) (duration: 01m 29s)

Catrope closed this task as Resolved.May 10 2018, 12:15 AM
Catrope claimed this task.