Page MenuHomePhabricator

Deploy ORES advanced editquality models to arwiki
Closed, ResolvedPublic

Description

  • Deploy models to beta and smoke test.
  • Deploy to production.
  • Enable new models via configuration.
  • Announce on-wiki.

Event Timeline

awight created this task.Apr 18 2018, 9:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 18 2018, 9:18 PM
awight updated the task description. (Show Details)Apr 18 2018, 9:26 PM
awight added a subscriber: Ghassanmas.

@Ghassanmas Would you be willing to draft a release announcement for these upcoming models? We expect to have the deployment ready by the end of the month.

Here's an example announcement: https://tr.wikipedia.org/w/index.php?title=Vikipedi:K%C3%B6y_%C3%A7e%C5%9Fmesi_(ilginize)&oldid=17339200#ORES_review_tool_deployment_in_Turkish_Wikipedia

Ghassanmas added a comment.EditedApr 19 2018, 11:51 AM

Yes of course ! I would be happy to work on that also its really an easy small task to do !. To follow up with deployment date, will be there an announcement on ai-request@lists.wikimedia.org when the exact time is known?

Thanks! The deployment will take several steps, and it's only the very last step "Enable new models via configuration" which is visible to wiki users. We can use this task for coordinating, and it should be the Global-Collaboration team who does the final deployment, since they own the "recent changes new filters" interface where ORES features will appear.

While I was working today on the draft, on the recent changes section I can see that the last edits which has high score in "reverted" is already highlights, shouldn't the highlighting be based on the 'damaging' model ? so is it deployed now? Arabic Wikipedia Recent Changes

I have published the draft in a very similar way to the example mentioned before, given the the tool is already working on the Arabic recent changes.

Ghassanmas updated the task description. (Show Details)Apr 27 2018, 5:54 PM
Ghassanmas updated the task description. (Show Details)

I looked at the generated models for arwiki and the goodfaith one is not good enough to use on the wiki. See T193905: arwiki goodfaith model is not usable. The damaging model is usable but not great:

  • At 15% precision we get 88% recall. This is excellent for a "may have problems" filter
  • At 49.7% precision (nearest to 45%), we get 10.6% recall
  • At 62.1% precision (nearest to 60%), we get 7.4% recall. This is where we put "likely have problems" by default, but we're probably better off with the 50/10 threshold above.
  • Queries for 75%, 90%, and higher precision levels all returned a setting with 100% precision but 0.3% recall. We'd normally put "very likely have problems" at 90% precision, but in this case we can't use that, so we just wouldn't have this filter.

I'm going to deploy this with a "may have problems" filter at 15/88, a "likely have problems" filter at 50/10, and no "very likely problems" filter, but it might be worth looking into improving the damaging model. I won't deploy any goodfaith filters at all, because that model provides no usable thresholds.

Change 431035 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] Enable ORES on arwiki

https://gerrit.wikimedia.org/r/431035

Deployment scheduled for Wednesday May 9th at 23:00 UTC.

Change 431035 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ORES on arwiki

https://gerrit.wikimedia.org/r/431035

Mentioned in SAL (#wikimedia-operations) [2018-05-10T00:06:08Z] <catrope@tin> Synchronized wmf-config/InitialiseSettings.php: Enable ORES on arwiki (T192498) (duration: 01m 20s)

Damaging is deployed, but goodfaith isn't, per my earlier comment. Should we close this task or keep it open?

awight closed this task as Resolved.Jun 7 2018, 1:32 PM
awight claimed this task.
awight moved this task from Pending deployment to Done on the Scoring-platform-team (Current) board.

@Catrope I think this is a good precedent, to not deploy or even undeploy low-quality models. Thanks for documenting in a future task! Let's go ahead and close this.