Page MenuHomePhabricator

Enable ORES RCFilters for German Wikipedia (dewiki)
Open, Needs TriagePublic

Description

See https://ores.wikimedia.org/v3/scores/dewiki

ORES models are ready. We should deploy the filters.

See https://labels.wmflabs.org/stats/dewiki/16 for those who labeled the most edits to train ORES. These contributors will be most likely to be interested in helping us announce the new filters.

Event Timeline

Halfak created this task.Feb 5 2019, 9:49 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptFeb 5 2019, 9:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 489984 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] Enable ORES on dewiki

https://gerrit.wikimedia.org/r/489984

Forgot to say this earlier: I've scheduled the above patch for deployment on Thursday February 14 at 00:00-01:00 UTC (Wednesday 16:00-17:00).

Ladsgroup added subscribers: Bmueller, Ladsgroup.EditedFeb 13 2019, 11:35 AM

Forgot to say this earlier: I've scheduled the above patch for deployment on Thursday February 14 at 00:00-01:00 UTC (Wednesday 16:00-17:00).

Please don't do it for German Wikipedia. The wiki banned using ORES a while back (three or four years ago, a little bit after the Mediaviewer stuff). Deploying it without community approval first probably will cause a mayhem.

Thanks for flagging this, @Ladsgroup. I believe that there was a protocol at some point that involved a community announcement. In the past, I believe this involved work from @Trizek-WMF to announce and discuss the UI change.

Change 489984 abandoned by Catrope:
Enable ORES on dewiki

https://gerrit.wikimedia.org/r/489984

Forgot to say this earlier: I've scheduled the above patch for deployment on Thursday February 14 at 00:00-01:00 UTC (Wednesday 16:00-17:00).

Please don't do it for German Wikipedia. The wiki banned using ORES a while back (three or four years ago, a little bit after the Mediaviewer stuff). Deploying it without community approval first probably will cause a mayhem.

Thanks for flagging that. I have canceled the scheduled deployment.

JTannerWMF moved this task from Inbox to Q2 2019-20 on the Growth-Team board.

The Growth team has written this code, and all that is left is community outreach. So @Ladsgroup can take the final steps to execute when ready.

JTannerWMF moved this task from Q2 2019-20 to External on the Growth-Team board.Apr 3 2019, 6:16 PM

@JTannerWMF, per T164331: Define a process for adding ORES filters to new wikis when ORES is enabled on those wikis, I believe we'd decided that it is the #Growth team's responsibility to get buy-in for enabling these features since you're maintaining the UI. I'd suggest reaching out to the Wikipedians who did the most labeling work.

User idUser nameLabels added
55655236Christoph Lutz1299
55652041Suzipl628
55655012Hofmsonj567
55655426Charlotte Kammeyer420
55651449Sungly392
55664782Tweedr179
55654694SebasLos168
48577506TIB-NOA100
29659065Queryzo50
27634366Gbeckmann50
6795WiseWoman50
5445698CennoxX50
30584244Thiemo Kreuz (WMDE)48
47527633Adelheid Heftberger35
14663884Florianschmidtwelzow28
27750966Luke08151524
756525Cirdan22
37807544Simulo18
13500683ToBeFree8
25981MB-one8
2331Catrope1
911299Crazy18801
JTannerWMF moved this task from External to Upcoming Work on the Growth-Team board.

When we have bandwidth we will discuss this task with @Trizek-WMF

The request to have ORES filters on de.wp has been made by some community members. IIRC what we discussed, they should take care of announcing it.

If I'm wrong, please fill a request.

@Halfak @Ladsgroup -- @Trizek-WMF and I talked about this. Looking at @Ladsgroup's comment above about how dewiki banned ORES, it seems like the wiki would have to have a new conversation about enabling these models, facilitated by someone from WMF Community Relations. Does that sound like the appropriate next to happen? If so, we will in fact need to file a Community Relations request because Growth team wasn't expecting to add this to our to-do list in the near-term; we're currently doing a ton of community relations as we expand the help panel and newcomer homepage to more wikis. Does building the ORES models come with any validation or other materials to help the community understand how they will perform?

Halfak added a comment.May 8 2019, 2:20 PM

You can get fitness statistics directly from the service. Here's some relevant statistics for the English Wikipedia models and the German Wikipedia models.

https://ores.wikimedia.org/v3/scores/dewiki/?models=damaging|goodfaith&model_info=statistics.roc_auc|statistics.pr_auc|statistics.precision|statistics.recall

https://ores.wikimedia.org/v3/scores/enwiki/?models=damaging|goodfaith&model_info=statistics.roc_auc|statistics.pr_auc|statistics.precision|statistics.recall

In comparison, the German Wikipedia model fares better than the English Wikipedia model. While this comparison is hard to make directly. Surely there are different *types* of vandalism that happen in dewiki vs enwiki. But this gives a general sense that, by the numbers, it looks like the models will work well.

Also, please convey that we prioritize fixing the issues they identify in the wild. So if German Wikipedians can tell us that ORES is generally getting some type of edit wrong, we'll prioritize the work to look into that problem highly.

@Ladsgroup -- I wanted to follow up on my question above. Since dewiki previously banned ORES, do you think that the wiki would need to have a new conversation and consensus about the models before they are deployed?

For clarity, I don't think ORES was ever "banned", but we did receive resistance when we first sought to document ORES on dewiki. That was back in 2014. I think *some* discussion is necessary before we do any deployment on German Wikipedia.

It looks like an old discussion was posted that suggested ORES was coming in some future iteration on RCFilters. See https://de.wikipedia.org/wiki/Wikipedia:Projektdiskussion/Some_updates_about_Recent_Changes_page

It doesn't look like there was any substantial push-back at that point.

MMiller_WMF reassigned this task from MMiller_WMF to Harej.May 30 2019, 9:24 PM
MMiller_WMF added subscribers: Harej, MMiller_WMF.

I discussed this with @Harej, and he is going to pursue the discussion with German Wikipedia. Once we have the go-ahead, the Growth team can deploy the models to the filters.

Restricted Application added a project: User-Harej. · View Herald TranscriptMay 30 2019, 9:24 PM
Harej removed Harej as the assignee of this task.Jun 20 2019, 12:37 AM

As a heads up, I will no longer be able to support this project.

Harej removed a subscriber: Harej.
JTannerWMF moved this task from Q1 2019-20 to Blocked on the Growth-Team board.Jul 18 2019, 1:36 PM

I am moving this to blocked, we can't proceed on this task until there is discussion with German Wikipedia. Tagging @Halfak for awareness.