Page MenuHomePhabricator

Define a process for adding ORES filters to new wikis when ORES is enabled on those wikis
Closed, ResolvedPublic

Description

When the Damaging and Good Faith tests are enabled on a new wiki—especially in the future, when Collab Team is paying less attention—there should be a process for
a) requesting that the ORES filters be turned on,
b) making sure that the necessary precision/recall data are available to set threshold level,
c) letting the community know that the new features are coming,
d) making sure the thresholds are set, preferences are adjusted and the features rolled out.

None of this is complicated, but we should define the steps for a routine, assign roles and document the process somewhere . Can we do all that on this ticket, or should we have a little meeting with Aaron, Roan, Benoit, me and whomever?

Event Timeline

Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptMay 3 2017, 12:09 AM

In my opinion:

  • A is under ORES' support team responsibility
  • B is under Collaboration and ORES team responsibility
  • C is under ORES' support team responsibility and the community ambassadors from the given wiki
  • D is under Collaboration team, ORES team and the community ambassadors from the given wiki's responsibility
Halfak added a comment.May 3 2017, 6:29 PM

I disagree on A. There are many tools that use ORES beyond Edit-Review-Improvements and I don't think we can be responsible for enabling ORES in every tool as support becomes available. In many tools, support is automatically detected. Maybe we should start talking about which endpoints for ORES will allow a system to automatically discover new support.

I feel like B is squarely on the shoulders of Scoring-platform-team. We'll make sure the data is there and we plan to build out basic documentation on how to choose a threshold for your use of a model.

I'm OK with sharing C. When it comes to "new features", I wouldn't imagine we'd be announcing anything like "new filters", but we might announce new models or new model capabilities.

IMO, D should eventually be on Edit-Review-Improvements with Scoring-platform-team consulting.

I disagree on A. There are many tools that use ORES beyond Edit-Review-Improvements and I don't think we can be responsible for enabling ORES in every tool as support becomes available. In many tools, support is automatically detected. Maybe we should start talking about which endpoints for ORES will allow a system to automatically discover new support.

Thankfully, we already have this. If you enable the ORES extension on a new wiki, then the ORES-based filters are added to the RCFilters UI automatically, using sensible default settings for the filter thresholds based on the test stats. I set these defaults to the values that were most commonly used, and we found that they worked well for the newest pair of models (fiwiki). For some models (e.g. ones with significantly higher or lower fitness), it is advisable to tweak these settings. and possibly remove some filters altogether. One thing I'd like to improve here is add the "Very likely good faith" filter to the defaults: right now it doesn't exist by default but we make it exist almost everywhere.

On a technical level: the code for the ORES-RCFilters integration is in the ORES extension, using hooks provided by MW core that let extensions add their own filters. The default config is in the OresFiltersThresholds setting in extension.json and the per-wiki configs are in wmf-config (slow link). The format of the config should be intuitive to people who already understand the format of the test stats output or are generally familiar with ORES.

(I'm assuming a world where the RCFilters beta feature is available on all wikis; that is not the case yet but will be on May 9th, which is soon enough that I wrote the above as if it'd already happened.)

@Halfak @Ladsgroup @awight, can I close this ticket? I.e., do we have a "process"?

By a "process" here, all we really need is for RevScoring to affirm that when you're going to enable ORES for a new wiki, you will give us a couple of weeks' notice, so 1) @Trizek-WMF can make announcements on the wiki and 2) Collaboration can turn on the new filters for that wiki at the right time. (Am I missing anything? @Catrope?)

And how should you inform us, then? I'd suggest 1) file a task against "Edit-Review-Improvements" with a title like "Enable ORES on WikiX August 1", and 2) send an email to Roan, me and Benoit. You're wondering why you need to send an email if you file the task? Because, depending on our vacation schedule, etc., we otherwise might not see the task for a week or more (we get a lot of new tasks every week).

RevScoring folks, does that sound OK? Can I say we now have a "process"?

Thanks for thinking about this, @jmatazzoni. I think your proposal is going to work, but after discussing internally, we'd like to ask to change it slightly for technical reasons.

Once ORES models are ready for a wiki, we would like to deploy them to the ORES service, and enable the Extension:ORES backend which starts to cache scores in MediaWiki. This lets us burn in the changes and start to populate the scores in the background. We enable these behaviors by adding to the $wgOresModels array.

It would be best if we could decouple the UI deployment from these backend preparations, and to that end we are suggesting a new configuration option $wgOresUiEnabled, perhaps? The idea would be that, once the backend has been deployed and looks sufficiently stable, your team can do the honors of announcing, toggling the UI to the enabled state, and possibly reverting the config if the users push back. I think this configuration should control both RC filters and the legacy ORES filters on Special:Contribution. I'll boldly create a subtask along these lines, feel free to invalidate if this doesn't sound good to you.

Keegan added a subscriber: Keegan.Jul 13 2017, 4:21 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 2 2018, 10:56 PM

@awight @Ladsgroup @Halfak, just coming back to this now, and wondering about Joe's question:

RevScoring folks, does that sound OK? Can I say we now have a "process"?

And if so, do we need to document that process somewhere else besides here in Phab?

Good question. I'm not sure where the right place to put this is. It's a wmf-specific protocol so probably wikitech over mediawiki.org. We already have ORES/Deployment that walks through the commands necessary to deploy/rollback. Maybe we should have something like Review Filters/Deployment process. Maybe @Trizek-WMF has some recommendations on where such socio-technical protocols should live.

kostajh assigned this task to Trizek-WMF.Nov 1 2018, 3:03 PM

OK, assigning to @Trizek-WMF. We had triaged this as "Resolved" so Benoît, feel free to close this if you don't think it also needs a page on wikitech, otherwise could you please add relevant documentation on wikitech?

Trizek-WMF removed Trizek-WMF as the assignee of this task.Nov 5 2018, 2:24 PM

We already have ORES/Deployment that walks through the commands necessary to deploy/rollback. Maybe we should have something like Review Filters/Deployment process. Maybe @Trizek-WMF has some recommendations on where such socio-technical protocols should live.

From a community request perspective, there a short page explaining how to request ORES on a wiki. Can someone change the template created by the button to automatically add a step to enable ORES on RecentChanges? When that step is reached, de deployment would be done (under whose responsibility, that's the question).

Since the request is made by volunteers, we expect those volunteers to follow up on the process.

kostajh closed this task as Resolved.Nov 7 2018, 3:18 PM
kostajh claimed this task.

Can someone change the template created by the button to automatically add a step to enable ORES on RecentChanges?

@Trizek-WMF , see @Catrope's comment above: "If you enable the ORES extension on a new wiki, then the ORES-based filters are added to the RCFilters UI automatically". Given there is already the page on Mediawiki I'll mark this as resolved. cc @MMiller_WMF

So, I imagined that there would be some public notifications and some statement about who is responsible for enabling the ORES filters. I was under the impression that enabling the ORES extension does *not* enable the filters. And this is by design so that my team can deploy the infrastructure and product teams can decide when and how to make UI changes.

The filters are indeed not automatically added to the UI. I guess that the conclusion of that task is that Growth will take care of it as a maintenance task. Or should document it so that someone can make it.