In my opinion:

• A is under ORES' support team responsibility
• B is under Collaboration and ORES team responsibility
• C is under ORES' support team responsibility and the community ambassadors from the given wiki
• D is under Collaboration team, ORES team and the community ambassadors from the given wiki's responsibility

I disagree on A. There are many tools that use ORES beyond Edit-Review-Improvements and I don't think we can be responsible for enabling ORES in every tool as support becomes available. In many tools, support is automatically detected. Maybe we should start talking about which endpoints for ORES will allow a system to automatically discover new support.

I feel like B is squarely on the shoulders of Machine Learning Platform. We'll make sure the data is there and we plan to build out basic documentation on how to choose a threshold for your use of a model.

I'm OK with sharing C. When it comes to "new features", I wouldn't imagine we'd be announcing anything like "new filters", but we might announce new models or new model capabilities.

IMO, D should eventually be on Edit-Review-Improvements with Machine Learning Platform consulting.

In T164331#3232600, @Halfak wrote:

I disagree on A. There are many tools that use ORES beyond Edit-Review-Improvements and I don't think we can be responsible for enabling ORES in every tool as support becomes available. In many tools, support is automatically detected. Maybe we should start talking about which endpoints for ORES will allow a system to automatically discover new support.

Thankfully, we already have this. If you enable the ORES extension on a new wiki, then the ORES-based filters are added to the RCFilters UI automatically, using sensible default settings for the filter thresholds based on the test stats. I set these defaults to the values that were most commonly used, and we found that they worked well for the newest pair of models (fiwiki). For some models (e.g. ones with significantly higher or lower fitness), it is advisable to tweak these settings. and possibly remove some filters altogether. One thing I'd like to improve here is add the "Very likely good faith" filter to the defaults: right now it doesn't exist by default but we make it exist almost everywhere.

On a technical level: the code for the ORES-RCFilters integration is in the ORES extension, using hooks provided by MW core that let extensions add their own filters. The default config is in the OresFiltersThresholds setting in extension.json and the per-wiki configs are in wmf-config (slow link). The format of the config should be intuitive to people who already understand the format of the test stats output or are generally familiar with ORES.

(I'm assuming a world where the RCFilters beta feature is available on all wikis; that is not the case yet but will be on May 9th, which is soon enough that I wrote the above as if it'd already happened.)

@Halfak @Ladsgroup @awight, can I close this ticket? I.e., do we have a "process"?

By a "process" here, all we really need is for RevScoring to affirm that when you're going to enable ORES for a new wiki, you will give us a couple of weeks' notice, so 1) @Trizek-WMF can make announcements on the wiki and 2) Collaboration can turn on the new filters for that wiki at the right time. (Am I missing anything? @Catrope?)

And how should you inform us, then? I'd suggest 1) file a task against "Edit-Review-Improvements" with a title like "Enable ORES on WikiX August 1", and 2) send an email to Roan, me and Benoit. You're wondering why you need to send an email if you file the task? Because, depending on our vacation schedule, etc., we otherwise might not see the task for a week or more (we get a lot of new tasks every week).

RevScoring folks, does that sound OK? Can I say we now have a "process"?

Thanks for thinking about this, @jmatazzoni. I think your proposal is going to work, but after discussing internally, we'd like to ask to change it slightly for technical reasons.

Once ORES models are ready for a wiki, we would like to deploy them to the ORES service, and enable the Extension:ORES backend which starts to cache scores in MediaWiki. This lets us burn in the changes and start to populate the scores in the background. We enable these behaviors by adding to the $wgOresModels array.

It would be best if we could decouple the UI deployment from these backend preparations, and to that end we are suggesting a new configuration option $wgOresUiEnabled, perhaps? The idea would be that, once the backend has been deployed and looks sufficiently stable, your team can do the honors of announcing, toggling the UI to the enabled state, and possibly reverting the config if the users push back. I think this configuration should control both RC filters and the legacy ORES filters on Special:Contribution. I'll boldly create a subtask along these lines, feel free to invalidate if this doesn't sound good to you.

@awight: A couple weeks ago I wrote https://gerrit.wikimedia.org/r/#/c/359364/ for T167908: Summarize what it will take to separate UI and infrastructure for ORES Extension which does basically exactly that.

@awight @Ladsgroup @Halfak, just coming back to this now, and wondering about Joe's question:

RevScoring folks, does that sound OK? Can I say we now have a "process"?

And if so, do we need to document that process somewhere else besides here in Phab?

Good question. I'm not sure where the right place to put this is. It's a wmf-specific protocol so probably wikitech over mediawiki.org. We already have ORES/Deployment that walks through the commands necessary to deploy/rollback. Maybe we should have something like Review Filters/Deployment process. Maybe @Trizek-WMF has some recommendations on where such socio-technical protocols should live.

So, I imagined that there would be some public notifications and some statement about who is responsible for enabling the ORES filters. I was under the impression that enabling the ORES extension does *not* enable the filters. And this is by design so that my team can deploy the infrastructure and product teams can decide when and how to make UI changes.

The filters are indeed not automatically added to the UI. I guess that the conclusion of that task is that Growth will take care of it as a maintenance task. Or should document it so that someone can make it.