@Scardenasmolinar - The Growth team is happy to meet and discuss any questions you may have about Community Configuration!

We don't have very thorough technical documentation yet, as the Community Configuration 2.0 project is still early on, but here's some info that might help: Community Configuration Product Requirements Document.

If you want to meet to chat through any technical questions, @Urbanecm_WMF or @Sgs could help.

Hi @KStoller-WMF, thanks for the help! While thinking about potential risks/failures of Automoderator, we found that if the revert risk range is changed drastically, the tool might be rendered useless or overly aggressive in removing edits. I see there is a potential feature (Feature 2) that could help mitigate that risk. Is there a plan to implement it, or are you still on the fence about it? We were also thinking it would be nice to be able to send an alert to admins when a configuration range changes drastically. Could that happen in the longer term?

@Scardenasmolinar do you think this would still be a problem if the config access would be limited to interface admins? For us on rowiki this works just fine for our revert bot, but we change the config very rarely.

While thinking about potential risks/failures of Automoderator, we found that if the revert risk range is changed drastically, the tool might be rendered useless or overly aggressive in removing edits.

That's definitely a legit concern. It's one of the risks we've discussed when conducting a Community Configuration pre-mortem. I don't think I have a perfect solution, but here are some ideas that might help:

• As @Strainu mentions, you could consider limiting config access to a more priviledged user group. Currently the Growth team limits Configuration changes to admins. Anyone can view the configuration, but only admins can actually edit the form.

• Admins (or any account holders) can watch the associated config pages, so there is some built-in oversight of changes. Example for English WIkipedia's GrowthExperimentsConfig.

• We also plan to work with communities to set guidelines for Community Configuration use. For example, on larger wikis, there may be a norm in place that a discussion is started and consensus is reached before a configuration change is made. Admins may then add a link or explanation in their Edit summary to explain the change.

• When setting up Automoderator configuration you might decide on which ranges are appropriate. Perhaps pre-determine some generally healthy ranges like rather than leave it totally open. Something like: High (catches the most vandalism, but will trigger more false positives), Medium, Low (will catch some vandalism and limit false positives).

• The new Community configuration form designs are still underway, but will likely include options to link out to metrics that can help a community monitor and see the impact of a configuration change. The metrics / monitoring won't be built into Configuration, but for example, Growth's Suggested Edits configuration could link out to this metrics dashboard so communities have the ability to see current feature usage and the impact of a change after they make it.

Does that help?

In T349374#9321472, @KStoller-WMF wrote:

While thinking about potential risks/failures of Automoderator, we found that if the revert risk range is changed drastically, the tool might be rendered useless or overly aggressive in removing edits.

That's definitely a legit concern. It's one of the risks we've discussed when conducting a Community Configuration pre-mortem. I don't think I have a perfect solution, but here are some ideas that might help:

• As @Strainu mentions, you could consider limiting config access to a more priviledged user group. Currently the Growth team limits Configuration changes to admins. Anyone can view the configuration, but only admins can actually edit the form.

• Admins (or any account holders) can watch the associated config pages, so there is some built-in oversight of changes. Example for English WIkipedia's GrowthExperimentsConfig.

• We also plan to work with communities to set guidelines for Community Configuration use. For example, on larger wikis, there may be a norm in place that a discussion is started and consensus is reached before a configuration change is made. Admins may then add a link or explanation in their Edit summary to explain the change.

• When setting up Automoderator configuration you might decide on which ranges are appropriate. Perhaps pre-determine some generally healthy ranges like rather than leave it totally open. Something like: High (catches the most vandalism, but will trigger more false positives), Medium, Low (will catch some vandalism and limit false positives).

• The new Community configuration form designs are still underway, but will likely include options to link out to metrics that can help a community monitor and see the impact of a configuration change. The metrics / monitoring won't be built into Configuration, but for example, Growth's Suggested Edits configuration could link out to this metrics dashboard so communities have the ability to see current feature usage and the impact of a change after they make it.

Does that help?

Thanks for the ideas! Those sound great, including the ranges for the revert risk configuration. As we advance in building Automoderator, maybe we can touch base with the Growth team to see if Feature 2 might also be a possibility in a not-so-distant future.

Thinking about how this might work alongside an ores-enhanced recent changes page with language-agnostic-revert-risk, I'd suggest that we use a selection for caution level
We could align the language and values to ores when it's configured so that there we can leverage existing tooling for users to see the impact of changing the value. Want to see which edits would get caught at various settings? go to recent changes and filter by risk level. Here's an example of the filter options for another model that has multiple thresholds set:

In T349374#9338025, @jsn.sherman wrote:

Thinking about how this might work alongside an ores-enhanced recent changes page with language-agnostic-revert-risk, I'd suggest that we use a selection for caution level
We could align the language and values to ores when it's configured so that there we can leverage existing tooling for users to see the impact of changing the value. Want to see which edits would get caught at various settings? go to recent changes and filter by risk level. Here's an example of the filter options for another model that has multiple thresholds set:

What do you think about this @aishwaryavardhana?

This is a good point & nice idea. Do we know what the fate of the ORES extension is with regards to moving to revert risk for these scores (or not)?

EDIT: I see there's a Slack thread about the extension, so I've just caught up on that.