Page MenuHomePhabricator

Make list of features and locations of ORES Review Tool for handoff
Closed, ResolvedPublic

Description

Collaboration-Team-Triage will take over user-facing stuff and put it into a new extension and all of the basic MW platform stuff will ideally be all that remains in Extension:ORES. We need to list out the user-facing functionality to maintain parity with. When all features are reimplemented, handoff is complete, and new Extension:ORES deployments won't substantially affect UX.

We can negotiate some of these features away if needed.


Inventory of existing features:

  • Score filtering is available on changes lists including Special:Contributions, watchlists, and recent changes.
  • Changes list can be filtered and highlighted by whether a change falls into each classification.
  • Scoring API is queried efficiently when filtering a changes feed.
  • Hide problem edits by default, according to preference. (Negotiable? Currently there are separate preferences for recent changes and watchlist. Special:Contributions does not have the option.)
  • Patrolled changes should have highlighting cleared and be filtered along with "very likely good", regardless of ORES scores.
  • Whether damaging and good faith filters are shown is dependent on whether those models are supported for the current wiki.

Current agreements:

  • Extension:ORES will be split so that infrastructure and UX code are in different directories and namespaces, to simplify ownership.
  • Special:Contributions will continue to use the basic filter style, and we need to figure out how preferences will interact with that.
  • Existing ORES preferences can be deprecated once RC filters is fully deployed.
  • We're punting on "patrolled" pages interacting with RC filters. This should be mentioned in RC filters and ORES release notes.
  • Scoring Platform team will be able to deploy the extension with the UI disabled, and Collaboration can throw the final switch.

Event Timeline

Halfak created this task.Jun 14 2017, 7:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2017, 7:01 PM
Halfak reassigned this task from Halfak to Ladsgroup.Jun 15 2017, 2:30 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJun 15 2017, 2:30 PM
awight updated the task description. (Show Details)Jun 20 2017, 11:59 PM
awight awarded a token.Jul 3 2017, 4:45 PM

OK. So.

For the UI, we have:

  • Special:Recentchanges (Filtering, highlighting)
  • Special:Watchlist (Highlighting)
  • &action=history (Highlighting)
  • api.php "oresscores"

For the backend, we have a maintenance script. We insert jobs in the job queue to make sure that ORES scores get stored in a table. We have that couple of tables (ores_model and ores_classification) and a set of indexes on those tables.

awight added a subscriber: awight.Jul 10 2017, 11:27 PM

Neat! I just saw that the RC filters have been implemented in Extension:ORES, which makes migration simple. I'll report back here with the "legacy" ORES features as I understand them.

Ladsgroup removed Ladsgroup as the assignee of this task.Jul 11 2017, 1:04 PM
Ladsgroup removed a project: User-Ladsgroup.
awight claimed this task.Jul 11 2017, 4:48 PM
awight triaged this task as High priority.
awight updated the task description. (Show Details)Jul 11 2017, 7:02 PM

Some questions/remarks:

  • Do we think splitting the ORES extension into an infrastructure extension (owned by Scoring Platform) and a UI extension (owned by Collaboration) is a good idea? I don't mind terribly either way, but extension splits can be a bit messy. On the other hand, it would make the division of responsibilities (both in terms of teams and in terms of technology) clearer.
    • @Mattflaschen-WMF suggests we could do this as a pseudo-extension or sub-extension within the extension
  • We only plan to offer very basic score filtering on the Contributions page (only a binary toggle for "hide probably good edits"); the richer multi-level filters (very likely good, likely good, etc) are currently only available on Recentchanges and will in the near-ish future also be available on Watchlist.
  • There are presently no preferences to control any of the default filters on the Contributions page, so the absence of a preference to control the "hide probably good edits" filter follows that pattern
    • Note that the current prefs are only for hiding non-problem edits, not for hiding problem edits, I think this is a typo/thinko in the task description
  • We may in the future get rid of the preferences for default filters on RC+Watchlist in favor of the "saved queries" feature (not certain about how we plan to do this yet); if and when this happens, the ORES-related prefs would be treated the same as the others (e.g. "hide minor edits by default")
  • Treatment of patrolled changes isn't totally optimal right now (they're not treated as "very likely good"), but I do believe they are hidden when "hide probably good edits" is used and not marked with an r. We can fix this up where needed.
  • Where do we think the (existing and new) threshold lookup code should go? There is existing code that uses model_info to resolve dynamically specified thresholds like recall_at_precision(precision=0.995), and we will need new code to work with @Halfak 's new threshold API. Do we want to put this on the "infrastructure" side of the fence so that future applications can use it, even though RCFilters is the only consumer of it right now?

We Are not currently planning to work on Contributions. So if we've somehow interfered with how ORES was working on that page or is meant to work on that page, please give me the details and we'll figure out how to fix it.

Thanks for the detailed notes!

Some questions/remarks:

  • Do we think splitting the ORES extension into an infrastructure extension (owned by Scoring Platform) and a UI extension (owned by Collaboration) is a good idea? I don't mind terribly either way, but extension splits can be a bit messy. On the other hand, it would make the division of responsibilities (both in terms of teams and in terms of technology) clearer.
    • @Mattflaschen-WMF suggests we could do this as a pseudo-extension or sub-extension within the extension

I had the same thought, that we do something neat with submodules when we're ready. This also strikes me as a "nice to have" that can wait for a later phase in the transition...

  • We only plan to offer very basic score filtering on the Contributions page (only a binary toggle for "hide probably good edits"); the richer multi-level filters (very likely good, likely good, etc) are currently only available on Recentchanges and will in the near-ish future also be available on Watchlist.

@Ladsgroup @Halfak ^ How do we judge whether this is going to miff our users?

  • There are presently no preferences to control any of the default filters on the Contributions page, so the absence of a preference to control the "hide probably good edits" filter follows that pattern
    • Note that the current prefs are only for hiding non-problem edits, not for hiding problem edits, I think this is a typo/thinko in the task description
  • We may in the future get rid of the preferences for default filters on RC+Watchlist in favor of the "saved queries" feature (not certain about how we plan to do this yet); if and when this happens, the ORES-related prefs would be treated the same as the others (e.g. "hide minor edits by default")

That sounds right to me. IMO that means we're already at parity, and will move beyond that to something better.

My instinct is to announce the patrolled "regression" and get some feedback from users before re-implementing.

  • Where do we think the (existing and new) threshold lookup code should go? There is existing code that uses model_info to resolve dynamically specified thresholds like recall_at_precision(precision=0.995), and we will need new code to work with @Halfak 's new threshold API. Do we want to put this on the "infrastructure" side of the fence so that future applications can use it, even though RCFilters is the only consumer of it right now?

IMO, +1 this is infrastructure.

Also, Roan is right to say we're moving in a direction where "Saved filters" (which includes the ability to save a Default filter set) on Watchlist and RC page will replace many of the existing Preferences for these pages.

E.g., preferences like "don't show minor edits" no longer make sense, because users can just add that to their defaults, right on the page. And the same is true for ORES settings like "hide probably good edits." Users can just add the relevant filter to their default settings. I'd argue it's a superior model, because it lets you adjust your settings interactively, right on the page, where you can see what kind of effect the setting is having on results.

  • @Mattflaschen-WMF suggests we could do this as a pseudo-extension or sub-extension within the extension

I had the same thought, that we do something neat with submodules when we're ready. This also strikes me as a "nice to have" that can wait for a later phase in the transition...

We could even have two separate directories/namespaces in the same repo.

  • We only plan to offer very basic score filtering on the Contributions page (only a binary toggle for "hide probably good edits"); the richer multi-level filters (very likely good, likely good, etc) are currently only available on Recentchanges and will in the near-ish future also be available on Watchlist.

@Ladsgroup @Halfak ^ How do we judge whether this is going to miff our users?

For the record: the Contributions page doesn't have any rich filters (generally speaking, not just for ORES) at present, only binary ones, and we have no plans to upgrade it,

My instinct is to announce the patrolled "regression" and get some feedback from users before re-implementing.

Yeah that's probably a good idea. How exactly patrolledness should interact with damaging/goodfaith filters is not totally obvious especially when considering rich/nonbinary filters. We kind of just ignored / forgot about that issue in RCFilters, and while we will of course not break any of the existing behavior in classic ORES's binary filter, I don't think we have any great ideas for how to deal with it right now (at least I don't).

awight updated the task description. (Show Details)Jul 12 2017, 4:57 PM

@Catrope @Halfak @Ladsgroup
When you have a minute, please double-check the "current agreements" in the task description, and let me know if we are in fact agreed.

awight updated the task description. (Show Details)Jul 17 2017, 4:28 PM
awight closed this task as Resolved.Jul 24 2017, 4:59 PM

This is complete from our perspective. @jmatazzoni has a plan for the preferences, which we'll discuss today. I'll record our talk in T167910.