Page MenuHomePhabricator

Metrics: TwoColumnEditConflictMerge
Closed, ResolvedPublic0 Estimated Story Points

Description

Need: To improve 2ColConf we would like to know how it is used and how it does compared to the current conflict resolvent page

GENERAL:

  • All following metrics should be collected together for each conflict and matchable to each other. Events should be in the order they happened in. So the format ideally would be:
  • If applicable, metrics should be collected for the old UI as well as for the 2ColConf
  • Infos on the conflict (for old and new version)
    • Namespace of page the conflict happened on (Q: "How does behavior on talkpages differ from articles?") | should be a STRING
    • Amount of conflicting text (as characters or words) (Q: "How does behavior differ between huge or small conflicts?") | should be a INTEGER
  • User behavior: (only partly applicable)
    • Text input actions to the Editor (could be just the amount of keypresses or +- editied) (Q: "Did people actually use the possibility to do edits in the editor?") | should be an INTEGER
    • Filters used (Q: "Do people use the filter options? Do they always switch to a particular one (which could be default then)"). Should be an ARRAY with STRING or possibly an ARRAY with OBJECTS with STRING and relative timestamp as INTEGER
  • Leaving the page : Do they… (for old and new version) | should be a STRING
    • Use the save button (Which would indicate the feature helped them)
    • Do they use the Cancel button
    • Do they navigate away via the browser (Back, Tab Close etc.)

Outcome

We've implemented this feature, and here is an accounting of the data we record:

  • EventLogging: EditConflict schema.
    • User ID, and name or IP address
    • Calculated user attributes: is anonymous, edit count
    • Page ID, namespace, and title
    • Revision ID of the page's latest, conflicting "other" revision
  • EventLogging: TwoColConflictConflict schema
    • Was this the legacy or TwoColConflict workflow?
    • Page namespace
    • Revision IDs for the previous edit and conflicting "other" edit
    • Literal wikitext "your" edit as entered into the edit text box
  • Statsd MediaWiki-core "edit.failures.conflict"
    • Count of edit failures due to a conflict. Segmented by namespace (some namespaces are omitted) or user edit count bucket
    • Resolved conflict count, segmented by namespace or user edit count bucket
  • Statsd MediaWiki.TwoColConflict
    • Pageload count (can this be more than the number of conflicts?). Segmented by namespace, and by user edit count bucket
    • Conflicts resolved, segmented by namespace or user edit count bucket

This matches our original requirements as follows:

  • All following metrics should be [...] matchable to each other.
    • This is possible for the EventLogging data, but not for the Statsd segments.
  • If applicable, metrics should be collected for the old UI as well as for the 2ColConf
  • Namespace of page the conflict happened on
    • Not always joinable with every other metric
    • Currently an integer, but the string label could be calculated if desired.
  • Amount of conflicting text (as characters or words)
    • We have logged the raw text, which if untruncated gives us enough information to backfill. But we do not record the conflict size, yet.
  • Text input actions to the Editor (could be just the amount of keypresses or +- edited)
  • Filters used
  • Use the save button
    • We can assume that a successful conflict = 1 save button click.
  • Do they use the Cancel button
    • Not explicitly tracked, but we can assume (1 - percent resolved) gives us the number of abandoned conflict resolutions.
  • Do they navigate away via the browser (Back, Tab Close etc.)

Event Timeline

Restricted Application added a project: User-Addshore. · View Herald TranscriptFeb 14 2017, 4:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore moved this task from Unsorted 💣 to Next on the User-Addshore board.Mar 3 2017, 7:06 PM
Addshore removed Addshore as the assignee of this task.Mar 21 2017, 10:13 AM
Addshore added a subscriber: Addshore.

Going to un-assign myself until I start work here, as in theory other people could pick this up...

Addshore moved this task from Next to Unsorted 💣 on the User-Addshore board.Jun 29 2017, 10:40 AM
Addshore changed the task status from Open to Stalled.Aug 31 2017, 10:57 AM

To be reviewed if it's still relevant.

thiemowmde set the point value for this task to 3.Feb 4 2020, 2:27 PM
thiemowmde added a subscriber: thiemowmde.

TODO here:

  • Check if we are already measuring some of the stuff, and which.
  • Decide if we need to measure the rest.
awight changed the task status from Stalled to Open.Feb 6 2020, 12:43 PM
awight claimed this task.
awight moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2020-02-04 board.
awight updated the task description. (Show Details)Feb 6 2020, 2:10 PM
awight updated the task description. (Show Details)Feb 6 2020, 2:22 PM

See the task description for a summary of what data we collect.

Many of the original criteria are unsatisfied or only partially satisfied, but I suggest we don't collect any more data without an explicit motivation.

Change 570703 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TwoColConflict@master] Remove documentation for missing metrics

https://gerrit.wikimedia.org/r/570703

Change 570703 merged by jenkins-bot:
[mediawiki/extensions/TwoColConflict@master] Remove documentation for missing metrics

https://gerrit.wikimedia.org/r/570703

thiemowmde reassigned this task from awight to Lena_WMDE.Feb 18 2020, 12:48 PM
thiemowmde moved this task from Demo to Done on the WMDE-QWERTY-Sprint-2020-02-04 board.
thiemowmde moved this task from Done to Review on the WMDE-QWERTY-Sprint-2020-02-04 board.
thiemowmde added a subscriber: awight.

@Lena_WMDE needs PM review of the missing metrics.

awight changed the point value for this task from 3 to 0.
Lena_WMDE closed this task as Resolved.Mar 17 2020, 12:32 PM

Marking this as resolved as relevant oustanding metrics are now being tracked through: T245420 and T245722.

Lena_WMDE moved this task from Demo to Done on the WMDE-QWERTY-Sprint-2020-03-04 board.
Lena_WMDE moved this task from Done to Demo on the WMDE-QWERTY-Sprint-2020-02-04 board.