Page MenuHomePhabricator

Prepare metrics to answer usability questions
Closed, ResolvedPublic1 Estimated Story Points

Description

We want to be able to answer these questions going forward,

  1. Are 'non-conflicting' edits causing the total number of chunks to be so large that is it easier to abandon?
    • Track total number of chunks and correlation to success/abandonment
    • What percentage of chunks have direct conflicts/overlapping edits in the same paragraph?
  2. Do users understand that they can edit?
    • Track if editing before submission, track 'edit other' and 'edit yours' separately
  3. What is causing abandonment?
    • When is it happening? How long are people spending on the page / how complete is the form?

Ensure that we have metrics to produce this data, and build the postprocessing ahead of when we'll need it.

Resources

The numbers below are calculated using this join script and this notebook.

Results

We have exit data beginning April 6th:

+-----------+-----+-------+
|exit_action|count|percent|
+-----------+-----+-------+
|     cancel|   36|    2.2|
|    unknown|  888|   53.3|
|       save|  742|   44.5|
+-----------+-----+-------+

Registered user exits
+-----------+-----+----------+
|exit_action|count|percentage|
+-----------+-----+----------+
|     cancel|   34|       2.1|
|    unknown|  858|      54.2|
|       save|  690|      43.6|
+-----------+-----+----------+

Anonymous user exits
+-----------+-----+----------+
|exit_action|count|percentage|
+-----------+-----+----------+
|     cancel|    2|       2.4|
|    unknown|   30|      35.7|
|       save|   52|      61.9|
+-----------+-----+----------+

Average experience level of users according to outcome
+-----------+--------------+-------------+
|exit_action|mean_editcount|std_editcount|
+-----------+--------------+-------------+
|     cancel|         14792|        29569|
|    unknown|         38723|        55944|
|       save|         29938|        51139|
+-----------+--------------+-------------+

Total elapsed time spent in the conflict workflow, grouped by outcome. I'm very suspicious of how short the times are, I'll verify later using another data source. There are negative values and other smells.

+-----------+--------------+-------------+-------------+-------------+
|exit_action|mean_elapsed_s|std_elapsed_s|min_elapsed_s|max_elapsed_s|
+-----------+--------------+-------------+-------------+-------------+
|     cancel|         139.0|        499.0|          -80|         3016|
|    unknown|          98.0|        706.0|         -211|        16267|
|       save|          86.0|        953.0|        -2876|        25658|
+-----------+--------------+-------------+-------------+-------------+

Overlapping chunk summary. These are the count of server-calculated actual conflicts, not related to the number of UI rows.

+-------+-----------------+------------------+------------------+
|summary|   user_editcount|   conflict_chunks|    conflict_chars|
+-------+-----------------+------------------+------------------+
|  count|              443|               443|               443|
|   mean|27603.18284424379|1.5914221218961626| 964.8081264108353|
| stddev|55763.25081415555|2.1213456222435263|1452.0718079693656|
|    min|                0|                 1|                 0|
|    max|           519415|                25|             14600|
+-------+-----------------+------------------+------------------+

+-----------+-----+-----------------------+----------------------+----------------------+---------------------+
|exit_action|count|mean_overlapping_chunks|std_overlapping_chunks|mean_overlapping_chars|std_overlapping_chars|
+-----------+-----+-----------------------+----------------------+----------------------+---------------------+
|     cancel|   19|                    1.6|                   2.5|                 697.4|                559.7|
|    unknown|  197|                    1.9|                   2.9|                1134.2|               1832.6|
|       save|  227|                    1.3|                   1.0|                 840.1|               1067.5|
+-----------+-----+-----------------------+----------------------+----------------------+---------------------+

TODO:

Event Timeline

awight created this task.Apr 7 2020, 1:47 PM
Restricted Application added a project: archived--TCB-Team. · View Herald TranscriptApr 7 2020, 1:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
awight updated the task description. (Show Details)Apr 7 2020, 1:50 PM

Change 587499 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TwoColConflict@master] [DNM] Log EditStepAttempt at 100% during conflicts

https://gerrit.wikimedia.org/r/587499

awight added a comment.Apr 8 2020, 9:53 AM

https://gerrit.wikimedia.org/r/587499

@Lena_WMDE This is a bookmark for our discussion about sampling. With the above line of code, we get 100% of edit tracking rather than the default 6.25% sampling.

awight claimed this task.Apr 8 2020, 8:00 PM
awight moved this task from Backlog to Tickets in sprint on the Two-Column-Edit-Conflict-Merge board.
awight set the point value for this task to 5.
awight added a comment.Apr 8 2020, 8:03 PM

I moved this into the sprint for transparency, but we don't have to do all "5 points" right now, there are incremental benefits.

Change 588409 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TwoColConflict@master] Include session token in exit event

https://gerrit.wikimedia.org/r/588409

This is getting really messy without a real key to join on. I can only do heuristic things with the page_title... There are two sticky IDs on the client side which we could use, but both are tricky: editing_session_id would be perfect, but it should remain private to editor code. session_token is meant for public access, but is only available on the client so we would have to include it in the TwoColConflictExit event (attaching patch now).

Change 588409 merged by jenkins-bot:
[mediawiki/extensions/TwoColConflict@master] Include session token in exit event

https://gerrit.wikimedia.org/r/588409

awight updated the task description. (Show Details)Apr 14 2020, 2:51 PM
awight moved this task from Sprint Backlog to Doing on the WMDE-QWERTY-Sprint-2020-04-01 board.

Starting to put results in the task description.

awight updated the task description. (Show Details)Apr 14 2020, 3:11 PM
awight updated the task description. (Show Details)Apr 14 2020, 3:17 PM
awight updated the task description. (Show Details)Apr 14 2020, 4:07 PM
awight updated the task description. (Show Details)Apr 14 2020, 4:13 PM
awight updated the task description. (Show Details)Apr 14 2020, 9:38 PM

There's a little bit more listed in the TODO at the end of the description, but the data sources aren't available yet. I'll pause this work until feedback and unblocking on data.

A bit of bad news, the column selection patch won't be deployed until 28 April, unless we use the SWAT process.

awight updated the task description. (Show Details)Apr 15 2020, 9:49 AM
WMDE-Fisch changed the point value for this task from 5 to 1.Apr 15 2020, 10:39 AM
WMDE-Fisch moved this task from Sprint Backlog to Demo on the WMDE-QWERTY-Sprint-2020-04-15 board.

Change 587499 merged by jenkins-bot:
[mediawiki/extensions/TwoColConflict@master] Configurable EditStepAttempt oversampling for conflicts

https://gerrit.wikimedia.org/r/587499

Change 591016 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/extensions/TwoColConflict@wmf/1.35.0-wmf.28] Configurable EditStepAttempt oversampling for conflicts

https://gerrit.wikimedia.org/r/591016

Change 591017 had a related patch set uploaded (by Awight; owner: Awight):
[operations/mediawiki-config@master] Temporarily enable event oversampling for conflicts

https://gerrit.wikimedia.org/r/591017

We've decided to increase the sampling to 100% for a period of time (likely until we move TwoColConflict to default on all wikis, or shortly after). This will allow us to review and confirm our assumptions about product decisions we have made or would like to make before full default.

Change 591016 merged by jenkins-bot:
[mediawiki/extensions/TwoColConflict@wmf/1.35.0-wmf.28] Configurable EditStepAttempt oversampling for conflicts

https://gerrit.wikimedia.org/r/591016

Change 591017 merged by jenkins-bot:
[operations/mediawiki-config@master] Temporarily enable event oversampling for conflicts

https://gerrit.wikimedia.org/r/591017

Mentioned in SAL (#wikimedia-operations) [2020-04-20T11:25:22Z] <awight@deploy1001> Synchronized php-1.35.0-wmf.28/extensions/TwoColConflict: SWAT: [[gerrit:591016|Configurable EditStepAttempt oversampling for conflicts (T249616)]] (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2020-04-20T11:27:27Z] <awight@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:591017|Temporarily enable event oversampling for conflicts (T249616)]] (duration: 01m 00s)

awight closed this task as Resolved.May 13 2020, 9:38 AM
awight moved this task from Demo to Done on the WMDE-QWERTY-Sprint-2020-04-29 board.