Page MenuHomePhabricator

Failure Types by User Type {lion}
Closed, ResolvedPublic

Description

We would love to combine what we've learned from failure types [1] and user types [2] into one graph so we can see how they're related. Perhaps a stacked bar chart with user types on the X axis and failure type proportions in the stacked bar. Please feel free to suggest other directions in the comments.

Dashboard displaying the data: https://edit-analysis.wmflabs.org/compare/

Event Timeline

Milimetric claimed this task.
Milimetric raised the priority of this task from to Needs Triage.
Milimetric updated the task description. (Show Details)
Milimetric added a project: Analytics-Kanban.
Milimetric moved this task to Next Up on the Analytics-Kanban board.
Milimetric added a subscriber: Milimetric.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 27 2015, 8:25 PM
mforns added a subscriber: mforns.EditedMar 5 2015, 2:52 PM

I wrote a query for this and got to the following results from 2014-12-03 to 2015-02-01:

+-------------+------------------------+------------+
| usertype    | failuretype            | percentage |
+-------------+------------------------+------------+
| 1-anonymous | editConflict           |     6.2265 |
| 1-anonymous | editPageDeleted        |     0.4512 |
| 1-anonymous | extensionAbuseFilter   |    28.1972 |
| 1-anonymous | extensionCaptcha       |    28.0410 |
| 1-anonymous | extensionSpamBlacklist |     0.9402 |
| 1-anonymous | responseEmpty          |     1.9213 |
| 1-anonymous | responseUnknown        |    33.7264 |
| 1-anonymous | userBadToken           |     0.4962 |
| 2-firstEdit | editConflict           |     2.1245 |
| 2-firstEdit | editPageDeleted        |     0.1808 |
| 2-firstEdit | extensionAbuseFilter   |    39.2474 |
| 2-firstEdit | extensionCaptcha       |    42.6828 |
| 2-firstEdit | extensionSpamBlacklist |     2.1697 |
| 2-firstEdit | responseEmpty          |     0.4859 |
| 2-firstEdit | responseUnknown        |     3.6388 |
| 2-firstEdit | userBadToken           |     3.8648 |
| 2-firstEdit | userNewUser            |     5.6052 |
| 3-over1     | editConflict           |     6.2647 |
| 3-over1     | editPageDeleted        |     4.2156 |
| 3-over1     | extensionAbuseFilter   |    25.0316 |
| 3-over1     | extensionCaptcha       |    46.7323 |
| 3-over1     | extensionSpamBlacklist |     1.6971 |
| 3-over1     | responseEmpty          |     0.4423 |
| 3-over1     | responseUnknown        |    11.1211 |
| 3-over1     | userBadToken           |     4.4503 |
| 3-over1     | userNewUser            |     0.0451 |
| 4-over5     | editConflict           |     8.0444 |
| 4-over5     | editPageDeleted        |     1.9631 |
| 4-over5     | extensionAbuseFilter   |    17.6038 |
| 4-over5     | extensionCaptcha       |    42.2597 |
| 4-over5     | extensionSpamBlacklist |     3.5848 |
| 4-over5     | responseEmpty          |     1.0456 |
| 4-over5     | responseUnknown        |    11.2131 |
| 4-over5     | userBadToken           |    14.1577 |
| 4-over5     | userNewUser            |     0.1280 |
| 5-over100   | editConflict           |    14.3394 |
| 5-over100   | editPageDeleted        |     1.3150 |
| 5-over100   | extensionAbuseFilter   |     5.4477 |
| 5-over100   | extensionCaptcha       |     2.0664 |
| 5-over100   | extensionSpamBlacklist |     6.0739 |
| 5-over100   | responseEmpty          |     4.2580 |
| 5-over100   | responseUnknown        |    14.0263 |
| 5-over100   | userBadToken           |    52.3482 |
| 5-over100   | userNewUser            |     0.1252 |
| 6-over1000  | editConflict           |    13.9556 |
| 6-over1000  | editPageDeleted        |     0.3556 |
| 6-over1000  | extensionAbuseFilter   |     1.5111 |
| 6-over1000  | extensionSpamBlacklist |     2.4889 |
| 6-over1000  | responseEmpty          |     3.4667 |
| 6-over1000  | responseUnknown        |    13.8667 |
| 6-over1000  | userBadToken           |    63.2889 |
| 6-over1000  | userNewUser            |     1.0667 |
+-------------+------------------------+------------+

I'll be working now in the visualization.

mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.Mar 5 2015, 3:14 PM
mforns claimed this task.Mar 9 2015, 4:28 PM
mforns set Security to None.

Change 195436 had a related patch set uploaded (by Mforns):
Query and visualization for failure vs user analysis

https://gerrit.wikimedia.org/r/195436

kevinator triaged this task as Normal priority.Mar 12 2015, 3:24 PM

Change 195436 merged by Milimetric:
Query and visualization for failure vs user analysis

https://gerrit.wikimedia.org/r/195436

Note: this and all other analyses done this quarter for the Edit schema are no longer valid. This is because we have an unknown number of invalid events in our logs that affect the outcomes in unknown ways. We have talked about cleaning up this data and James F. has stated that it's a high priority for him. As of right now, we don't have enough information to know if it's possible to fix all the data, or enough data to regain confidence in it.

kevinator renamed this task from Failure Types by User Type to Failure Types by User Type {lion}.Apr 27 2015, 4:18 PM
mforns moved this task from Paused to In Progress on the Analytics-Kanban board.May 19 2015, 7:25 PM

Change 212276 had a related patch set uploaded (by Mforns):
Productionize failure types by user type

https://gerrit.wikimedia.org/r/212276

Change 212276 merged by Milimetric:
Productionize failure types by user type

https://gerrit.wikimedia.org/r/212276

Change 214036 had a related patch set uploaded (by Mforns):
Add stacked bars component to compare layout

https://gerrit.wikimedia.org/r/214036

Change 214036 merged by Milimetric:
Add stacked bars component to compare layout

https://gerrit.wikimedia.org/r/214036

kevinator updated the task description. (Show Details)Jun 8 2015, 3:39 PM
kevinator added a subscriber: kevinator.EditedJun 8 2015, 3:51 PM

Is there a way to make the "over5" appear as the 4th column and not the last?

Maybe this should be the names

a-anon
b-1 edit
c-1+
d-5+
e-100+
f-1000+

Also I run into a bug:

  1. using chrome on Mac OS X
  2. load https://edit-analysis.wmflabs.org/compare/
  3. click "Both"
  4. Open Developer Tools in Chrome

Result: The wikitext bar chart never loads and I see this error in the log:

Mon, 08 Jun 2015 18:48:24 GMT https://edit-analysis.wmflabs.org/compare/scripts.js?v=8e691e4:12

Uncaught TypeError: Unable to process binding "if: function (){return data().showAB.a }"
Message: Cannot read property 'showAB' of undefined

Feel free to ping me if you need more info

mforns added a comment.Jun 8 2015, 7:13 PM

@kevinator

Is there a way to make the "over5" appear as the 4th column and not the last?

Yes, as this is a generic solution. columns are ordered alphabetically by name. So, to reorder them, we'll need to modify the query to output ordering prefixes, like you suggested.
I'll do that in short.

Also I run into a bug:
...
Uncaught TypeError: Unable to process binding "if: function (){return data().showAB.a }"

The exact code printed by the error doesn't exist any more in the last version of Dashiki, I guess that is a caching problem in some layer?

Change 216982 had a related patch set uploaded (by Mforns):
Add ordering to Failure types query

https://gerrit.wikimedia.org/r/216982

Marcel, I think the stacked bars need the same "if: data" binding as the compare-sunburst.html code uses.

Change 217823 had a related patch set uploaded (by Mforns):
Support numeric prefixes in stacked-bar columns

https://gerrit.wikimedia.org/r/217823

Change 218910 had a related patch set uploaded (by Mforns):
Comments out failure_types_by_user_type report

https://gerrit.wikimedia.org/r/218910

Change 218910 merged by Mforns:
Comments out failure_types_by_user_type report

https://gerrit.wikimedia.org/r/218910

Change 217823 merged by Milimetric:
Support numeric prefixes in stacked-bar columns

https://gerrit.wikimedia.org/r/217823

Change 216982 merged by Mforns:
Add ordering to Failure types query

https://gerrit.wikimedia.org/r/216982

Change 218940 had a related patch set uploaded (by Mforns):
Uncomment failure_types_by_user_type report

https://gerrit.wikimedia.org/r/218940

Change 218940 merged by Mforns:
Uncomment failure_types_by_user_type report

https://gerrit.wikimedia.org/r/218940

Change 219044 had a related patch set uploaded (by Mforns):
Fix stacked-bar chart

https://gerrit.wikimedia.org/r/219044

Change 219044 merged by Milimetric:
Fix stacked-bar chart

https://gerrit.wikimedia.org/r/219044

Milimetric moved this task from Done to In Code Review on the Analytics-Kanban board.
kevinator closed this task as Resolved.Jun 19 2015, 10:46 PM