Page MenuHomePhabricator

Analyze results of stage 0 page previews test
Closed, ResolvedPublic

Description

Refer to https://www.mediawiki.org/wiki/Beta_Features/Hovercards/2016_A/B_Tests and T139319: Analyze results of Hovercards A/B test - Hungarian for metrics (plus additional metrics)

Begin analyzing data on July 19 (1 week after the duplicates issue was fixed, per T170018#3424998; with all other known bugs having been resolved earlier, except T167365: Check whether a 'disabled' event is sent when the user disables the feature and fix if the event is not sent)

Repeat analysis done for T139319: Analyze results of Hovercards A/B test - Hungarian:

  • cards per page
  • cards per session
  • disable rates
  • back button
  • pageviews (on vs off)

Related Objects

StatusAssignedTask
ResolvedDereckson
ResolvedJdlrobson
Resolvedovasileva
Resolvedovasileva
Resolved Tbayer
Resolved Tbayer
Resolved Tbayer
Resolved Tbayer
Resolved Tbayer
Resolvedphuedx
ResolvedJdlrobson
Resolved Tbayer
Declined Tbayer
Declined Tbayer
Resolvedphuedx
Resolved Tbayer
Resolvedpmiazga
Resolved Tbayer
Resolvedphuedx
Resolvedpmiazga
Resolvedphuedx
Resolvedovasileva
Resolvedphuedx
Resolvedphuedx

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2017, 11:31 AM
ovasileva triaged this task as High priority.Feb 16 2017, 4:38 PM
ovasileva moved this task from Incoming to Product Owner Backlog on the Readers-Web-Backlog board.

@ovasileva this is high priority but hasn't happened since April.. or has it and we just forgot to update the card? cc @Tbayer

@Jdlrobson - it is indeed still high priority as it is a blocker for any future deployment. Adding it as a subtask to the deployment task and marking as stalled due to various eventlogging bugs.

Tbayer claimed this task.EditedJun 7 2017, 10:53 PM

@Jdlrobson: Yeah, we haven't been very disciplined about updating the status of this task and tracking the blocking EL bugs here; that conversation has been happening elsewhere. I'll add some right now.

Tbayer updated the task description. (Show Details)Jul 15 2017, 1:57 AM
Tbayer moved this task from Triage to Blocked on the Reading-analysis board.Jul 16 2017, 7:16 PM
Tbayer changed the task status from Stalled to Open.Jul 16 2017, 7:33 PM
Tbayer updated the task description. (Show Details)Jul 21 2017, 5:06 AM
Tbayer updated the task description. (Show Details)Jul 21 2017, 5:08 AM

(work log)

I'm repeating last year's analysis for the final report where we used a daily time series to assess the variation in the metrics, e.g. to compare the number of pageviews per session between previews ON/OFF. However, it looks like the decreased sample rate combined with our decision to shorten the timespan to one week (instead of six weeks as last time), and perhaps also the sampling fix made recently in connection with the RelatedArticles issue, is creating a problem here, with too few sessions on all but the largest wikis in the test (ruwiki, itwiki) to make a meaningful recreation of last year's chart, and even there too sparse to be confident (50-100/day in control on ruwiki).

Will discuss an increase of the sampling rate - the schema's event rate is still well in the safe zone currently.

SELECT
wiki,
LEFT(timestamp, 8) AS date,
event_popupEnabled AS popupEnabled,
COUNT(DISTINCT event_pageToken)/COUNT(DISTINCT event_sessionToken) AS page_tokens_per_session,
COUNT(DISTINCT event_sessionToken) AS sessions
FROM log.Popups_16364296
WHERE wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki') # https://gerrit.wikimedia.org/r/#/c/343675/
AND event_isAnon = 1
AND LEFT(timestamp, 8) >= '20170713'
AND LEFT(timestamp, 8) < '20170721'
AND SUBSTRING( userAgent, INSTR(userAgent,'"browser_family": "')+19, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "')+19), '"' )-1 ) = 'Firefox'
AND event_action = 'pageLoaded'
GROUP BY wiki, date, event_popupEnabled
ORDER BY wiki, date, event_popupEnabled;

+--------+----------+--------------+-------------------------+----------+
| wiki   | date     | popupEnabled | page_tokens_per_session | sessions |
+--------+----------+--------------+-------------------------+----------+
| cawiki | 20170713 |            0 |                  1.0000 |        1 |
| cawiki | 20170713 |            1 |                  1.0000 |        7 |
| cawiki | 20170714 |            0 |                  1.0000 |        2 |
| cawiki | 20170714 |            1 |                  1.0000 |        5 |
| cawiki | 20170715 |            1 |                  1.0000 |        2 |
| cawiki | 20170716 |            1 |                  1.7143 |        7 |
| cawiki | 20170717 |            1 |                  1.1875 |       16 |
| cawiki | 20170718 |            0 |                  1.0000 |        3 |
| cawiki | 20170718 |            1 |                  1.2727 |       11 |
| cawiki | 20170719 |            0 |                  1.0000 |        2 |
| cawiki | 20170719 |            1 |                  1.5833 |       12 |
| cawiki | 20170720 |            1 |                  1.0000 |        8 |
| elwiki | 20170713 |            0 |                  1.0000 |        1 |
| elwiki | 20170713 |            1 |                  1.6667 |       27 |
| elwiki | 20170714 |            0 |                  1.0000 |        2 |
| elwiki | 20170714 |            1 |                  1.2667 |       30 |
| elwiki | 20170715 |            0 |                  1.0000 |        3 |
| elwiki | 20170715 |            1 |                  1.1429 |       28 |
| elwiki | 20170716 |            0 |                  3.5000 |        2 |
| elwiki | 20170716 |            1 |                  1.4500 |       20 |
| elwiki | 20170717 |            0 |                  1.2000 |        5 |
| elwiki | 20170717 |            1 |                  1.3333 |       33 |
| elwiki | 20170718 |            1 |                  2.7931 |       29 |
| elwiki | 20170719 |            0 |                  1.0000 |        2 |
| elwiki | 20170719 |            1 |                  1.4194 |       31 |
| elwiki | 20170720 |            0 |                  3.4000 |        5 |
| elwiki | 20170720 |            1 |                  1.3200 |       25 |
| hewiki | 20170713 |            0 |                  1.0000 |        1 |
| hewiki | 20170713 |            1 |                  1.3333 |        9 |
| hewiki | 20170714 |            0 |                  1.3333 |        3 |
| hewiki | 20170714 |            1 |                  1.4286 |        7 |
| hewiki | 20170715 |            0 |                  1.0000 |        1 |
| hewiki | 20170715 |            1 |                  1.1333 |       15 |
| hewiki | 20170716 |            1 |                  1.3846 |       13 |
| hewiki | 20170717 |            1 |                  1.7500 |       16 |
| hewiki | 20170718 |            0 |                  1.0000 |        1 |
| hewiki | 20170718 |            1 |                  1.2667 |       15 |
| hewiki | 20170719 |            0 |                  1.0000 |        1 |
| hewiki | 20170719 |            1 |                  1.4444 |        9 |
| hewiki | 20170720 |            0 |                  1.0000 |        1 |
| hewiki | 20170720 |            1 |                  2.5455 |       11 |
| huwiki | 20170713 |            0 |                  1.0000 |        4 |
| huwiki | 20170713 |            1 |                  1.4286 |       49 |
| huwiki | 20170714 |            0 |                  1.1429 |        7 |
| huwiki | 20170714 |            1 |                  1.3673 |       49 |
| huwiki | 20170715 |            0 |                  1.3333 |        3 |
| huwiki | 20170715 |            1 |                  1.8364 |       55 |
| huwiki | 20170716 |            0 |                  1.3750 |        8 |
| huwiki | 20170716 |            1 |                  1.5254 |       59 |
| huwiki | 20170717 |            0 |                  1.8000 |        5 |
| huwiki | 20170717 |            1 |                  1.4074 |       54 |
| huwiki | 20170718 |            0 |                  1.3333 |        6 |
| huwiki | 20170718 |            1 |                  1.2361 |       72 |
| huwiki | 20170719 |            0 |                  1.1429 |        7 |
| huwiki | 20170719 |            1 |                  1.5185 |       54 |
| huwiki | 20170720 |            0 |                  1.1667 |        6 |
| huwiki | 20170720 |            1 |                  1.7797 |       59 |
| itwiki | 20170713 |            0 |                  3.0976 |       41 |
| itwiki | 20170713 |            1 |                  1.8577 |      281 |
| itwiki | 20170714 |            0 |                  1.8621 |       29 |
| itwiki | 20170714 |            1 |                  2.2068 |      295 |
| itwiki | 20170715 |            0 |                  1.4783 |       23 |
| itwiki | 20170715 |            1 |                  2.1298 |      208 |
| itwiki | 20170716 |            0 |                  2.0690 |       29 |
| itwiki | 20170716 |            1 |                  2.0564 |      195 |
| itwiki | 20170717 |            0 |                  1.5429 |       35 |
| itwiki | 20170717 |            1 |                  1.6648 |      364 |
| itwiki | 20170718 |            0 |                  1.5667 |       30 |
| itwiki | 20170718 |            1 |                  2.0606 |      330 |
| itwiki | 20170719 |            0 |                  4.7931 |       29 |
| itwiki | 20170719 |            1 |                  1.7982 |      342 |
| itwiki | 20170720 |            0 |                  1.6111 |       36 |
| itwiki | 20170720 |            1 |                  1.6667 |      288 |
| ruwiki | 20170713 |            0 |                  1.5200 |       75 |
| ruwiki | 20170713 |            1 |                  1.3720 |      715 |
| ruwiki | 20170714 |            0 |                  1.5263 |       76 |
| ruwiki | 20170714 |            1 |                  1.6000 |      665 |
| ruwiki | 20170715 |            0 |                  2.4576 |       59 |
| ruwiki | 20170715 |            1 |                  1.7126 |      494 |
| ruwiki | 20170716 |            0 |                  1.2656 |       64 |
| ruwiki | 20170716 |            1 |                  1.5010 |      509 |
| ruwiki | 20170717 |            0 |                  1.5978 |       92 |
| ruwiki | 20170717 |            1 |                  1.5444 |      698 |
| ruwiki | 20170718 |            0 |                  1.7273 |       77 |
| ruwiki | 20170718 |            1 |                  1.6259 |      695 |
| ruwiki | 20170719 |            0 |                  1.4800 |       75 |
| ruwiki | 20170719 |            1 |                  2.2725 |      734 |
| ruwiki | 20170720 |            0 |                  1.3659 |       82 |
| ruwiki | 20170720 |            1 |                  1.4539 |      705 |
+--------+----------+--------------+-------------------------+----------+
89 rows in set (26.73 sec)


Compare last year's chart:

(work log)
Here is the analogous chart for the "previews seen per pageviews" metric (corresponding to this chart from last year, but for a "seen" threshold of 1s instead of 1.5s). This too illustrates that the values vary too much at the current sampling rate to apply our usual time series heuristic, except for perhaps ruwiki and itwiki if we had a longer timespan:


I'm looking at other options including expanding to Chrome and using e.g. bootstrapping instead of the daily time series, but clearly the sample rate has been too low.

SELECT wiki,
LEFT(timestamp, 8) AS date,
COUNT(DISTINCT event_linkInteractionToken)/COUNT(DISTINCT event_sessionToken) AS cards_viewed_per_session,
COUNT(DISTINCT event_linkInteractionToken)/COUNT(DISTINCT event_pageToken) AS cards_viewed_per_page,
COUNT(DISTINCT event_sessionToken) AS sessions,
COUNT(DISTINCT event_pageToken) AS pageviews
FROM log.Popups_16364296
WHERE wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki') # https://gerrit.wikimedia.org/r/#/c/343675/
AND event_isAnon = 1
AND LEFT(timestamp, 8) >= '20170713'
AND LEFT(timestamp, 8) < '20170724'
AND SUBSTRING( userAgent, INSTR(userAgent,'"browser_family": "')+19, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "')+19), '"' )-1 ) = 'Firefox'
AND event_popupEnabled = 1
AND (event_action = 'pageLoaded'
OR (event_totalInteractionTime >= event_perceivedWait + 1000 ))  # 1s instead of 1.5s earlier
GROUP BY wiki, date, event_popupEnabled
ORDER BY wiki, date, event_popupEnabled;
ovasileva updated the task description. (Show Details)Aug 9 2017, 6:50 PM

Update: Here is a look at the disable actions, which are still very rare.

SELECT LEFT(timestamp, 8) AS yearmonthday, COUNT(*) AS disables
FROM log.Popups_16364296
WHERE event_action = 'disabled'
GROUP BY yearmonthday
ORDER BY yearmonthday;

+--------------+----------+
| yearmonthday | disables |
+--------------+----------+
| 20170703     |        1 |
| 20170705     |        2 |
| 20170706     |        4 |
| 20170718     |        1 |
| 20170723     |        1 |
| 20170724     |        1 |
| 20170727     |        1 |
| 20170728     |        1 |
| 20170730     |        1 |
| 20170731     |        2 |
| 20170801     |        1 |
| 20170803     |        1 |
| 20170804     |        1 |
| 20170806     |        1 |
| 20170807     |        1 |
| 20170808     |        1 |
+--------------+----------+
16 rows in set (1 min 5.56 sec)
Tbayer added a comment.EditedSep 21 2017, 5:31 PM

Update: While we're still sorting out the session-based metrics (also now in the light of T175918), here are histograms for per-pageview number of cards viewed, for the three largest wikis in the test:


SELECT wiki,
cards_1500viewed_on_page,
COUNT(*) AS frequency
FROM
(SELECT wiki,
event_pageToken,
COUNT(DISTINCT event_linkInteractionToken) AS cards_1500viewed_on_page
FROM log.Popups_16364296
WHERE
(event_action = 'pageLoaded'
OR (event_totalInteractionTime >= event_perceivedWait + 1500 ))
AND wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki')
AND event_isAnon = 1
AND event_popupEnabled = 1

AND LEFT(timestamp, 8) >= '20170725'
AND LEFT(timestamp, 8) < '20170822'
GROUP BY wiki, event_popupEnabled, event_pageToken) AS pageviewlist
GROUP BY wiki, cards_1500viewed_on_page
ORDER BY wiki, cards_1500viewed_on_page

[edit: fixed chart labels - as could be seen from the query, this was actually still the data for the previously used limit of 1.5s instead of 1s]

Update: Here is a look at the disable actions, which are still very rare.

SELECT LEFT(timestamp, 8) AS yearmonthday, COUNT(*) AS disables
FROM log.Popups_16364296
WHERE event_action = 'disabled'
GROUP BY yearmonthday
ORDER BY yearmonthday;
+--------------+----------+
| yearmonthday | disables |
+--------------+----------+
| 20170703     |        1 |
| 20170705     |        2 |
| 20170706     |        4 |
| 20170718     |        1 |
| 20170723     |        1 |
| 20170724     |        1 |
| 20170727     |        1 |
| 20170728     |        1 |
| 20170730     |        1 |
| 20170731     |        2 |
| 20170801     |        1 |
| 20170803     |        1 |
| 20170804     |        1 |
| 20170806     |        1 |
| 20170807     |        1 |
| 20170808     |        1 |
+--------------+----------+
16 rows in set (1 min 5.56 sec)

For comparison, note the total number of session per day, below. Caveat: the logging of disable events may very well have been affected by T175918 too.

SELECT LEFT(timestamp, 8) AS yearmonthday, event_popupenabled  AS enabled,
COUNT(DISTINCT event_sessionToken) AS sessions
FROM log.Popups_16364296
GROUP BY yearmonthday, enabled
ORDER BY yearmonthday, enabled
--------------

yearmonthday    enabled sessions
20170329        0       306
20170329        1       2077
20170330        0       1013
20170330        1       7447
20170331        0       692
20170331        1       5157
20170401        0       518
20170401        1       3919
20170402        0       594
20170402        1       4297
20170403        0       634
20170403        1       5258
20170404        0       740
20170404        1       5423
20170405        0       600
20170405        1       4829
20170406        0       682
20170406        1       5192
20170407        0       688
20170407        1       5157
20170408        0       561
20170408        1       4231
20170409        0       608
20170409        1       5033
20170410        0       746
20170410        1       5814
20170411        0       705
20170411        1       5555
20170412        0       708
20170412        1       5379
20170413        0       667
20170413        1       5336
20170414        0       685
20170414        1       5018
20170415        0       526
20170415        1       4226
20170416        0       547
20170416        1       4249
20170417        0       619
20170417        1       4826
20170418        0       777
20170418        1       6062
20170419        0       792
20170419        1       6185
20170420        0       722
20170420        1       5607
20170421        0       675
20170421        1       5174
20170422        0       558
20170422        1       4138
20170423        0       650
20170423        1       4871
20170424        0       697
20170424        1       5669
20170425        0       699
20170425        1       5581
20170426        0       711
20170426        1       5663
20170427        0       685
20170427        1       5320
20170428        0       680
20170428        1       5405
20170429        0       525
20170429        1       4081
20170430        0       574
20170430        1       4544
20170501        0       649
20170501        1       5403
20170502        0       827
20170502        1       5895
20170503        0       837
20170503        1       6265
20170504        0       826
20170504        1       5964
20170505        0       641
20170505        1       4903
20170506        0       501
20170506        1       3813
20170507        0       589
20170507        1       4571
20170508        0       750
20170508        1       5594
20170509        0       699
20170509        1       5717
20170510        0       770
20170510        1       6270
20170511        0       763
20170511        1       6133
20170512        0       677
20170512        1       5358
20170513        0       590
20170513        1       4613
20170514        0       665
20170514        1       5269
20170515        0       798
20170515        1       6332
20170516        0       757
20170516        1       6199
20170517        0       794
20170517        1       5870
20170518        0       716
20170518        1       5741
20170519        0       703
20170519        1       5276
20170520        0       588
20170520        1       4387
20170521        0       694
20170521        1       4946
20170522        0       777
20170522        1       5823
20170523        0       724
20170523        1       5567
20170524        0       655
20170524        1       5323
20170525        0       707
20170525        1       5123
20170526        0       589
20170526        1       4433
20170527        0       492
20170527        1       3513
20170528        0       524
20170528        1       3948
20170529        0       744
20170529        1       5634
20170530        0       707
20170530        1       5660
20170531        0       737
20170531        1       6004
20170601        0       705
20170601        1       5435
20170602        0       660
20170602        1       4936
20170603        0       550
20170603        1       4217
20170604        0       611
20170604        1       4868
20170605        0       775
20170605        1       5717
20170606        0       803
20170606        1       5842
20170607        0       681
20170607        1       5435
20170608        0       660
20170608        1       4975
20170609        0       682
20170609        1       5087
20170610        0       462
20170610        1       3769
20170611        0       536
20170611        1       3968
20170612        0       672
20170612        1       5406
20170613        0       734
20170613        1       5799
20170614        0       708
20170614        1       5338
20170615        0       689
20170615        1       5240
20170616        0       616
20170616        1       5003
20170617        0       523
20170617        1       3875
20170618        0       519
20170618        1       4131
20170619        0       707
20170619        1       5047
20170620        0       610
20170620        1       4841
20170621        0       674
20170621        1       4853
20170622        0       668
20170622        1       4910
20170623        0       568
20170623        1       4485
20170624        0       423
20170624        1       3366
20170625        0       492
20170625        1       3622
20170626        0       637
20170626        1       4967
20170627        0       696
20170627        1       5676
20170628        0       638
20170628        1       5168
20170629        0       585
20170629        1       5211
20170630        0       710
20170630        1       5389
20170701        0       491
20170701        1       4068
20170702        0       517
20170702        1       4171
20170703        0       606
20170703        1       5155
20170704        0       654
20170704        1       5120
20170705        0       657
20170705        1       5152
20170706        0       663
20170706        1       5156
20170707        0       578
20170707        1       4689
20170708        0       460
20170708        1       3433
20170709        0       514
20170709        1       3910
20170710        0       651
20170710        1       5111
20170711        0       649
20170711        1       5224
20170712        0       642
20170712        1       4912
20170713        0       618
20170713        1       4903
20170714        0       571
20170714        1       4392
20170715        0       443
20170715        1       3334
20170716        0       470
20170716        1       3679
20170717        0       623
20170717        1       5242
20170718        0       673
20170718        1       5381
20170719        0       595
20170719        1       5170
20170720        0       635
20170720        1       4944
20170721        0       630
20170721        1       4987
20170722        0       499
20170722        1       3732
20170723        0       462
20170723        1       3745
20170724        0       2804
20170724        1       22025
20170725        0       5385
20170725        1       43164
20170726        0       5650
20170726        1       44462
20170727        0       5527
20170727        1       43926
20170728        0       5592
20170728        1       45245
20170729        0       4107
20170729        1       33526
20170730        0       4612
20170730        1       36200
20170731        0       6007
20170731        1       48128
20170801        0       5581
20170801        1       44669
20170802        0       5193
20170802        1       41592
20170803        0       5342
20170803        1       42372
20170804        0       5291
20170804        1       43722
20170805        0       3885
20170805        1       30461
20170806        0       4004
20170806        1       31503
20170807        0       5376
20170807        1       43045
20170808        0       5564
20170808        1       45918
20170809        0       5204
20170809        1       41867
20170810        0       5000
20170810        1       40724
20170811        0       5263
20170811        1       42844
20170812        0       3846
20170812        1       30506
20170813        0       3977
20170813        1       31700
20170814        0       5062
20170814        1       40743
20170815        0       5449
20170815        1       44625
20170816        0       5632
20170816        1       46444
20170817        0       5712
20170817        1       46553
20170818        0       5487
20170818        1       45069
20170819        0       4319
20170819        1       34533
20170820        0       4408
20170820        1       34860
20170821        0       5756
20170821        1       47235
20170822        0       5989
20170822        1       48153
20170823        0       5666
20170823        1       46834
20170824        0       4619
20170824        1       37936
20170825        0       22
20170825        1       180
20170826        0       19
20170826        1       53
20170827        0       7
20170827        1       51
20170828        0       2808
20170828        1       2904
20170829        0       46902
20170829        1       47191
20170830        0       418121
20170830        1       417253
20170831        0       889361
20170831        1       888109
20170901        0       824839
20170901        1       823351
20170902        0       615171
20170902        1       614036
20170903        0       687526
20170903        1       682189
20170904        0       916125
20170904        1       913115
20170905        0       1052800
20170905        1       1049049
20170906        0       9192
20170906        1       9145

For the record, the equivalent of T158172#3625184 for the "seen" limit we later settled on (1000ms instead of 1500ms):


In numbers, the percentage of pageviews with no preview card viewed (left bar in the first chart):

ruwikiitwikihuwiki
80.8%84.5%87.2%

Again, these numbers may well have been affected by T175918 too.

SELECT wiki,
cards_1000viewed_on_page,
COUNT(*) AS frequency
FROM
(SELECT wiki,
event_pageToken,
COUNT(DISTINCT event_linkInteractionToken) AS cards_1000viewed_on_page
FROM log.Popups_16364296
WHERE
(event_action = 'pageLoaded'
OR (event_totalInteractionTime >= event_perceivedWait + 1000 ))
AND wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki')
AND event_isAnon = 1
AND event_popupEnabled = 1
AND LEFT(timestamp, 8) >= '20170725'
AND LEFT(timestamp, 8) < '20170822'
GROUP BY wiki, event_popupEnabled, event_pageToken) AS pageviewlist
GROUP BY wiki, cards_1000viewed_on_page
ORDER BY wiki, cards_1000viewed_on_page
ovasileva closed this task as Resolved.Dec 7 2017, 12:47 PM

resolving for now, some of the data above might be included in T182315: Write and publish report on enwiki and dewiki a/b test analysis for page previews