Page MenuHomePhabricator

Stop Hovercards A/B tests
Closed, ResolvedPublic1 Story Points

Description

Stop hovercards test on Russian and Italian Wikipedias on Nov 28, 2016

Details

Related Gerrit Patches:

Event Timeline

ovasileva created this task.Sep 1 2016, 1:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 1 2016, 1:27 PM
ovasileva triaged this task as Medium priority.Sep 7 2016, 3:43 PM
ovasileva moved this task from Incoming to Triaged but Future on the Readers-Web-Backlog board.
ovasileva raised the priority of this task from Medium to High.Nov 9 2016, 11:33 AM
ovasileva moved this task from Triaged but Future to Upcoming on the Readers-Web-Backlog board.
ovasileva renamed this task from Stop Russian A/B Hovercards test to Stop Hovercards A/B tests.Nov 15 2016, 5:24 PM
ovasileva updated the task description. (Show Details)
ovasileva set the point value for this task to 1.
Elitre added a subscriber: Elitre.Nov 17 2016, 7:53 PM

Olga, can you please comment on the reasons of the delay? Thank you.

(I mean, I'm confused as to which results are being referred to in updates like https://lists.wikimedia.org/pipermail/wikitech-l/2016-November/087041.html , if the test isn't over? TY!)

Hi @Elitre - we wanted to wait to formalize the results first before pausing the test. Also, since the results are positive for all the tested wikipedias, we wanted to reach out to each community to see if they would like to keep the feature on and schedule a date for that - that way we're minimizing the time that users who have gotten used to and like the feature would have to go without it in the future. We're lining up the pause for next sprint, so the tests should be off early next week.

Thanks - it is still unclear when you stopped collecting the results.

@Elitre - good point, results between september 23 and nov 10 were used for the report - I will add a note within https://www.mediawiki.org/wiki/Beta_Features/Hovercards/2016_A/B_Tests

This won't be possible on the 25th (it's Black Friday), can you please update the date?

Do we also want to stop Hungarian at the same time? (T135440) That task says it should have been on for 4 weeks.. which it has.

ovasileva updated the task description. (Show Details)Nov 23 2016, 4:30 PM

Updated.

^ in case you missed by Hungarian question.

ovasileva added a comment.EditedNov 24 2016, 11:02 AM

Let's use this task for Italian and Russian only. Hungarian will be separate.

You meant Italian, I think.

Change 323878 had a related patch set uploaded (by Jdlrobson):
Disable Popups A/B test on Russian and Italian Wikipedias

https://gerrit.wikimedia.org/r/323878

Change 323878 merged by jenkins-bot:
Disable Popups A/B test on Russian and Italian Wikipedias

https://gerrit.wikimedia.org/r/323878

Mentioned in SAL (#wikimedia-operations) [2016-11-28T19:40:57Z] <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:323878|Disable Popups A/B test on Russian and Italian Wikipedias]] T144490 (duration: 00m 45s)

ovasileva closed this task as Resolved.Nov 29 2016, 6:36 PM
ovasileva claimed this task.
Tbayer reopened this task as Open.Dec 3 2016, 5:52 AM

This was deployed on November 28, right? Strangely, the event rate on ruwiki and itwiki did not drop to 0 on that day, but spiked up instead (and is still above the level during the experiment). Does anyone have an idea what is happening here?


(Source: Grafana)

SELECT LEFT(timestamp, 8) AS date,  SUM(IF(wiki = 'huwiki',1,0)) AS hu_events, SUM(IF(wiki = 'itwiki',1,0)) AS it_events, SUM(IF(wiki = 'ruwiki',1,0)) AS ru_events FROM log.Popups_15906495 GROUP BY date ORDER BY date;
+----------+-----------+-----------+-----------+
| date     | hu_events | it_events | ru_events |
+----------+-----------+-----------+-----------+
| 20160922 |     22373 |       190 |       874 |
| 20160923 |    252265 |    350540 |    456066 |
| 20160924 |    254901 |    390318 |    643459 |
| 20160925 |    324654 |    426799 |    791504 |
| 20160926 |    334940 |    567353 |    820562 |
| 20160927 |    324204 |    558794 |    833451 |
| 20160928 |    321263 |    542015 |    802483 |
| 20160929 |    318069 |    567851 |    787320 |
| 20160930 |    270345 |    499840 |    715921 |
| 20161001 |    256650 |    432035 |    649106 |
| 20161002 |    340225 |    447202 |    795038 |
| 20161003 |    378865 |    589536 |    850620 |
| 20161004 |    372057 |    608689 |    868430 |
| 20161005 |    360868 |    598153 |    876730 |
| 20161006 |    351156 |    575780 |    869818 |
| 20161007 |    284766 |    545600 |    794498 |
| 20161008 |    290617 |    464138 |    699583 |
| 20161009 |    359913 |    496480 |    879238 |
| 20161010 |    358192 |    613233 |    897591 |
| 20161011 |    379136 |    588225 |    883906 |
| 20161012 |    361303 |    537058 |    873012 |
| 20161013 |    334339 |    620592 |    859166 |
| 20161014 |    295078 |    545790 |    797310 |
| 20161015 |    291274 |    412622 |    745353 |
| 20161016 |    371009 |    462605 |    918903 |
| 20161017 |    395491 |    613814 |    904706 |
| 20161018 |    389665 |    604258 |    882674 |
| 20161019 |    366972 |    609495 |    908835 |
| 20161020 |    350543 |    591556 |    859973 |
| 20161021 |    324606 |    551238 |    808875 |
| 20161022 |    299873 |    398239 |    739978 |
| 20161023 |    429062 |    480075 |    897525 |
| 20161024 |    377297 |    586267 |    924990 |
| 20161025 |    245068 |    373090 |    505130 |
| 20161026 |    237707 |    325321 |    481555 |
| 20161027 |    212735 |    314426 |    458149 |
| 20161028 |    178487 |    298161 |    441065 |
| 20161029 |    158383 |    208532 |    362883 |
| 20161030 |    176043 |    230485 |    400665 |
| 20161031 |    176074 |    269808 |    502976 |
| 20161101 |    218963 |    276659 |    459230 |
| 20161102 |    213715 |    351468 |    457224 |
| 20161103 |    203780 |    337219 |    434363 |
| 20161104 |    194891 |    307948 |    411688 |
| 20161105 |    184705 |    241223 |    360731 |
| 20161106 |    245406 |    268756 |    410224 |
| 20161107 |    254943 |    326759 |    510960 |
| 20161108 |    243133 |    382894 |    533145 |
| 20161109 |    250315 |    370752 |    596424 |
| 20161110 |    222688 |    332372 |    519704 |
| 20161111 |    164748 |    253374 |    329380 |
| 20161112 |    163069 |    183770 |    266660 |
| 20161113 |    188084 |    198105 |    327085 |
| 20161114 |    218525 |    284719 |    333396 |
| 20161115 |    184172 |    253395 |    310091 |
| 20161116 |    182185 |    241378 |    304051 |
| 20161117 |    177384 |    255444 |    320599 |
| 20161118 |    148287 |    241122 |    326730 |
| 20161119 |    146392 |    186191 |    266068 |
| 20161120 |    179485 |    187592 |    314642 |
| 20161121 |    197307 |    270744 |    364417 |
| 20161122 |    182042 |    245283 |    356413 |
| 20161123 |    173449 |    227899 |    331687 |
| 20161124 |    149255 |    206639 |    286691 |
| 20161125 |    124820 |    190420 |    254125 |
| 20161126 |    134671 |    172719 |    269824 |
| 20161127 |    175692 |    188738 |    295870 |
| 20161128 |    173118 |    410525 |    684143 |
| 20161129 |    169018 |   1143970 |   2538553 |
| 20161130 |    160289 |    886354 |   1626406 |
| 20161201 |    194309 |    849213 |   1633035 |
| 20161202 |    161196 |    656357 |    652790 |
+----------+-----------+-----------+-----------+
72 rows in set (2 min 10.43 sec)

Russian Wikipedia was sampled at 0.01 and Italian Wikipedia at 0.02. Upon disabling this experiment the sample rate became 0.1 (e.g. 5 times as more on Italian, 10 times more on Russian).
Although we disabled the A/B test, we did not disable the beta feature, so now we are logging events for all beta feature users.
On top of this, it's likely that there are cache considerations. Any anonymous users who were bucketed and are visiting pages that were generated before the deploy will still see popups until that page is edited or purged.
I don't know how trustworthy the preferences page is, but that suggests Italian Wikipedia has 2 495 users and Russian has 1... ? (!?!) .
It doesn't surprise me the spike went up, but I'd expect it to consistently drop over next 2 weeks and then it will account for 0.1 of all beta feature users.

@Tbayer can you show a more recent graph?

Jdlrobson added a comment.EditedDec 6 2016, 6:19 PM

@Tbayer specificially we'd like to answer these questions:
#1 For those events are the users logged in or not?
#2 What is the number of distinct session ids? Does this match with 10% of the number of beta feature users?

@Tbayer can you show a more recent graph?

Here's a link to a current one on Grafana: https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?var-schema=Popups&from=now-30d&to=now (one can always change the timespan by clicking on the top right)

@Tbayer specificially we'd like to answer these questions:
#1 For those events are the users logged in or not?

The vast majority (>98%) of events come from anonymous users, before and after Nov 28. The spike is due to anons only, the number of logged-in events stayed roughly the same. See detailed data below.

#2 What is the number of distinct session ids?

See detailed daily list below (similar takeaway: >99% of sessions are from anons, before and after)

Does this match with 10% of the number of beta feature users?

The logged-in events maybe, depending on how many of these beta feature users are active per day. E.g. for Dec 5, it would mean that around 8% of them were active on itwiki (= 19 / (2495*0.10) ), and 7% on ruwiki (= 68 / (9480 * 0.10)).

But not the overall events.

I don't know how trustworthy the preferences page is, but that suggests Italian Wikipedia has 2 495 users and Russian has 1... ? (!?!) .

I'm seeing "9,480 users are trying this feature" in my ruwiki preferences (BTW, used ULS to switch my user interface language to English there). We could check with a separate database query, but I would assume the number is by and large valid. A more important reason to do such a query would be to restrict that count to active users (editors) as it was done for skin usage in T147696.

It doesn't surprise me the spike went up, but I'd expect it to consistently drop over next 2 weeks and then it will account for 0.1 of all beta feature users.

Considering the above, I don't think we have a good explanation yet why the spike happened.

Data:

# events per day
SELECT LEFT(timestamp, 8) AS date,  
SUM(IF(wiki = 'itwiki' AND event_isAnon = 0,1,0)) AS it_loggedin, 
SUM(IF(wiki = 'itwiki' AND event_isAnon = 1,1,0)) AS it_anon, 
SUM(IF(wiki = 'ruwiki' AND event_isAnon = 0,1,0)) AS ru_loggedin, 
SUM(IF(wiki = 'ruwiki' AND event_isAnon = 1,1,0)) AS ru_anon
FROM log.Popups_15906495 
GROUP BY date ORDER BY date;
+----------+-------------+---------+-------------+---------+
| date     | it_loggedin | it_anon | ru_loggedin | ru_anon |
+----------+-------------+---------+-------------+---------+
| 20160922 |          81 |     109 |         513 |     361 |
| 20160923 |        5153 |  345387 |        6760 |  449306 |
| 20160924 |        3066 |  387252 |        6267 |  637192 |
| 20160925 |        4855 |  421944 |       12037 |  779467 |
| 20160926 |        5504 |  561849 |        4244 |  816318 |
| 20160927 |        5699 |  553095 |        4773 |  828678 |
| 20160928 |        5940 |  536075 |        5963 |  796520 |
| 20160929 |        6009 |  561842 |        7202 |  780118 |
| 20160930 |        6587 |  493253 |        5191 |  710730 |
| 20161001 |        3853 |  428182 |        4638 |  644468 |
| 20161002 |        7042 |  440160 |        3792 |  791246 |
| 20161003 |        5880 |  583656 |        4993 |  845627 |
| 20161004 |        6712 |  601977 |        3599 |  864831 |
| 20161005 |        6182 |  591971 |        5998 |  870732 |
| 20161006 |        8875 |  566905 |        9498 |  860320 |
| 20161007 |        5486 |  540114 |        8034 |  786464 |
| 20161008 |        3939 |  460199 |        5180 |  694403 |
| 20161009 |        4871 |  491609 |        9249 |  869989 |
| 20161010 |        7390 |  605843 |        7681 |  889910 |
| 20161011 |        4117 |  584108 |        7882 |  876024 |
| 20161012 |        4710 |  532348 |       13004 |  860008 |
| 20161013 |        7204 |  613388 |       12974 |  846192 |
| 20161014 |        5523 |  540267 |        8820 |  788490 |
| 20161015 |        5021 |  407601 |        9073 |  736280 |
| 20161016 |        5265 |  457340 |        9898 |  909005 |
| 20161017 |        3686 |  610128 |        6723 |  897983 |
| 20161018 |        8630 |  595628 |        4632 |  878042 |
| 20161019 |        8020 |  601475 |        7906 |  900929 |
| 20161020 |        6339 |  585217 |        5146 |  854827 |
| 20161021 |        5927 |  545311 |        7268 |  801607 |
| 20161022 |        5981 |  392258 |        5339 |  734639 |
| 20161023 |        5897 |  474178 |        9549 |  887976 |
| 20161024 |        5929 |  580338 |        5439 |  919551 |
| 20161025 |        4107 |  368983 |        3520 |  501610 |
| 20161026 |        1407 |  323914 |        4498 |  477057 |
| 20161027 |        2460 |  311966 |        3565 |  454584 |
| 20161028 |        2597 |  295564 |        4523 |  436542 |
| 20161029 |        2102 |  206430 |        3020 |  359863 |
| 20161030 |        2835 |  227650 |        6867 |  393798 |
| 20161031 |        3144 |  266664 |        3527 |  499449 |
| 20161101 |        3196 |  273463 |        1810 |  457420 |
| 20161102 |        3499 |  347969 |        5629 |  451595 |
| 20161103 |        2868 |  334351 |        4630 |  429733 |
| 20161104 |        2146 |  305802 |        4630 |  407058 |
| 20161105 |        3578 |  237645 |        2319 |  358412 |
| 20161106 |        1509 |  267247 |        2580 |  407644 |
| 20161107 |        2257 |  324502 |        4440 |  506520 |
| 20161108 |        2502 |  380392 |        4737 |  528408 |
| 20161109 |        2221 |  368531 |        3147 |  593277 |
| 20161110 |        2727 |  329645 |        5722 |  513982 |
| 20161111 |        2715 |  250659 |        1719 |  327661 |
| 20161112 |        1881 |  181889 |        2292 |  264368 |
| 20161113 |        3034 |  195071 |        2021 |  325064 |
| 20161114 |        2110 |  282609 |        2468 |  330928 |
| 20161115 |        1975 |  251420 |        1666 |  308425 |
| 20161116 |        2001 |  239377 |        2731 |  301320 |
| 20161117 |        1315 |  254129 |        3323 |  317276 |
| 20161118 |        4499 |  236623 |        2580 |  324150 |
| 20161119 |        2121 |  184070 |        1700 |  264368 |
| 20161120 |        1597 |  185995 |        3134 |  311508 |
| 20161121 |        2306 |  268438 |        3961 |  360456 |
| 20161122 |        1674 |  243609 |        2457 |  353956 |
| 20161123 |        1560 |  226339 |        2807 |  328880 |
| 20161124 |        1244 |  205395 |        1999 |  284692 |
| 20161125 |        1366 |  189054 |        1783 |  252342 |
| 20161126 |        2152 |  170567 |        1173 |  268651 |
| 20161127 |        1616 |  187122 |        5715 |  290155 |
| 20161128 |        1430 |  409095 |        2135 |  682008 |
| 20161129 |         329 | 1143641 |        1612 | 2536941 |
| 20161130 |         447 |  885907 |        1998 | 1624408 |
| 20161201 |         790 |  848423 |        3125 | 1629910 |
| 20161202 |         475 |  660008 |        3044 |  651932 |
| 20161203 |        1002 |  395483 |        3312 |  315228 |
| 20161204 |         464 |  308050 |        5015 |  308499 |
| 20161205 |        1257 |  274383 |        3706 |  246538 |
| 20161206 |        1405 |   15548 |        6143 |  166406 |
+----------+-------------+---------+-------------+---------+
76 rows in set (8 min 44.92 sec)

# sessions per day
SELECT LEFT(timestamp, 8) AS date,  
COUNT(DISTINCT (CASE WHEN wiki = 'itwiki' AND event_isAnon = 0 THEN event_sessionToken END)) AS it_loggedin,
COUNT(DISTINCT (CASE WHEN wiki = 'itwiki' AND event_isAnon = 1 THEN event_sessionToken END)) AS it_anon,
COUNT(DISTINCT (CASE WHEN wiki = 'ruwiki' AND event_isAnon = 0 THEN event_sessionToken END)) AS ru_loggedin,
COUNT(DISTINCT (CASE WHEN wiki = 'ruwiki' AND event_isAnon = 1 THEN event_sessionToken END)) AS ru_anon
FROM log.Popups_15906495 
GROUP BY date ORDER BY date;
+----------+-------------+---------+-------------+---------+
| date     | it_loggedin | it_anon | ru_loggedin | ru_anon |
+----------+-------------+---------+-------------+---------+
| 20160922 |           6 |      32 |          17 |      71 |
| 20160923 |          88 |   20050 |         114 |   23835 |
| 20160924 |          68 |   18503 |          88 |   28409 |
| 20160925 |          85 |   20489 |         108 |   34131 |
| 20160926 |          98 |   29339 |          82 |   39909 |
| 20160927 |          87 |   28561 |          99 |   39663 |
| 20160928 |         101 |   28585 |          79 |   39877 |
| 20160929 |          98 |   28837 |         100 |   39301 |
| 20160930 |          85 |   25844 |          95 |   35318 |
| 20161001 |          83 |   20229 |          65 |   28499 |
| 20161002 |          75 |   21614 |          87 |   33999 |
| 20161003 |         110 |   29761 |          97 |   40117 |
| 20161004 |         100 |   30409 |          91 |   40158 |
| 20161005 |         112 |   29738 |          84 |   40103 |
| 20161006 |          97 |   28837 |          75 |   39442 |
| 20161007 |          96 |   26823 |          90 |   36155 |
| 20161008 |          83 |   20001 |          86 |   29941 |
| 20161009 |          72 |   22688 |          92 |   36281 |
| 20161010 |         100 |   30179 |         109 |   40861 |
| 20161011 |          97 |   29953 |         109 |   40677 |
| 20161012 |          87 |   28193 |         104 |   39893 |
| 20161013 |         104 |   30196 |         111 |   39413 |
| 20161014 |          98 |   27916 |         106 |   36807 |
| 20161015 |          94 |   20145 |          93 |   30239 |
| 20161016 |          98 |   21398 |          99 |   36747 |
| 20161017 |          89 |   30589 |         109 |   41445 |
| 20161018 |          96 |   30402 |          92 |   41215 |
| 20161019 |         115 |   30551 |         113 |   41061 |
| 20161020 |          97 |   29422 |          99 |   40241 |
| 20161021 |         100 |   27432 |         116 |   36877 |
| 20161022 |          83 |   20007 |          95 |   30867 |
| 20161023 |          86 |   23136 |         113 |   37182 |
| 20161024 |          99 |   32081 |         109 |   42345 |
| 20161025 |         122 |   29402 |          95 |   38464 |
| 20161026 |          96 |   27139 |          82 |   37142 |
| 20161027 |          89 |   24698 |          90 |   36487 |
| 20161028 |          74 |   24430 |          83 |   33969 |
| 20161029 |          78 |   16614 |          74 |   27448 |
| 20161030 |          84 |   17920 |          75 |   28735 |
| 20161031 |          88 |   21239 |          86 |   38524 |
| 20161101 |          83 |   21892 |         102 |   36874 |
| 20161102 |         111 |   28024 |         100 |   36257 |
| 20161103 |          79 |   26870 |          95 |   33909 |
| 20161104 |          75 |   25206 |          88 |   29924 |
| 20161105 |          78 |   18961 |          70 |   25931 |
| 20161106 |          76 |   19754 |          75 |   28671 |
| 20161107 |          76 |   25161 |          90 |   38311 |
| 20161108 |          87 |   30013 |         107 |   39893 |
| 20161109 |          82 |   27941 |         106 |   42750 |
| 20161110 |          93 |   26547 |         101 |   38578 |
| 20161111 |          79 |   26064 |          92 |   34989 |
| 20161112 |          74 |   18085 |          76 |   27277 |
| 20161113 |          71 |   18685 |          88 |   31042 |
| 20161114 |          96 |   28716 |          83 |   34160 |
| 20161115 |          94 |   25670 |          83 |   31720 |
| 20161116 |          93 |   24838 |          81 |   30287 |
| 20161117 |          77 |   25667 |          88 |   33203 |
| 20161118 |          97 |   25553 |          78 |   35691 |
| 20161119 |          65 |   18153 |          72 |   27071 |
| 20161120 |          75 |   18536 |          79 |   30515 |
| 20161121 |         109 |   29170 |          92 |   39469 |
| 20161122 |          78 |   25695 |          95 |   37765 |
| 20161123 |          79 |   24004 |         101 |   35898 |
| 20161124 |          77 |   22830 |          89 |   32106 |
| 20161125 |          69 |   21258 |          88 |   27503 |
| 20161126 |          76 |   17051 |          65 |   24607 |
| 20161127 |          68 |   18103 |          80 |   27677 |
| 20161128 |          56 |   39357 |          80 |   62152 |
| 20161129 |          11 |  101135 |          52 |  226643 |
| 20161130 |          19 |   82552 |          48 |  168260 |
| 20161201 |          19 |   90956 |          82 |  192779 |
| 20161202 |          17 |   75869 |          75 |   94438 |
| 20161203 |          14 |   46492 |          68 |   49834 |
| 20161204 |          13 |   37891 |          72 |   49504 |
| 20161205 |          19 |   36421 |          68 |   42207 |
| 20161206 |          23 |    2851 |          66 |   29591 |
+----------+-------------+---------+-------------+---------+
76 rows in set (1 hour 17 min 22.42 sec)

Oh, and contrary to what one might have expected, the spike is due to an increase in Hovercards ON events, whereas Hovercards OFF events dropped sharply around November 28. Either stopping the test had the opposite effect from the one intended, in that vastly *more* anons saw Hovercards after November 28, or something is very wrong with the instrumentation here.

# events per day, Hovercards ON vs OFF
SELECT LEFT(timestamp, 8) AS date,  
SUM(IF(wiki = 'itwiki' AND event_popupEnabled = 0,1,0)) AS it_OFF, 
SUM(IF(wiki = 'itwiki' AND event_popupEnabled = 1,1,0)) AS it_ON, 
SUM(IF(wiki = 'ruwiki' AND event_popupEnabled = 0,1,0)) AS ru_OFF,
SUM(IF(wiki = 'ruwiki' AND event_popupEnabled = 1,1,0)) AS ru_ON
FROM log.Popups_15906495 
GROUP BY date ORDER BY date;

+----------+--------+---------+--------+---------+
| date     | it_OFF | it_ON   | ru_OFF | ru_ON   |
+----------+--------+---------+--------+---------+
| 20160922 |     88 |     102 |     19 |     855 |
| 20160923 | 282340 |   68200 | 409713 |   46353 |
| 20160924 | 322640 |   67678 | 588126 |   55333 |
| 20160925 | 353194 |   73605 | 711140 |   80364 |
| 20160926 | 466975 |  100378 | 746295 |   74267 |
| 20160927 | 456009 |  102785 | 754385 |   79066 |
| 20160928 | 449058 |   92957 | 724661 |   77822 |
| 20160929 | 466210 |  101641 | 718848 |   68472 |
| 20160930 | 410038 |   89802 | 655910 |   60011 |
| 20161001 | 355739 |   76296 | 587642 |   61464 |
| 20161002 | 364457 |   82745 | 723302 |   71736 |
| 20161003 | 484446 |  105090 | 776568 |   74052 |
| 20161004 | 500194 |  108495 | 787834 |   80596 |
| 20161005 | 491123 |  107030 | 795850 |   80880 |
| 20161006 | 471291 |  104489 | 790335 |   79483 |
| 20161007 | 447215 |   98385 | 722652 |   71846 |
| 20161008 | 380825 |   83313 | 637511 |   62072 |
| 20161009 | 402972 |   93508 | 794949 |   84289 |
| 20161010 | 505192 |  108041 | 817552 |   80039 |
| 20161011 | 478768 |  109457 | 801848 |   82058 |
| 20161012 | 438821 |   98237 | 784054 |   88958 |
| 20161013 | 506178 |  114414 | 770842 |   88324 |
| 20161014 | 448170 |   97620 | 719785 |   77525 |
| 20161015 | 336016 |   76606 | 677021 |   68332 |
| 20161016 | 384429 |   78176 | 829265 |   89638 |
| 20161017 | 506462 |  107352 | 822117 |   82589 |
| 20161018 | 501453 |  102805 | 810544 |   72130 |
| 20161019 | 498234 |  111261 | 823217 |   85618 |
| 20161020 | 482211 |  109345 | 782828 |   77145 |
| 20161021 | 446587 |  104651 | 733371 |   75504 |
| 20161022 | 328975 |   69264 | 678148 |   61830 |
| 20161023 | 388424 |   91651 | 817301 |   80224 |
| 20161024 | 479649 |  106618 | 839609 |   85381 |
| 20161025 | 304018 |   69072 | 455825 |   49305 |
| 20161026 | 265477 |   59844 | 436435 |   45120 |
| 20161027 | 253465 |   60961 | 419581 |   38568 |
| 20161028 | 240005 |   58156 | 402473 |   38592 |
| 20161029 | 166323 |   42209 | 331942 |   30941 |
| 20161030 | 189454 |   41031 | 363379 |   37286 |
| 20161031 | 223210 |   46598 | 454544 |   48432 |
| 20161101 | 224480 |   52179 | 419698 |   39532 |
| 20161102 | 287646 |   63822 | 414685 |   42539 |
| 20161103 | 274127 |   63092 | 393537 |   40826 |
| 20161104 | 254555 |   53393 | 377000 |   34688 |
| 20161105 | 196905 |   44318 | 328343 |   32388 |
| 20161106 | 222970 |   45786 | 373168 |   37056 |
| 20161107 | 267494 |   59265 | 464035 |   46925 |
| 20161108 | 315502 |   67392 | 482576 |   50569 |
| 20161109 | 302701 |   68051 | 542900 |   53524 |
| 20161110 | 273856 |   58516 | 473871 |   45833 |
| 20161111 | 197998 |   55376 | 287696 |   41684 |
| 20161112 | 137607 |   46163 | 232696 |   33964 |
| 20161113 | 151071 |   47034 | 287357 |   39728 |
| 20161114 | 212238 |   72481 | 294195 |   39201 |
| 20161115 | 190461 |   62934 | 273159 |   36932 |
| 20161116 | 183706 |   57672 | 268446 |   35605 |
| 20161117 | 194840 |   60604 | 280576 |   40023 |
| 20161118 | 184223 |   56899 | 288318 |   38412 |
| 20161119 | 145459 |   40732 | 232705 |   33363 |
| 20161120 | 144206 |   43386 | 279648 |   34994 |
| 20161121 | 207422 |   63322 | 321752 |   42665 |
| 20161122 | 187839 |   57444 | 312105 |   44308 |
| 20161123 | 175787 |   52112 | 289838 |   41849 |
| 20161124 | 157419 |   49220 | 250418 |   36273 |
| 20161125 | 144027 |   46393 | 224990 |   29135 |
| 20161126 | 132235 |   40484 | 236860 |   32964 |
| 20161127 | 143300 |   45438 | 258675 |   37195 |
| 20161128 | 156610 |  253915 | 248911 |  435232 |
| 20161129 |   2347 | 1141623 |   3407 | 2535146 |
| 20161130 |   1494 |  884860 |   1912 | 1624494 |
| 20161201 |    649 |  848564 |   2277 | 1630758 |
| 20161202 |    848 |  659635 |   1379 |  653597 |
| 20161203 |    459 |  396026 |    637 |  317903 |
| 20161204 |    152 |  308362 |    411 |  313103 |
| 20161205 |    200 |  275440 |    594 |  249650 |
| 20161206 |      7 |   17234 |   1826 |  173669 |
| 20161207 |      0 |     402 |     57 |    9854 |
+----------+--------+---------+--------+---------+
77 rows in set (6 min 55.65 sec)
phuedx added a subscriber: phuedx.EditedDec 7 2016, 7:45 PM

TL;DR: When we disabled the experiment, we enabled Page Previews for all anons viewing pages that hadn't been touched (evicted from – at least – our Varnishes). Anons will continue to see previews for links on a page for as long as that page is cached (IIRC 14 days).


If the experiment is enabled, we instruct the Resource Loader to always load the Page Previews code. This instruction is generated on the server and embedded in the response and, consequently, is cached in our Varnishes. After the experiment has been disabled, those cached pages aren't automatically evicted. We have to wait for the page to be touched so that the cached response is evicted and a new response without the instruction to always load the Page Previews code is cached.

Now, during initialization, Page Previews checks whether previews should be shown to the user in the mw.popups.getEnabledState method. Since the experiment is disabled, it'll check whether the user has not disabled Page Previews:

return mw.storage.get( 'mwe-popups-enabled' ) !== '0';

If a user hasn't explicitly enabled or disabled Page Previews via the settings modal, then their storage will not have this key present. Herein lay the bug: when a key isn't present in the storage, mw.storage.get will return null and null is strictly not equal to '0', i.e. the line above will evaluate to true and the user will be shown previews.

This bug would not have occurred if the line above was written as follows:

return mw.storage.get( 'mwe-popups-enabled' ) === '1';

Lessons

  1. The unit tests for mw.popups.getEnabledState and it's DocBlock are quite clear about this behaviour. When we have a "Disable $feature" task, we need to ask ourselves "What happens when we actually disable it?" during planning/estimation.

@phuedx, is there a task that captures the bug described at T144490#2855170?

phuedx added a comment.Dec 8 2016, 2:40 PM

@bmansurov: Nope. I can write one real quick if you need.

phuedx added a comment.Dec 8 2016, 5:07 PM

@phuedx, is there a task that captures the bug described at T144490#2855170?

Now, yes! T152687: Disable Page Previews for anons

ovasileva closed this task as Resolved.Dec 14 2016, 8:54 PM