Stop Hovercards A/B tests
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

Stop hovercards test on Russian and Italian Wikipedias on Nov 28, 2016

Details

	Subject	Repo	Branch	Lines +/-
	Disable Popups A/B test on Russian and Italian Wikipedias	operations/mediawiki-config	master	+0 -27

Customize query in gerrit

Related Objects

Mentioned In: T157111: [8 hours] Create a Page Previews performance dashboard
T152687: Disable Page Previews for anons
T144491: Stop Italian Hovercards a/b test
Mentioned Here: T152687: Disable Page Previews for anons
T147696: Statistics about /active/ users of skins on the Wikimedia cluster
T135440: Stop Hovercards Hungarian A/B test

Event Timeline

ovasileva created this task.Sep 1 2016, 1:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 1 2016, 1:27 PM

ovasileva triaged this task as Medium priority.Sep 7 2016, 3:43 PM

ovasileva moved this task from Incoming to Triaged but Future on the Web-Team-Backlog board.

• Tbayer subscribed.Sep 12 2016, 9:33 AM

ovasileva raised the priority of this task from Medium to High.Nov 9 2016, 11:33 AM

ovasileva moved this task from Triaged but Future to Upcoming on the Web-Team-Backlog board.

ovasileva added a project: Reading-Web-Sprint-86-🔪🦃.Nov 9 2016, 11:36 AM

ovasileva renamed this task from Stop Russian A/B Hovercards test to Stop Hovercards A/B tests.Nov 15 2016, 5:24 PM

ovasileva updated the task description. (Show Details)

ovasileva set the point value for this task to 1.

ovasileva mentioned this in T144491: Stop Italian Hovercards a/b test.

Olga, can you please comment on the reasons of the delay? Thank you.

(I mean, I'm confused as to which results are being referred to in updates like https://lists.wikimedia.org/pipermail/wikitech-l/2016-November/087041.html , if the test isn't over? TY!)

Hi @Elitre - we wanted to wait to formalize the results first before pausing the test. Also, since the results are positive for all the tested wikipedias, we wanted to reach out to each community to see if they would like to keep the feature on and schedule a date for that - that way we're minimizing the time that users who have gotten used to and like the feature would have to go without it in the future. We're lining up the pause for next sprint, so the tests should be off early next week.

Thanks - it is still unclear when you stopped collecting the results.

@Elitre - good point, results between september 23 and nov 10 were used for the report - I will add a note within https://www.mediawiki.org/wiki/Beta_Features/Hovercards/2016_A/B_Tests

Jdlrobson moved this task from Needs Analysis to To Do on the Reading-Web-Sprint-86-🔪🦃 board.Nov 21 2016, 5:55 PM

ovasileva moved this task from Upcoming to 2016-17 Q2 on the Web-Team-Backlog board.Nov 21 2016, 7:46 PM

This won't be possible on the 25th (it's Black Friday), can you please update the date?

Jdlrobson moved this task from To Do to Blocked Externally on the Reading-Web-Sprint-86-🔪🦃 board.Nov 23 2016, 4:09 PM

Do we also want to stop Hungarian at the same time? (T135440) That task says it should have been on for 4 weeks.. which it has.

Updated.

^ in case you missed by Hungarian question.

Let's use this task for Italian and Russian only. Hungarian will be separate.

You meant Italian, I think.

Jdlrobson moved this task from Blocked Externally to To Do on the Reading-Web-Sprint-86-🔪🦃 board.Nov 28 2016, 7:19 PM

Change 323878 had a related patch set uploaded (by Jdlrobson):
Disable Popups A/B test on Russian and Italian Wikipedias

https://gerrit.wikimedia.org/r/323878

gerritbot added a project: Patch-For-Review.Nov 28 2016, 7:22 PM

Jdlrobson moved this task from To Do to Code Review on the Reading-Web-Sprint-86-🔪🦃 board.Nov 28 2016, 7:24 PM

Change 323878 merged by jenkins-bot:
Disable Popups A/B test on Russian and Italian Wikipedias

https://gerrit.wikimedia.org/r/323878

Jdlrobson moved this task from Code Review to Ready for Signoff on the Reading-Web-Sprint-86-🔪🦃 board.Nov 28 2016, 7:40 PM

Mentioned in SAL (#wikimedia-operations) [2016-11-28T19:40:57Z] <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:323878|Disable Popups A/B test on Russian and Italian Wikipedias]] T144490 (duration: 00m 45s)

ovasileva closed this task as Resolved.Nov 29 2016, 6:36 PM

ovasileva claimed this task.

This was deployed on November 28, right? Strangely, the event rate on ruwiki and itwiki did not drop to 0 on that day, but spiked up instead (and is still above the level during the experiment). Does anyone have an idea what is happening here?

(Source: Grafana)

SELECT LEFT(timestamp, 8) AS date,  SUM(IF(wiki = 'huwiki',1,0)) AS hu_events, SUM(IF(wiki = 'itwiki',1,0)) AS it_events, SUM(IF(wiki = 'ruwiki',1,0)) AS ru_events FROM log.Popups_15906495 GROUP BY date ORDER BY date;
+----------+-----------+-----------+-----------+
| date     | hu_events | it_events | ru_events |
+----------+-----------+-----------+-----------+
| 20160922 |     22373 |       190 |       874 |
| 20160923 |    252265 |    350540 |    456066 |
| 20160924 |    254901 |    390318 |    643459 |
| 20160925 |    324654 |    426799 |    791504 |
| 20160926 |    334940 |    567353 |    820562 |
| 20160927 |    324204 |    558794 |    833451 |
| 20160928 |    321263 |    542015 |    802483 |
| 20160929 |    318069 |    567851 |    787320 |
| 20160930 |    270345 |    499840 |    715921 |
| 20161001 |    256650 |    432035 |    649106 |
| 20161002 |    340225 |    447202 |    795038 |
| 20161003 |    378865 |    589536 |    850620 |
| 20161004 |    372057 |    608689 |    868430 |
| 20161005 |    360868 |    598153 |    876730 |
| 20161006 |    351156 |    575780 |    869818 |
| 20161007 |    284766 |    545600 |    794498 |
| 20161008 |    290617 |    464138 |    699583 |
| 20161009 |    359913 |    496480 |    879238 |
| 20161010 |    358192 |    613233 |    897591 |
| 20161011 |    379136 |    588225 |    883906 |
| 20161012 |    361303 |    537058 |    873012 |
| 20161013 |    334339 |    620592 |    859166 |
| 20161014 |    295078 |    545790 |    797310 |
| 20161015 |    291274 |    412622 |    745353 |
| 20161016 |    371009 |    462605 |    918903 |
| 20161017 |    395491 |    613814 |    904706 |
| 20161018 |    389665 |    604258 |    882674 |
| 20161019 |    366972 |    609495 |    908835 |
| 20161020 |    350543 |    591556 |    859973 |
| 20161021 |    324606 |    551238 |    808875 |
| 20161022 |    299873 |    398239 |    739978 |
| 20161023 |    429062 |    480075 |    897525 |
| 20161024 |    377297 |    586267 |    924990 |
| 20161025 |    245068 |    373090 |    505130 |
| 20161026 |    237707 |    325321 |    481555 |
| 20161027 |    212735 |    314426 |    458149 |
| 20161028 |    178487 |    298161 |    441065 |
| 20161029 |    158383 |    208532 |    362883 |
| 20161030 |    176043 |    230485 |    400665 |
| 20161031 |    176074 |    269808 |    502976 |
| 20161101 |    218963 |    276659 |    459230 |
| 20161102 |    213715 |    351468 |    457224 |
| 20161103 |    203780 |    337219 |    434363 |
| 20161104 |    194891 |    307948 |    411688 |
| 20161105 |    184705 |    241223 |    360731 |
| 20161106 |    245406 |    268756 |    410224 |
| 20161107 |    254943 |    326759 |    510960 |
| 20161108 |    243133 |    382894 |    533145 |
| 20161109 |    250315 |    370752 |    596424 |
| 20161110 |    222688 |    332372 |    519704 |
| 20161111 |    164748 |    253374 |    329380 |
| 20161112 |    163069 |    183770 |    266660 |
| 20161113 |    188084 |    198105 |    327085 |
| 20161114 |    218525 |    284719 |    333396 |
| 20161115 |    184172 |    253395 |    310091 |
| 20161116 |    182185 |    241378 |    304051 |
| 20161117 |    177384 |    255444 |    320599 |
| 20161118 |    148287 |    241122 |    326730 |
| 20161119 |    146392 |    186191 |    266068 |
| 20161120 |    179485 |    187592 |    314642 |
| 20161121 |    197307 |    270744 |    364417 |
| 20161122 |    182042 |    245283 |    356413 |
| 20161123 |    173449 |    227899 |    331687 |
| 20161124 |    149255 |    206639 |    286691 |
| 20161125 |    124820 |    190420 |    254125 |
| 20161126 |    134671 |    172719 |    269824 |
| 20161127 |    175692 |    188738 |    295870 |
| 20161128 |    173118 |    410525 |    684143 |
| 20161129 |    169018 |   1143970 |   2538553 |
| 20161130 |    160289 |    886354 |   1626406 |
| 20161201 |    194309 |    849213 |   1633035 |
| 20161202 |    161196 |    656357 |    652790 |
+----------+-----------+-----------+-----------+
72 rows in set (2 min 10.43 sec)

ovasileva moved this task from Ready for Signoff to Needs More Work on the Reading-Web-Sprint-86-🔪🦃 board.Dec 5 2016, 10:51 AM

MBinder_WMF added a project: Reading-Web-Sprint-87-♨️😭.Dec 5 2016, 5:12 PM

Jdlrobson moved this task from Needs Analysis to Needs More Work on the Reading-Web-Sprint-87-♨️😭 board.Dec 5 2016, 6:14 PM

Russian Wikipedia was sampled at 0.01 and Italian Wikipedia at 0.02. Upon disabling this experiment the sample rate became 0.1 (e.g. 5 times as more on Italian, 10 times more on Russian).
Although we disabled the A/B test, we did not disable the beta feature, so now we are logging events for all beta feature users.
On top of this, it's likely that there are cache considerations. Any anonymous users who were bucketed and are visiting pages that were generated before the deploy will still see popups until that page is edited or purged.
I don't know how trustworthy the preferences page is, but that suggests Italian Wikipedia has 2 495 users and Russian has 1... ? (!?!) .
It doesn't surprise me the spike went up, but I'd expect it to consistently drop over next 2 weeks and then it will account for 0.1 of all beta feature users.

Jdlrobson moved this task from Needs More Work to Ready for Signoff on the Reading-Web-Sprint-87-♨️😭 board.Dec 5 2016, 9:35 PM

@Tbayer can you show a more recent graph?

@Tbayer specificially we'd like to answer these questions:
#1 For those events are the users logged in or not?
#2 What is the number of distinct session ids? Does this match with 10% of the number of beta feature users?

In T144490#2851243, @Jdlrobson wrote:

@Tbayer can you show a more recent graph?

Here's a link to a current one on Grafana: https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?var-schema=Popups&from=now-30d&to=now (one can always change the timespan by clicking on the top right)

In T144490#2851252, @Jdlrobson wrote:

@Tbayer specificially we'd like to answer these questions:
#1 For those events are the users logged in or not?

The vast majority (>98%) of events come from anonymous users, before and after Nov 28. The spike is due to anons only, the number of logged-in events stayed roughly the same. See detailed data below.

#2 What is the number of distinct session ids?

See detailed daily list below (similar takeaway: >99% of sessions are from anons, before and after)

Does this match with 10% of the number of beta feature users?

The logged-in events maybe, depending on how many of these beta feature users are active per day. E.g. for Dec 5, it would mean that around 8% of them were active on itwiki (= 19 / (2495*0.10) ), and 7% on ruwiki (= 68 / (9480 * 0.10)).

But not the overall events.

In T144490#2848870, @Jdlrobson wrote:

I don't know how trustworthy the preferences page is, but that suggests Italian Wikipedia has 2 495 users and Russian has 1... ? (!?!) .

I'm seeing "9,480 users are trying this feature" in my ruwiki preferences (BTW, used ULS to switch my user interface language to English there). We could check with a separate database query, but I would assume the number is by and large valid. A more important reason to do such a query would be to restrict that count to active users (editors) as it was done for skin usage in T147696.

It doesn't surprise me the spike went up, but I'd expect it to consistently drop over next 2 weeks and then it will account for 0.1 of all beta feature users.

Considering the above, I don't think we have a good explanation yet why the spike happened.

Data:

# events per day
SELECT LEFT(timestamp, 8) AS date,  
SUM(IF(wiki = 'itwiki' AND event_isAnon = 0,1,0)) AS it_loggedin, 
SUM(IF(wiki = 'itwiki' AND event_isAnon = 1,1,0)) AS it_anon, 
SUM(IF(wiki = 'ruwiki' AND event_isAnon = 0,1,0)) AS ru_loggedin, 
SUM(IF(wiki = 'ruwiki' AND event_isAnon = 1,1,0)) AS ru_anon
FROM log.Popups_15906495 
GROUP BY date ORDER BY date;
+----------+-------------+---------+-------------+---------+
| date     | it_loggedin | it_anon | ru_loggedin | ru_anon |
+----------+-------------+---------+-------------+---------+
| 20160922 |          81 |     109 |         513 |     361 |
| 20160923 |        5153 |  345387 |        6760 |  449306 |
| 20160924 |        3066 |  387252 |        6267 |  637192 |
| 20160925 |        4855 |  421944 |       12037 |  779467 |
| 20160926 |        5504 |  561849 |        4244 |  816318 |
| 20160927 |        5699 |  553095 |        4773 |  828678 |
| 20160928 |        5940 |  536075 |        5963 |  796520 |
| 20160929 |        6009 |  561842 |        7202 |  780118 |
| 20160930 |        6587 |  493253 |        5191 |  710730 |
| 20161001 |        3853 |  428182 |        4638 |  644468 |
| 20161002 |        7042 |  440160 |        3792 |  791246 |
| 20161003 |        5880 |  583656 |        4993 |  845627 |
| 20161004 |        6712 |  601977 |        3599 |  864831 |
| 20161005 |        6182 |  591971 |        5998 |  870732 |
| 20161006 |        8875 |  566905 |        9498 |  860320 |
| 20161007 |        5486 |  540114 |        8034 |  786464 |
| 20161008 |        3939 |  460199 |        5180 |  694403 |
| 20161009 |        4871 |  491609 |        9249 |  869989 |
| 20161010 |        7390 |  605843 |        7681 |  889910 |
| 20161011 |        4117 |  584108 |        7882 |  876024 |
| 20161012 |        4710 |  532348 |       13004 |  860008 |
| 20161013 |        7204 |  613388 |       12974 |  846192 |
| 20161014 |        5523 |  540267 |        8820 |  788490 |
| 20161015 |        5021 |  407601 |        9073 |  736280 |
| 20161016 |        5265 |  457340 |        9898 |  909005 |
| 20161017 |        3686 |  610128 |        6723 |  897983 |
| 20161018 |        8630 |  595628 |        4632 |  878042 |
| 20161019 |        8020 |  601475 |        7906 |  900929 |
| 20161020 |        6339 |  585217 |        5146 |  854827 |
| 20161021 |        5927 |  545311 |        7268 |  801607 |
| 20161022 |        5981 |  392258 |        5339 |  734639 |
| 20161023 |        5897 |  474178 |        9549 |  887976 |
| 20161024 |        5929 |  580338 |        5439 |  919551 |
| 20161025 |        4107 |  368983 |        3520 |  501610 |
| 20161026 |        1407 |  323914 |        4498 |  477057 |
| 20161027 |        2460 |  311966 |        3565 |  454584 |
| 20161028 |        2597 |  295564 |        4523 |  436542 |
| 20161029 |        2102 |  206430 |        3020 |  359863 |
| 20161030 |        2835 |  227650 |        6867 |  393798 |
| 20161031 |        3144 |  266664 |        3527 |  499449 |
| 20161101 |        3196 |  273463 |        1810 |  457420 |
| 20161102 |        3499 |  347969 |        5629 |  451595 |
| 20161103 |        2868 |  334351 |        4630 |  429733 |
| 20161104 |        2146 |  305802 |        4630 |  407058 |
| 20161105 |        3578 |  237645 |        2319 |  358412 |
| 20161106 |        1509 |  267247 |        2580 |  407644 |
| 20161107 |        2257 |  324502 |        4440 |  506520 |
| 20161108 |        2502 |  380392 |        4737 |  528408 |
| 20161109 |        2221 |  368531 |        3147 |  593277 |
| 20161110 |        2727 |  329645 |        5722 |  513982 |
| 20161111 |        2715 |  250659 |        1719 |  327661 |
| 20161112 |        1881 |  181889 |        2292 |  264368 |
| 20161113 |        3034 |  195071 |        2021 |  325064 |
| 20161114 |        2110 |  282609 |        2468 |  330928 |
| 20161115 |        1975 |  251420 |        1666 |  308425 |
| 20161116 |        2001 |  239377 |        2731 |  301320 |
| 20161117 |        1315 |  254129 |        3323 |  317276 |
| 20161118 |        4499 |  236623 |        2580 |  324150 |
| 20161119 |        2121 |  184070 |        1700 |  264368 |
| 20161120 |        1597 |  185995 |        3134 |  311508 |
| 20161121 |        2306 |  268438 |        3961 |  360456 |
| 20161122 |        1674 |  243609 |        2457 |  353956 |
| 20161123 |        1560 |  226339 |        2807 |  328880 |
| 20161124 |        1244 |  205395 |        1999 |  284692 |
| 20161125 |        1366 |  189054 |        1783 |  252342 |
| 20161126 |        2152 |  170567 |        1173 |  268651 |
| 20161127 |        1616 |  187122 |        5715 |  290155 |
| 20161128 |        1430 |  409095 |        2135 |  682008 |
| 20161129 |         329 | 1143641 |        1612 | 2536941 |
| 20161130 |         447 |  885907 |        1998 | 1624408 |
| 20161201 |         790 |  848423 |        3125 | 1629910 |
| 20161202 |         475 |  660008 |        3044 |  651932 |
| 20161203 |        1002 |  395483 |        3312 |  315228 |
| 20161204 |         464 |  308050 |        5015 |  308499 |
| 20161205 |        1257 |  274383 |        3706 |  246538 |
| 20161206 |        1405 |   15548 |        6143 |  166406 |
+----------+-------------+---------+-------------+---------+
76 rows in set (8 min 44.92 sec)

# sessions per day
SELECT LEFT(timestamp, 8) AS date,  
COUNT(DISTINCT (CASE WHEN wiki = 'itwiki' AND event_isAnon = 0 THEN event_sessionToken END)) AS it_loggedin,
COUNT(DISTINCT (CASE WHEN wiki = 'itwiki' AND event_isAnon = 1 THEN event_sessionToken END)) AS it_anon,
COUNT(DISTINCT (CASE WHEN wiki = 'ruwiki' AND event_isAnon = 0 THEN event_sessionToken END)) AS ru_loggedin,
COUNT(DISTINCT (CASE WHEN wiki = 'ruwiki' AND event_isAnon = 1 THEN event_sessionToken END)) AS ru_anon
FROM log.Popups_15906495 
GROUP BY date ORDER BY date;
+----------+-------------+---------+-------------+---------+
| date     | it_loggedin | it_anon | ru_loggedin | ru_anon |
+----------+-------------+---------+-------------+---------+
| 20160922 |           6 |      32 |          17 |      71 |
| 20160923 |          88 |   20050 |         114 |   23835 |
| 20160924 |          68 |   18503 |          88 |   28409 |
| 20160925 |          85 |   20489 |         108 |   34131 |
| 20160926 |          98 |   29339 |          82 |   39909 |
| 20160927 |          87 |   28561 |          99 |   39663 |
| 20160928 |         101 |   28585 |          79 |   39877 |
| 20160929 |          98 |   28837 |         100 |   39301 |
| 20160930 |          85 |   25844 |          95 |   35318 |
| 20161001 |          83 |   20229 |          65 |   28499 |
| 20161002 |          75 |   21614 |          87 |   33999 |
| 20161003 |         110 |   29761 |          97 |   40117 |
| 20161004 |         100 |   30409 |          91 |   40158 |
| 20161005 |         112 |   29738 |          84 |   40103 |
| 20161006 |          97 |   28837 |          75 |   39442 |
| 20161007 |          96 |   26823 |          90 |   36155 |
| 20161008 |          83 |   20001 |          86 |   29941 |
| 20161009 |          72 |   22688 |          92 |   36281 |
| 20161010 |         100 |   30179 |         109 |   40861 |
| 20161011 |          97 |   29953 |         109 |   40677 |
| 20161012 |          87 |   28193 |         104 |   39893 |
| 20161013 |         104 |   30196 |         111 |   39413 |
| 20161014 |          98 |   27916 |         106 |   36807 |
| 20161015 |          94 |   20145 |          93 |   30239 |
| 20161016 |          98 |   21398 |          99 |   36747 |
| 20161017 |          89 |   30589 |         109 |   41445 |
| 20161018 |          96 |   30402 |          92 |   41215 |
| 20161019 |         115 |   30551 |         113 |   41061 |
| 20161020 |          97 |   29422 |          99 |   40241 |
| 20161021 |         100 |   27432 |         116 |   36877 |
| 20161022 |          83 |   20007 |          95 |   30867 |
| 20161023 |          86 |   23136 |         113 |   37182 |
| 20161024 |          99 |   32081 |         109 |   42345 |
| 20161025 |         122 |   29402 |          95 |   38464 |
| 20161026 |          96 |   27139 |          82 |   37142 |
| 20161027 |          89 |   24698 |          90 |   36487 |
| 20161028 |          74 |   24430 |          83 |   33969 |
| 20161029 |          78 |   16614 |          74 |   27448 |
| 20161030 |          84 |   17920 |          75 |   28735 |
| 20161031 |          88 |   21239 |          86 |   38524 |
| 20161101 |          83 |   21892 |         102 |   36874 |
| 20161102 |         111 |   28024 |         100 |   36257 |
| 20161103 |          79 |   26870 |          95 |   33909 |
| 20161104 |          75 |   25206 |          88 |   29924 |
| 20161105 |          78 |   18961 |          70 |   25931 |
| 20161106 |          76 |   19754 |          75 |   28671 |
| 20161107 |          76 |   25161 |          90 |   38311 |
| 20161108 |          87 |   30013 |         107 |   39893 |
| 20161109 |          82 |   27941 |         106 |   42750 |
| 20161110 |          93 |   26547 |         101 |   38578 |
| 20161111 |          79 |   26064 |          92 |   34989 |
| 20161112 |          74 |   18085 |          76 |   27277 |
| 20161113 |          71 |   18685 |          88 |   31042 |
| 20161114 |          96 |   28716 |          83 |   34160 |
| 20161115 |          94 |   25670 |          83 |   31720 |
| 20161116 |          93 |   24838 |          81 |   30287 |
| 20161117 |          77 |   25667 |          88 |   33203 |
| 20161118 |          97 |   25553 |          78 |   35691 |
| 20161119 |          65 |   18153 |          72 |   27071 |
| 20161120 |          75 |   18536 |          79 |   30515 |
| 20161121 |         109 |   29170 |          92 |   39469 |
| 20161122 |          78 |   25695 |          95 |   37765 |
| 20161123 |          79 |   24004 |         101 |   35898 |
| 20161124 |          77 |   22830 |          89 |   32106 |
| 20161125 |          69 |   21258 |          88 |   27503 |
| 20161126 |          76 |   17051 |          65 |   24607 |
| 20161127 |          68 |   18103 |          80 |   27677 |
| 20161128 |          56 |   39357 |          80 |   62152 |
| 20161129 |          11 |  101135 |          52 |  226643 |
| 20161130 |          19 |   82552 |          48 |  168260 |
| 20161201 |          19 |   90956 |          82 |  192779 |
| 20161202 |          17 |   75869 |          75 |   94438 |
| 20161203 |          14 |   46492 |          68 |   49834 |
| 20161204 |          13 |   37891 |          72 |   49504 |
| 20161205 |          19 |   36421 |          68 |   42207 |
| 20161206 |          23 |    2851 |          66 |   29591 |
+----------+-------------+---------+-------------+---------+
76 rows in set (1 hour 17 min 22.42 sec)

Oh, and contrary to what one might have expected, the spike is due to an increase in Hovercards ON events, whereas Hovercards OFF events dropped sharply around November 28. Either stopping the test had the opposite effect from the one intended, in that vastly *more* anons saw Hovercards after November 28, or something is very wrong with the instrumentation here.

# events per day, Hovercards ON vs OFF
SELECT LEFT(timestamp, 8) AS date,  
SUM(IF(wiki = 'itwiki' AND event_popupEnabled = 0,1,0)) AS it_OFF, 
SUM(IF(wiki = 'itwiki' AND event_popupEnabled = 1,1,0)) AS it_ON, 
SUM(IF(wiki = 'ruwiki' AND event_popupEnabled = 0,1,0)) AS ru_OFF,
SUM(IF(wiki = 'ruwiki' AND event_popupEnabled = 1,1,0)) AS ru_ON
FROM log.Popups_15906495 
GROUP BY date ORDER BY date;

+----------+--------+---------+--------+---------+
| date     | it_OFF | it_ON   | ru_OFF | ru_ON   |
+----------+--------+---------+--------+---------+
| 20160922 |     88 |     102 |     19 |     855 |
| 20160923 | 282340 |   68200 | 409713 |   46353 |
| 20160924 | 322640 |   67678 | 588126 |   55333 |
| 20160925 | 353194 |   73605 | 711140 |   80364 |
| 20160926 | 466975 |  100378 | 746295 |   74267 |
| 20160927 | 456009 |  102785 | 754385 |   79066 |
| 20160928 | 449058 |   92957 | 724661 |   77822 |
| 20160929 | 466210 |  101641 | 718848 |   68472 |
| 20160930 | 410038 |   89802 | 655910 |   60011 |
| 20161001 | 355739 |   76296 | 587642 |   61464 |
| 20161002 | 364457 |   82745 | 723302 |   71736 |
| 20161003 | 484446 |  105090 | 776568 |   74052 |
| 20161004 | 500194 |  108495 | 787834 |   80596 |
| 20161005 | 491123 |  107030 | 795850 |   80880 |
| 20161006 | 471291 |  104489 | 790335 |   79483 |
| 20161007 | 447215 |   98385 | 722652 |   71846 |
| 20161008 | 380825 |   83313 | 637511 |   62072 |
| 20161009 | 402972 |   93508 | 794949 |   84289 |
| 20161010 | 505192 |  108041 | 817552 |   80039 |
| 20161011 | 478768 |  109457 | 801848 |   82058 |
| 20161012 | 438821 |   98237 | 784054 |   88958 |
| 20161013 | 506178 |  114414 | 770842 |   88324 |
| 20161014 | 448170 |   97620 | 719785 |   77525 |
| 20161015 | 336016 |   76606 | 677021 |   68332 |
| 20161016 | 384429 |   78176 | 829265 |   89638 |
| 20161017 | 506462 |  107352 | 822117 |   82589 |
| 20161018 | 501453 |  102805 | 810544 |   72130 |
| 20161019 | 498234 |  111261 | 823217 |   85618 |
| 20161020 | 482211 |  109345 | 782828 |   77145 |
| 20161021 | 446587 |  104651 | 733371 |   75504 |
| 20161022 | 328975 |   69264 | 678148 |   61830 |
| 20161023 | 388424 |   91651 | 817301 |   80224 |
| 20161024 | 479649 |  106618 | 839609 |   85381 |
| 20161025 | 304018 |   69072 | 455825 |   49305 |
| 20161026 | 265477 |   59844 | 436435 |   45120 |
| 20161027 | 253465 |   60961 | 419581 |   38568 |
| 20161028 | 240005 |   58156 | 402473 |   38592 |
| 20161029 | 166323 |   42209 | 331942 |   30941 |
| 20161030 | 189454 |   41031 | 363379 |   37286 |
| 20161031 | 223210 |   46598 | 454544 |   48432 |
| 20161101 | 224480 |   52179 | 419698 |   39532 |
| 20161102 | 287646 |   63822 | 414685 |   42539 |
| 20161103 | 274127 |   63092 | 393537 |   40826 |
| 20161104 | 254555 |   53393 | 377000 |   34688 |
| 20161105 | 196905 |   44318 | 328343 |   32388 |
| 20161106 | 222970 |   45786 | 373168 |   37056 |
| 20161107 | 267494 |   59265 | 464035 |   46925 |
| 20161108 | 315502 |   67392 | 482576 |   50569 |
| 20161109 | 302701 |   68051 | 542900 |   53524 |
| 20161110 | 273856 |   58516 | 473871 |   45833 |
| 20161111 | 197998 |   55376 | 287696 |   41684 |
| 20161112 | 137607 |   46163 | 232696 |   33964 |
| 20161113 | 151071 |   47034 | 287357 |   39728 |
| 20161114 | 212238 |   72481 | 294195 |   39201 |
| 20161115 | 190461 |   62934 | 273159 |   36932 |
| 20161116 | 183706 |   57672 | 268446 |   35605 |
| 20161117 | 194840 |   60604 | 280576 |   40023 |
| 20161118 | 184223 |   56899 | 288318 |   38412 |
| 20161119 | 145459 |   40732 | 232705 |   33363 |
| 20161120 | 144206 |   43386 | 279648 |   34994 |
| 20161121 | 207422 |   63322 | 321752 |   42665 |
| 20161122 | 187839 |   57444 | 312105 |   44308 |
| 20161123 | 175787 |   52112 | 289838 |   41849 |
| 20161124 | 157419 |   49220 | 250418 |   36273 |
| 20161125 | 144027 |   46393 | 224990 |   29135 |
| 20161126 | 132235 |   40484 | 236860 |   32964 |
| 20161127 | 143300 |   45438 | 258675 |   37195 |
| 20161128 | 156610 |  253915 | 248911 |  435232 |
| 20161129 |   2347 | 1141623 |   3407 | 2535146 |
| 20161130 |   1494 |  884860 |   1912 | 1624494 |
| 20161201 |    649 |  848564 |   2277 | 1630758 |
| 20161202 |    848 |  659635 |   1379 |  653597 |
| 20161203 |    459 |  396026 |    637 |  317903 |
| 20161204 |    152 |  308362 |    411 |  313103 |
| 20161205 |    200 |  275440 |    594 |  249650 |
| 20161206 |      7 |   17234 |   1826 |  173669 |
| 20161207 |      0 |     402 |     57 |    9854 |
+----------+--------+---------+--------+---------+
77 rows in set (6 min 55.65 sec)

CKoerner_WMF subscribed.Dec 7 2016, 4:17 PM

TL;DR: When we disabled the experiment, we enabled Page Previews for all anons viewing pages that hadn't been touched (evicted from – at least – our Varnishes). Anons will continue to see previews for links on a page for as long as that page is cached (IIRC 14 days).

If the experiment is enabled, we instruct the Resource Loader to always load the Page Previews code. This instruction is generated on the server and embedded in the response and, consequently, is cached in our Varnishes. After the experiment has been disabled, those cached pages aren't automatically evicted. We have to wait for the page to be touched so that the cached response is evicted and a new response without the instruction to always load the Page Previews code is cached.

Now, during initialization, Page Previews checks whether previews should be shown to the user in [the mw.popups.getEnabledState method](https://github.com/wikimedia/mediawiki-extensions-Popups/blob/master/resources/ext.popups.core/ext.popups.core.js#L242-L248). Since the experiment is disabled, it'll check whether the user has not disabled Page Previews:

return mw.storage.get( 'mwe-popups-enabled' ) !== '0';

If a user hasn't explicitly enabled or disabled Page Previews via the settings modal, then their storage will not have this key present. Herein lay the bug: when a key isn't present in the storage, mw.storage.get will return null and null is strictly not equal to '0', i.e. the line above will evaluate to true and the user will be shown previews.

This bug would not have occurred if the line above was written as follows:

return mw.storage.get( 'mwe-popups-enabled' ) === '1';

Lessons

[The unit tests for mw.popups.getEnabledState](https://github.com/wikimedia/mediawiki-extensions-Popups/blob/master/tests/qunit/ext.popups.core.test.js#L320-L327) and it's DocBlock are quite clear about this behaviour. When we have a "Disable $feature" task, we need to ask ourselves "What happens when we actually disable it?" during planning/estimation.