Analyze results of stage 0 page previews test
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ovasileva
	Feb 15 2017, 11:31 AM

Description

Refer to https://www.mediawiki.org/wiki/Beta_Features/Hovercards/2016_A/B_Tests and T139319: Analyze results of Hovercards A/B test - Hungarian for metrics (plus additional metrics)

Begin analyzing data on July 19 (1 week after the duplicates issue was fixed, per T170018#3424998; with all other known bugs having been resolved earlier, except T167365: Check whether a 'disabled' event is sent when the user disables the feature and fix if the event is not sent)

Repeat analysis done for T139319: Analyze results of Hovercards A/B test - Hungarian:

cards per page
cards per session
disable rates
back button
pageviews (on vs off)

Related Objects
Search...

Status	Assigned	Task
Resolved	Dereckson	T68374 Enable Hovercards on se.wikimedia.org (Swedish chapter wiki)
Resolved	Jdlrobson	T70860 [GOAL] Graduate Page Previews feature (Popups extension) out of Beta Feature
Resolved	ovasileva	T154635 [EPIC] Deploy page previews to English and German Wikipedia
Resolved	ovasileva	T162672 Deploy page previews to 90% of users on all wikis but English and German
Resolved	• Tbayer	T158172 Analyze results of stage 0 page previews test
Resolved	• Tbayer	T167273 pageLoaded events are logged infrequently, if at all
Resolved	• Tbayer	T164256 pageTitleHover not set correctly in Popups schema
Resolved	• Tbayer	T165461 Various sanity checks for data from the rewritten Popups instrumentation
Resolved	• Tbayer	T166323 perceivedWait value should be logged for all card interaction-ending events
Resolved	phuedx	T167365 Check whether a 'disabled' event is sent when the user disables the feature and fix if the event is not sent
Resolved	Jdlrobson	T169365 There is no possibility to check changes on user options save.
Resolved	• Tbayer	T167391 [Spike] Isolate the source of Popups event duplication
Declined	• Tbayer	T169581 Histogram for the number of duplicated events per browser session
Declined	• Tbayer	T169582 Examine several sessions which contained duplicate events
Resolved	phuedx	T170018 Duplicate events sent in Firefox after back button press
Resolved	• Tbayer	T167236 userSessionToken in RelatedArticles schema does not seem to survive beyond one pageview
Resolved	pmiazga	T159490 PagePreviews events system sends duplicate and incorrect events
Resolved	• Tbayer	T161769 Schema:Popups sends extraneous link interaction events in control condition
Resolved	phuedx	T159539 [2 hours] PagePreviews `totalInteractionTime` for `opened` event is calculated incorrectly
Resolved	pmiazga	T162924 PagePreviews sending "dwelledButAbandoned" after "opened"
Resolved	phuedx	T168449 event.linkInteractionToken can be undefined
Resolved	ovasileva	T149947 [Spike 1hr] Double-check through old hovercards EL related bugs
Resolved	phuedx	T140485 Schema:Popups logs "dwelledButAbandoned" and link open events at the same time
Resolved	phuedx	T173422 Investigate the increase in the number of requests to Swift after the Page Previews deploy

Event Timeline

ovasileva created this task.Feb 15 2017, 11:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2017, 11:31 AM

ovasileva mentioned this in T139319: Analyze results of Hovercards A/B test - Hungarian.Feb 15 2017, 11:32 AM

ovasileva triaged this task as High priority.Feb 16 2017, 4:38 PM

ovasileva moved this task from Incoming to Product Owner Backlog on the Web-Team-Backlog board.

ovasileva moved this task from 2016-17-Q3-Goal to Next Up on the Page-Previews board.Apr 13 2017, 7:20 PM

ovasileva edited projects, added Page-Previews; removed Page-Previews (2016-17-Q3-Goal).

@ovasileva this is high priority but hasn't happened since April.. or has it and we just forgot to update the card? cc @Tbayer

@Jdlrobson - it is indeed still high priority as it is a blocker for any future deployment. Adding it as a subtask to the deployment task and marking as stalled due to various eventlogging bugs.

ovasileva changed the task status from Open to Stalled.Jun 7 2017, 10:53 AM

ovasileva added a parent task: T162672: Deploy page previews to 90% of users on all wikis but English and German.

@Jdlrobson: Yeah, we haven't been very disciplined about updating the status of this task and tracking the blocking EL bugs here; that conversation has been happening elsewhere. I'll add some right now.

• Tbayer added subtasks: T167273: pageLoaded events are logged infrequently, if at all, T164256: pageTitleHover not set correctly in Popups schema, T165461: Various sanity checks for data from the rewritten Popups instrumentation, T166323: perceivedWait value should be logged for all card interaction-ending events, T161769: Schema:Popups sends extraneous link interaction events in control condition.Jun 7 2017, 10:58 PM

• Tbayer added a subtask: T167365: Check whether a 'disabled' event is sent when the user disables the feature and fix if the event is not sent.Jun 7 2017, 11:39 PM

phuedx mentioned this in T162672: Deploy page previews to 90% of users on all wikis but English and German.Jun 8 2017, 9:35 AM

Jdlrobson added a subtask: T167391: [Spike] Isolate the source of Popups event duplication.Jun 8 2017, 9:28 PM

• Tbayer closed subtask T164256: pageTitleHover not set correctly in Popups schema as Resolved.Jun 13 2017, 10:14 PM

• Tbayer closed subtask T166323: perceivedWait value should be logged for all card interaction-ending events as Resolved.Jun 13 2017, 10:39 PM

• Tbayer closed subtask T167273: pageLoaded events are logged infrequently, if at all as Resolved.Jul 4 2017, 5:22 PM

Jdlrobson added a parent task: T154635: [EPIC] Deploy page previews to English and German Wikipedia.Jul 12 2017, 3:35 PM

(Added T167236: userSessionToken in RelatedArticles schema does not seem to survive beyond one pageview as as past blocker here, because it involved a fix to the Popups instrumentation too: T167236#3347801 )

• Tbayer added a subtask: T159490: PagePreviews events system sends duplicate and incorrect events.Jul 15 2017, 1:29 AM

• Tbayer updated the task description. (Show Details)Jul 15 2017, 1:57 AM

• Tbayer mentioned this in T165461: Various sanity checks for data from the rewritten Popups instrumentation.Jul 15 2017, 2:00 AM

• Tbayer added a subtask: T168449: event.linkInteractionToken can be undefined.Jul 15 2017, 4:34 AM

• Tbayer added a subtask: T149947: [Spike 1hr] Double-check through old hovercards EL related bugs.Jul 16 2017, 7:08 PM

• Tbayer moved this task from Triage to Blocked on the Reading-analysis board.Jul 16 2017, 7:16 PM

• Tbayer changed the task status from Stalled to Open.Jul 16 2017, 7:33 PM

ovasileva closed subtask T165461: Various sanity checks for data from the rewritten Popups instrumentation as Resolved.Jul 19 2017, 5:39 PM

ovasileva mentioned this in T161769: Schema:Popups sends extraneous link interaction events in control condition.

ovasileva closed subtask T161769: Schema:Popups sends extraneous link interaction events in control condition as Resolved.

ovasileva mentioned this in T167391: [Spike] Isolate the source of Popups event duplication.

ovasileva closed subtask T167391: [Spike] Isolate the source of Popups event duplication as Resolved.

Liuxinyu970226 subscribed.Jul 19 2017, 11:00 PM

• Tbayer moved this task from Blocked to In progress on the Reading-analysis board.Jul 21 2017, 5:04 AM

• Tbayer updated the task description. (Show Details)Jul 21 2017, 5:06 AM

• Tbayer updated the task description. (Show Details)Jul 21 2017, 5:08 AM

(work log)

I'm repeating last year's analysis for the final report where we used a daily time series to assess the variation in the metrics, e.g. to compare the number of pageviews per session between previews ON/OFF. However, it looks like the decreased sample rate combined with our decision to shorten the timespan to one week (instead of six weeks as last time), and perhaps also the sampling fix made recently in connection with the RelatedArticles issue, is creating a problem here, with too few sessions on all but the largest wikis in the test (ruwiki, itwiki) to make a meaningful recreation of last year's chart, and even there too sparse to be confident (50-100/day in control on ruwiki).

Will discuss an increase of the sampling rate - the schema's event rate is still well in the safe zone currently.

SELECT
wiki,
LEFT(timestamp, 8) AS date,
event_popupEnabled AS popupEnabled,
COUNT(DISTINCT event_pageToken)/COUNT(DISTINCT event_sessionToken) AS page_tokens_per_session,
COUNT(DISTINCT event_sessionToken) AS sessions
FROM log.Popups_16364296
WHERE wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki') # https://gerrit.wikimedia.org/r/#/c/343675/
AND event_isAnon = 1
AND LEFT(timestamp, 8) >= '20170713'
AND LEFT(timestamp, 8) < '20170721'
AND SUBSTRING( userAgent, INSTR(userAgent,'"browser_family": "')+19, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "')+19), '"' )-1 ) = 'Firefox'
AND event_action = 'pageLoaded'
GROUP BY wiki, date, event_popupEnabled
ORDER BY wiki, date, event_popupEnabled;

+--------+----------+--------------+-------------------------+----------+
| wiki   | date     | popupEnabled | page_tokens_per_session | sessions |
+--------+----------+--------------+-------------------------+----------+
| cawiki | 20170713 |            0 |                  1.0000 |        1 |
| cawiki | 20170713 |            1 |                  1.0000 |        7 |
| cawiki | 20170714 |            0 |                  1.0000 |        2 |
| cawiki | 20170714 |            1 |                  1.0000 |        5 |
| cawiki | 20170715 |            1 |                  1.0000 |        2 |
| cawiki | 20170716 |            1 |                  1.7143 |        7 |
| cawiki | 20170717 |            1 |                  1.1875 |       16 |
| cawiki | 20170718 |            0 |                  1.0000 |        3 |
| cawiki | 20170718 |            1 |                  1.2727 |       11 |
| cawiki | 20170719 |            0 |                  1.0000 |        2 |
| cawiki | 20170719 |            1 |                  1.5833 |       12 |
| cawiki | 20170720 |            1 |                  1.0000 |        8 |
| elwiki | 20170713 |            0 |                  1.0000 |        1 |
| elwiki | 20170713 |            1 |                  1.6667 |       27 |
| elwiki | 20170714 |            0 |                  1.0000 |        2 |
| elwiki | 20170714 |            1 |                  1.2667 |       30 |
| elwiki | 20170715 |            0 |                  1.0000 |        3 |
| elwiki | 20170715 |            1 |                  1.1429 |       28 |
| elwiki | 20170716 |            0 |                  3.5000 |        2 |
| elwiki | 20170716 |            1 |                  1.4500 |       20 |
| elwiki | 20170717 |            0 |                  1.2000 |        5 |
| elwiki | 20170717 |            1 |                  1.3333 |       33 |
| elwiki | 20170718 |            1 |                  2.7931 |       29 |
| elwiki | 20170719 |            0 |                  1.0000 |        2 |
| elwiki | 20170719 |            1 |                  1.4194 |       31 |
| elwiki | 20170720 |            0 |                  3.4000 |        5 |
| elwiki | 20170720 |            1 |                  1.3200 |       25 |
| hewiki | 20170713 |            0 |                  1.0000 |        1 |
| hewiki | 20170713 |            1 |                  1.3333 |        9 |
| hewiki | 20170714 |            0 |                  1.3333 |        3 |
| hewiki | 20170714 |            1 |                  1.4286 |        7 |
| hewiki | 20170715 |            0 |                  1.0000 |        1 |
| hewiki | 20170715 |            1 |                  1.1333 |       15 |
| hewiki | 20170716 |            1 |                  1.3846 |       13 |
| hewiki | 20170717 |            1 |                  1.7500 |       16 |
| hewiki | 20170718 |            0 |                  1.0000 |        1 |
| hewiki | 20170718 |            1 |                  1.2667 |       15 |
| hewiki | 20170719 |            0 |                  1.0000 |        1 |
| hewiki | 20170719 |            1 |                  1.4444 |        9 |
| hewiki | 20170720 |            0 |                  1.0000 |        1 |
| hewiki | 20170720 |            1 |                  2.5455 |       11 |
| huwiki | 20170713 |            0 |                  1.0000 |        4 |
| huwiki | 20170713 |            1 |                  1.4286 |       49 |
| huwiki | 20170714 |            0 |                  1.1429 |        7 |
| huwiki | 20170714 |            1 |                  1.3673 |       49 |
| huwiki | 20170715 |            0 |                  1.3333 |        3 |
| huwiki | 20170715 |            1 |                  1.8364 |       55 |
| huwiki | 20170716 |            0 |                  1.3750 |        8 |
| huwiki | 20170716 |            1 |                  1.5254 |       59 |
| huwiki | 20170717 |            0 |                  1.8000 |        5 |
| huwiki | 20170717 |            1 |                  1.4074 |       54 |
| huwiki | 20170718 |            0 |                  1.3333 |        6 |
| huwiki | 20170718 |            1 |                  1.2361 |       72 |
| huwiki | 20170719 |            0 |                  1.1429 |        7 |
| huwiki | 20170719 |            1 |                  1.5185 |       54 |
| huwiki | 20170720 |            0 |                  1.1667 |        6 |
| huwiki | 20170720 |            1 |                  1.7797 |       59 |
| itwiki | 20170713 |            0 |                  3.0976 |       41 |
| itwiki | 20170713 |            1 |                  1.8577 |      281 |
| itwiki | 20170714 |            0 |                  1.8621 |       29 |
| itwiki | 20170714 |            1 |                  2.2068 |      295 |
| itwiki | 20170715 |            0 |                  1.4783 |       23 |
| itwiki | 20170715 |            1 |                  2.1298 |      208 |
| itwiki | 20170716 |            0 |                  2.0690 |       29 |
| itwiki | 20170716 |            1 |                  2.0564 |      195 |
| itwiki | 20170717 |            0 |                  1.5429 |       35 |
| itwiki | 20170717 |            1 |                  1.6648 |      364 |
| itwiki | 20170718 |            0 |                  1.5667 |       30 |
| itwiki | 20170718 |            1 |                  2.0606 |      330 |
| itwiki | 20170719 |            0 |                  4.7931 |       29 |
| itwiki | 20170719 |            1 |                  1.7982 |      342 |
| itwiki | 20170720 |            0 |                  1.6111 |       36 |
| itwiki | 20170720 |            1 |                  1.6667 |      288 |
| ruwiki | 20170713 |            0 |                  1.5200 |       75 |
| ruwiki | 20170713 |            1 |                  1.3720 |      715 |
| ruwiki | 20170714 |            0 |                  1.5263 |       76 |
| ruwiki | 20170714 |            1 |                  1.6000 |      665 |
| ruwiki | 20170715 |            0 |                  2.4576 |       59 |
| ruwiki | 20170715 |            1 |                  1.7126 |      494 |
| ruwiki | 20170716 |            0 |                  1.2656 |       64 |
| ruwiki | 20170716 |            1 |                  1.5010 |      509 |
| ruwiki | 20170717 |            0 |                  1.5978 |       92 |
| ruwiki | 20170717 |            1 |                  1.5444 |      698 |
| ruwiki | 20170718 |            0 |                  1.7273 |       77 |
| ruwiki | 20170718 |            1 |                  1.6259 |      695 |
| ruwiki | 20170719 |            0 |                  1.4800 |       75 |
| ruwiki | 20170719 |            1 |                  2.2725 |      734 |
| ruwiki | 20170720 |            0 |                  1.3659 |       82 |
| ruwiki | 20170720 |            1 |                  1.4539 |      705 |
+--------+----------+--------------+-------------------------+----------+
89 rows in set (26.73 sec)

Daily pageviews per session, previews ON vs. OFF, ruwiki (Firefox) 017-07-13...2017-07-20.png (371×600 px, 16 KB)

Compare last year's chart:

Daily pageviews per session, previews ON vs. OFF, ruwiki (Firefox) 2016 instrumentation.png (367×580 px, 43 KB)

• Tbayer mentioned this in T171325: Increase sampling rate of page previews to 1% on ruwiki, huwiki, itwiki.Jul 21 2017, 5:16 PM

(work log)
Here is the analogous chart for the "previews seen per pageviews" metric (corresponding to this chart from last year, but for a "seen" threshold of 1s instead of 1.5s). This too illustrates that the values vary too much at the current sampling rate to apply our usual time series heuristic, except for perhaps ruwiki and itwiki if we had a longer timespan:

Previews seen (>=1s) per pageview, Firefox, 2017-07-13..23.png (674×982 px, 80 KB)

I'm looking at other options including expanding to Chrome and using e.g. bootstrapping instead of the daily time series, but clearly the sample rate has been too low.

SELECT wiki,
LEFT(timestamp, 8) AS date,
COUNT(DISTINCT event_linkInteractionToken)/COUNT(DISTINCT event_sessionToken) AS cards_viewed_per_session,
COUNT(DISTINCT event_linkInteractionToken)/COUNT(DISTINCT event_pageToken) AS cards_viewed_per_page,
COUNT(DISTINCT event_sessionToken) AS sessions,
COUNT(DISTINCT event_pageToken) AS pageviews
FROM log.Popups_16364296
WHERE wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki') # https://gerrit.wikimedia.org/r/#/c/343675/
AND event_isAnon = 1
AND LEFT(timestamp, 8) >= '20170713'
AND LEFT(timestamp, 8) < '20170724'
AND SUBSTRING( userAgent, INSTR(userAgent,'"browser_family": "')+19, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "')+19), '"' )-1 ) = 'Firefox'
AND event_popupEnabled = 1
AND (event_action = 'pageLoaded'
OR (event_totalInteractionTime >= event_perceivedWait + 1000 ))  # 1s instead of 1.5s earlier
GROUP BY wiki, date, event_popupEnabled
ORDER BY wiki, date, event_popupEnabled;

phuedx closed subtask T167365: Check whether a 'disabled' event is sent when the user disables the feature and fix if the event is not sent as Resolved.Jul 25 2017, 8:44 AM

ovasileva updated the task description. (Show Details)Aug 9 2017, 6:50 PM

Update: Here is a look at the disable actions, which are still very rare.

SELECT LEFT(timestamp, 8) AS yearmonthday, COUNT(*) AS disables
FROM log.Popups_16364296
WHERE event_action = 'disabled'
GROUP BY yearmonthday
ORDER BY yearmonthday;

+--------------+----------+
| yearmonthday | disables |
+--------------+----------+
| 20170703     |        1 |
| 20170705     |        2 |
| 20170706     |        4 |
| 20170718     |        1 |
| 20170723     |        1 |
| 20170724     |        1 |
| 20170727     |        1 |
| 20170728     |        1 |
| 20170730     |        1 |
| 20170731     |        2 |
| 20170801     |        1 |
| 20170803     |        1 |
| 20170804     |        1 |
| 20170806     |        1 |
| 20170807     |        1 |
| 20170808     |        1 |
+--------------+----------+
16 rows in set (1 min 5.56 sec)

Update: While we're still sorting out the session-based metrics (also now in the light of T175918), here are histograms for per-pageview number of cards viewed, for the three largest wikis in the test:

Distribution of the number of previews seen >=1.5s per pageview (2017, stage 0).png (371×600 px, 15 KB)

Distribution of the number of previews seen >=1.5s per pageview (2017, stage 0), only >0 views.png (371×600 px, 16 KB)

SELECT wiki,
cards_1500viewed_on_page,
COUNT(*) AS frequency
FROM
(SELECT wiki,
event_pageToken,
COUNT(DISTINCT event_linkInteractionToken) AS cards_1500viewed_on_page
FROM log.Popups_16364296
WHERE
(event_action = 'pageLoaded'
OR (event_totalInteractionTime >= event_perceivedWait + 1500 ))
AND wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki')
AND event_isAnon = 1
AND event_popupEnabled = 1

AND LEFT(timestamp, 8) >= '20170725'
AND LEFT(timestamp, 8) < '20170822'
GROUP BY wiki, event_popupEnabled, event_pageToken) AS pageviewlist
GROUP BY wiki, cards_1500viewed_on_page
ORDER BY wiki, cards_1500viewed_on_page

[edit: fixed chart labels - as could be seen from the query, this was actually still the data for the previously used limit of 1.5s instead of 1s]

In T158172#3514807, @Tbayer wrote:

Update: Here is a look at the disable actions, which are still very rare.

SELECT LEFT(timestamp, 8) AS yearmonthday, COUNT(*) AS disables
FROM log.Popups_16364296
WHERE event_action = 'disabled'
GROUP BY yearmonthday
ORDER BY yearmonthday;

+--------------+----------+
| yearmonthday | disables |
+--------------+----------+
| 20170703     |        1 |
| 20170705     |        2 |
| 20170706     |        4 |
| 20170718     |        1 |
| 20170723     |        1 |
| 20170724     |        1 |
| 20170727     |        1 |
| 20170728     |        1 |
| 20170730     |        1 |
| 20170731     |        2 |
| 20170801     |        1 |
| 20170803     |        1 |
| 20170804     |        1 |
| 20170806     |        1 |
| 20170807     |        1 |
| 20170808     |        1 |
+--------------+----------+
16 rows in set (1 min 5.56 sec)

For comparison, note the total number of session per day, below. Caveat: the logging of disable events may very well have been affected by T175918 too.

SELECT LEFT(timestamp, 8) AS yearmonthday, event_popupenabled  AS enabled,
COUNT(DISTINCT event_sessionToken) AS sessions
FROM log.Popups_16364296
GROUP BY yearmonthday, enabled
ORDER BY yearmonthday, enabled
--------------

yearmonthday    enabled sessions
20170329        0       306
20170329        1       2077
20170330        0       1013
20170330        1       7447
20170331        0       692
20170331        1       5157
20170401        0       518
20170401        1       3919
20170402        0       594
20170402        1       4297
20170403        0       634
20170403        1       5258
20170404        0       740
20170404        1       5423
20170405        0       600
20170405        1       4829
20170406        0       682
20170406        1       5192
20170407        0       688
20170407        1       5157
20170408        0       561
20170408        1       4231
20170409        0       608
20170409        1       5033
20170410        0       746
20170410        1       5814
20170411        0       705
20170411        1       5555
20170412        0       708
20170412        1       5379
20170413        0       667
20170413        1       5336
20170414        0       685
20170414        1       5018
20170415        0       526
20170415        1       4226
20170416        0       547
20170416        1       4249
20170417        0       619
20170417        1       4826
20170418        0       777
20170418        1       6062
20170419        0       792
20170419        1       6185
20170420        0       722
20170420        1       5607
20170421        0       675
20170421        1       5174
20170422        0       558
20170422        1       4138
20170423        0       650
20170423        1       4871
20170424        0       697
20170424        1       5669
20170425        0       699
20170425        1       5581
20170426        0       711
20170426        1       5663
20170427        0       685
20170427        1       5320
20170428        0       680
20170428        1       5405
20170429        0       525
20170429        1       4081
20170430        0       574
20170430        1       4544
20170501        0       649
20170501        1       5403
20170502        0       827
20170502        1       5895
20170503        0       837
20170503        1       6265
20170504        0       826
20170504        1       5964
20170505        0       641
20170505        1       4903
20170506        0       501
20170506        1       3813
20170507        0       589
20170507        1       4571
20170508        0       750
20170508        1       5594
20170509        0       699
20170509        1       5717
20170510        0       770
20170510        1       6270
20170511        0       763
20170511        1       6133
20170512        0       677
20170512        1       5358
20170513        0       590
20170513        1       4613
20170514        0       665
20170514        1       5269
20170515        0       798
20170515        1       6332
20170516        0       757
20170516        1       6199
20170517        0       794
20170517        1       5870
20170518        0       716
20170518        1       5741
20170519        0       703
20170519        1       5276
20170520        0       588
20170520        1       4387
20170521        0       694
20170521        1       4946
20170522        0       777
20170522        1       5823
20170523        0       724
20170523        1       5567
20170524        0       655
20170524        1       5323
20170525        0       707
20170525        1       5123
20170526        0       589
20170526        1       4433
20170527        0       492
20170527        1       3513
20170528        0       524
20170528        1       3948
20170529        0       744
20170529        1       5634
20170530        0       707
20170530        1       5660
20170531        0       737
20170531        1       6004
20170601        0       705
20170601        1       5435
20170602        0       660
20170602        1       4936
20170603        0       550
20170603        1       4217
20170604        0       611
20170604        1       4868
20170605        0       775
20170605        1       5717
20170606        0       803
20170606        1       5842
20170607        0       681
20170607        1       5435
20170608        0       660
20170608        1       4975
20170609        0       682
20170609        1       5087
20170610        0       462
20170610        1       3769
20170611        0       536
20170611        1       3968
20170612        0       672
20170612        1       5406
20170613        0       734
20170613        1       5799
20170614        0       708
20170614        1       5338
20170615        0       689
20170615        1       5240
20170616        0       616
20170616        1       5003
20170617        0       523
20170617        1       3875
20170618        0       519
20170618        1       4131
20170619        0       707
20170619        1       5047
20170620        0       610
20170620        1       4841
20170621        0       674
20170621        1       4853
20170622        0       668
20170622        1       4910
20170623        0       568
20170623        1       4485
20170624        0       423
20170624        1       3366
20170625        0       492
20170625        1       3622
20170626        0       637
20170626        1       4967
20170627        0       696
20170627        1       5676
20170628        0       638
20170628        1       5168
20170629        0       585
20170629        1       5211
20170630        0       710
20170630        1       5389
20170701        0       491
20170701        1       4068
20170702        0       517
20170702        1       4171
20170703        0       606
20170703        1       5155
20170704        0       654
20170704        1       5120
20170705        0       657
20170705        1       5152
20170706        0       663
20170706        1       5156
20170707        0       578
20170707        1       4689
20170708        0       460
20170708        1       3433
20170709        0       514
20170709        1       3910
20170710        0       651
20170710        1       5111
20170711        0       649
20170711        1       5224
20170712        0       642
20170712        1       4912
20170713        0       618
20170713        1       4903
20170714        0       571
20170714        1       4392
20170715        0       443
20170715        1       3334
20170716        0       470
20170716        1       3679
20170717        0       623
20170717        1       5242
20170718        0       673
20170718        1       5381
20170719        0       595
20170719        1       5170
20170720        0       635
20170720        1       4944
20170721        0       630
20170721        1       4987
20170722        0       499
20170722        1       3732
20170723        0       462
20170723        1       3745
20170724        0       2804
20170724        1       22025
20170725        0       5385
20170725        1       43164
20170726        0       5650
20170726        1       44462
20170727        0       5527
20170727        1       43926
20170728        0       5592
20170728        1       45245
20170729        0       4107
20170729        1       33526
20170730        0       4612
20170730        1       36200
20170731        0       6007
20170731        1       48128
20170801        0       5581
20170801        1       44669
20170802        0       5193
20170802        1       41592
20170803        0       5342
20170803        1       42372
20170804        0       5291
20170804        1       43722
20170805        0       3885
20170805        1       30461
20170806        0       4004
20170806        1       31503
20170807        0       5376
20170807        1       43045
20170808        0       5564
20170808        1       45918
20170809        0       5204
20170809        1       41867
20170810        0       5000
20170810        1       40724
20170811        0       5263
20170811        1       42844
20170812        0       3846
20170812        1       30506
20170813        0       3977
20170813        1       31700
20170814        0       5062
20170814        1       40743
20170815        0       5449
20170815        1       44625
20170816        0       5632
20170816        1       46444
20170817        0       5712
20170817        1       46553
20170818        0       5487
20170818        1       45069
20170819        0       4319
20170819        1       34533
20170820        0       4408
20170820        1       34860
20170821        0       5756
20170821        1       47235
20170822        0       5989
20170822        1       48153
20170823        0       5666
20170823        1       46834
20170824        0       4619
20170824        1       37936
20170825        0       22
20170825        1       180
20170826        0       19
20170826        1       53
20170827        0       7
20170827        1       51
20170828        0       2808
20170828        1       2904
20170829        0       46902
20170829        1       47191
20170830        0       418121
20170830        1       417253
20170831        0       889361
20170831        1       888109
20170901        0       824839
20170901        1       823351
20170902        0       615171
20170902        1       614036
20170903        0       687526
20170903        1       682189
20170904        0       916125
20170904        1       913115
20170905        0       1052800
20170905        1       1049049
20170906        0       9192
20170906        1       9145

For the record, the equivalent of T158172#3625184 for the "seen" limit we later settled on (1000ms instead of 1500ms):

Distribution of the number of previews seen >=1s per pageview (2017, stage 0).png (413×600 px, 14 KB)

Distribution of the number of previews seen >=1s per pageview (2017, stage 0), only >0 views.png (413×600 px, 15 KB)

In numbers, the percentage of pageviews with no preview card viewed (left bar in the first chart):

ruwiki	itwiki	huwiki
80.8%	84.5%	87.2%

Again, these numbers may well have been affected by T175918 too.

SELECT wiki,
cards_1000viewed_on_page,
COUNT(*) AS frequency
FROM
(SELECT wiki,
event_pageToken,
COUNT(DISTINCT event_linkInteractionToken) AS cards_1000viewed_on_page
FROM log.Popups_16364296
WHERE
(event_action = 'pageLoaded'
OR (event_totalInteractionTime >= event_perceivedWait + 1000 ))
AND wiki IN ('huwiki', 'itwiki', 'ruwiki', 'cawiki', 'elwiki', 'hewiki')
AND event_isAnon = 1
AND event_popupEnabled = 1
AND LEFT(timestamp, 8) >= '20170725'
AND LEFT(timestamp, 8) < '20170822'
GROUP BY wiki, event_popupEnabled, event_pageToken) AS pageviewlist
GROUP BY wiki, cards_1000viewed_on_page
ORDER BY wiki, cards_1000viewed_on_page

• Tbayer mentioned this in T174815: Schema:Popups suddenly stopped logging events in MariaDB, but they are still being sent according to Grafana.Oct 16 2017, 7:09 PM

ovasileva mentioned this in T182315: Write and publish report on enwiki and dewiki a/b test analysis for page previews.Dec 7 2017, 12:44 PM

resolving for now, some of the data above might be included in T182315: Write and publish report on enwiki and dewiki a/b test analysis for page previews

Liuxinyu970226 unsubscribed.Dec 13 2017, 2:47 PM

MBinder_WMF moved this task from Product Owner Backlog to 2017-18 Q2 on the Web-Team-Backlog board.Mar 6 2018, 6:44 PM

	F10259642: Distribution of the number of previews seen >=1s per pageview (2017, stage 0), only >0 views.png
	Oct 16 2017, 12:55 PM

	F10259643: Distribution of the number of previews seen >=1s per pageview (2017, stage 0).png
	Oct 16 2017, 12:55 PM

	F10025037: Distribution of the number of previews seen >=1.5s per pageview (2017, stage 0), only >0 views.png
	Oct 6 2017, 1:26 AM

	F10025039: Distribution of the number of previews seen >=1.5s per pageview (2017, stage 0).png
	Oct 6 2017, 1:26 AM

	F9704766: Distribution of the number of previews seen >=1s per pageview (2017, stage 0).png
	Sep 21 2017, 5:31 PM

	F9704768: Distribution of the number of previews seen >=1s per pageview (2017, stage 0), only >0 views.png
	Sep 21 2017, 5:31 PM

	F8833249: Previews seen (>=1s) per pageview, Firefox, 2017-07-13..23.png
	Jul 24 2017, 3:00 PM

	F8807247: Daily pageviews per session, previews ON vs. OFF, ruwiki (Firefox) 2016 instrumentation.png
	Jul 21 2017, 4:01 PM

Analyze results of stage 0 page previews testClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Analyze results of stage 0 page previews test
Closed, ResolvedPublic
Actions

Related Objects
Search...