Page MenuHomePhabricator
Create Task
Maniphest T146620

Schema:Popups losing pageLoaded events since September 8
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To
ovasileva
Authored By
• Tbayer
Sep 26 2016, 8:42 AM
Tags
Referenced Files
F4531896: pageLoaded events in the Popups schema (Chrome only, by version).png
Sep 28 2016, 5:30 AM
F4531091: MobileWebSearch events 2016-07-29..2016-09-27 from Grafana.png
Sep 27 2016, 11:08 PM
F4526933: Schema:Popups sessions per day (huwiki, anons, Firefox).png
Sep 26 2016, 6:06 PM
F4526930: Schema:Popups pageLoaded events per day (huwiki, anons, Firefox).png
Sep 26 2016, 6:06 PM
F4526376: Screen Shot 2016-09-26 at 10.48.23.png
Sep 26 2016, 2:49 PM
F4525281: Schema:Popups pageLoaded events per day (huwiki, anons).png
Sep 26 2016, 8:42 AM
F4525284: Schema:Popups sessions per day (huwiki, anons).png
Sep 26 2016, 8:42 AM
Subscribers
Aklapper
• bmansurov
Jdlrobson
• Jhernandez
MBinder_WMF
ovasileva
phuedx

Description

On September 8/9, the daily rate of pageLoaded events registered by the Popups schema on the Hungarian Wikipedia fell by more than a half, while the number of daily sessions dropped by about a third. This indicates that something is seriously wrong with the instrumentation.

Schema:Popups pageLoaded events per day (huwiki, anons).png (481×736 px, 46 KB)

Schema:Popups sessions per day (huwiki, anons).png (508×758 px, 47 KB)

This also impacts the number of pageviews per session (session depth), one of the key metrics we are using for T139319: Analyze results of Hovercards A/B test - Hungarian.

The timing matches the deployment of the fix for the "multiple hovers" issue discovered here (rolled out on September 8 per T141922#2608417, I understand it's this code change). However, that fix was only meant to affect the control group (Hovercards OFF) and only link interaction events, not pageLoaded events.

For what it's worth (perhaps not much), https://gerrit.wikimedia.org/r/309469 was also deployed on September 8, per the Server Admin Log.

Data source:

SELECT LEFT(timestamp, 8) AS date, event_popupEnabled, 
COUNT(*) AS pageloaded_events,
COUNT(DISTINCT event_pageToken) AS pageTokens,
COUNT(DISTINCT event_sessionToken) AS sessions,
COUNT(DISTINCT event_pageToken)/COUNT(DISTINCT event_sessionToken) AS page_tokens_per_session
FROM log.Popups_15777589
WHERE wiki ='huwiki'
AND event_isAnon = 1
AND event_action = 'pageLoaded'
GROUP BY date, event_popupEnabled
ORDER BY date, event_popupEnabled;

+----------+--------------------+-------------------+------------+----------+-------------------------+
| date     | event_popupEnabled | pageloaded_events | pageTokens | sessions | page_tokens_per_session |
+----------+--------------------+-------------------+------------+----------+-------------------------+
| 20160728 |                  0 |              4856 |       4849 |     1908 |                  2.5414 |
| 20160728 |                  1 |              4872 |       4867 |     1924 |                  2.5296 |
| 20160729 |                  0 |             23179 |      23144 |     9419 |                  2.4572 |
| 20160729 |                  1 |             22031 |      22016 |     9549 |                  2.3056 |
| 20160730 |                  0 |             22471 |      22453 |     7885 |                  2.8476 |
| 20160730 |                  1 |             19140 |      19107 |     7752 |                  2.4648 |
| 20160731 |                  0 |             24962 |      24947 |     9298 |                  2.6831 |
| 20160731 |                  1 |             22800 |      22783 |     9173 |                  2.4837 |
| 20160801 |                  0 |             28484 |      28469 |    10571 |                  2.6931 |
| 20160801 |                  1 |             26546 |      26521 |    10822 |                  2.4507 |
| 20160802 |                  0 |             27020 |      26998 |    10768 |                  2.5072 |
| 20160802 |                  1 |             24803 |      24786 |    10553 |                  2.3487 |
| 20160803 |                  0 |             26116 |      26102 |    10059 |                  2.5949 |
| 20160803 |                  1 |             23768 |      23749 |    10256 |                  2.3156 |
| 20160804 |                  0 |             25470 |      25456 |     9914 |                  2.5677 |
| 20160804 |                  1 |             23275 |      23246 |    10282 |                  2.2608 |
| 20160805 |                  0 |             25097 |      25082 |     9456 |                  2.6525 |
| 20160805 |                  1 |             22889 |      22864 |     9693 |                  2.3588 |
| 20160806 |                  0 |             25771 |      25743 |     9129 |                  2.8199 |
| 20160806 |                  1 |             22314 |      22283 |     8947 |                  2.4906 |
| 20160807 |                  0 |             26062 |      26028 |     9753 |                  2.6687 |
| 20160807 |                  1 |             23380 |      23362 |     9515 |                  2.4553 |
| 20160808 |                  0 |             28008 |      27986 |    11184 |                  2.5023 |
| 20160808 |                  1 |             25173 |      25147 |    10768 |                  2.3353 |
| 20160809 |                  0 |             27877 |      27860 |    11252 |                  2.4760 |
| 20160809 |                  1 |             26651 |      26634 |    11181 |                  2.3821 |
| 20160810 |                  0 |             29146 |      29116 |    11865 |                  2.4539 |
| 20160810 |                  1 |             26395 |      26361 |    11662 |                  2.2604 |
| 20160811 |                  0 |             30941 |      30914 |    11542 |                  2.6784 |
| 20160811 |                  1 |             27123 |      27098 |    11416 |                  2.3737 |
| 20160812 |                  0 |             24377 |      24351 |     9783 |                  2.4891 |
| 20160812 |                  1 |             22254 |      22239 |     9668 |                  2.3003 |
| 20160813 |                  0 |             23373 |      23345 |     8278 |                  2.8201 |
| 20160813 |                  1 |             19675 |      19657 |     8367 |                  2.3493 |
| 20160814 |                  0 |             30283 |      30271 |     9296 |                  3.2563 |
| 20160814 |                  1 |             23231 |      23196 |     9212 |                  2.5180 |
| 20160815 |                  0 |             28601 |      28590 |    11201 |                  2.5525 |
| 20160815 |                  1 |             25648 |      25608 |    11203 |                  2.2858 |
| 20160816 |                  0 |             28462 |      28434 |    11077 |                  2.5669 |
| 20160816 |                  1 |             25256 |      25239 |    10861 |                  2.3238 |
| 20160817 |                  0 |             28334 |      28311 |    11125 |                  2.5448 |
| 20160817 |                  1 |             27150 |      27116 |    11052 |                  2.4535 |
| 20160818 |                  0 |             28235 |      28196 |    11091 |                  2.5422 |
| 20160818 |                  1 |             26178 |      26151 |    11006 |                  2.3761 |
| 20160819 |                  0 |             24760 |      24740 |     9556 |                  2.5889 |
| 20160819 |                  1 |             21827 |      21811 |     9598 |                  2.2725 |
| 20160820 |                  0 |             25262 |      25250 |     9084 |                  2.7796 |
| 20160820 |                  1 |             23946 |      23922 |     9043 |                  2.6454 |
| 20160821 |                  0 |             30939 |      30914 |    10598 |                  2.9170 |
| 20160821 |                  1 |             26655 |      26637 |    10269 |                  2.5939 |
| 20160822 |                  0 |             34995 |      34967 |    13422 |                  2.6052 |
| 20160822 |                  1 |             31905 |      31881 |    13119 |                  2.4301 |
| 20160823 |                  0 |             27896 |      27871 |    11008 |                  2.5319 |
| 20160823 |                  1 |             25769 |      25743 |    11198 |                  2.2989 |
| 20160824 |                  0 |             27544 |      27531 |    10529 |                  2.6148 |
| 20160824 |                  1 |             25387 |      25336 |    10702 |                  2.3674 |
| 20160825 |                  0 |             26607 |      26586 |    10440 |                  2.5466 |
| 20160825 |                  1 |             24138 |      24121 |    10348 |                  2.3310 |
| 20160826 |                  0 |             22099 |      22081 |     9257 |                  2.3853 |
| 20160826 |                  1 |             21492 |      21467 |     9270 |                  2.3157 |
| 20160827 |                  0 |             21308 |      21300 |     7693 |                  2.7688 |
| 20160827 |                  1 |             17986 |      17967 |     7730 |                  2.3243 |
| 20160828 |                  0 |             24897 |      24884 |     9163 |                  2.7157 |
| 20160828 |                  1 |             21786 |      21775 |     8959 |                  2.4305 |
| 20160829 |                  0 |             26666 |      26633 |    11038 |                  2.4128 |
| 20160829 |                  1 |             25690 |      25673 |    11052 |                  2.3229 |
| 20160830 |                  0 |             28716 |      28701 |    11258 |                  2.5494 |
| 20160830 |                  1 |             24306 |      24282 |    11042 |                  2.1991 |
| 20160831 |                  0 |             25817 |      25805 |    10603 |                  2.4337 |
| 20160831 |                  1 |             23104 |      23086 |    10351 |                  2.2303 |
| 20160901 |                  0 |             23926 |      23904 |    10072 |                  2.3733 |
| 20160901 |                  1 |             21914 |      21892 |     9751 |                  2.2451 |
| 20160902 |                  0 |             23572 |      23554 |     9666 |                  2.4368 |
| 20160902 |                  1 |             21736 |      21704 |     9725 |                  2.2318 |
| 20160903 |                  0 |             21651 |      21627 |     8375 |                  2.5823 |
| 20160903 |                  1 |             19780 |      19761 |     8308 |                  2.3786 |
| 20160904 |                  0 |             26163 |      26132 |    10192 |                  2.5640 |
| 20160904 |                  1 |             24289 |      24258 |    10187 |                  2.3813 |
| 20160905 |                  0 |             31511 |      31482 |    12678 |                  2.4832 |
| 20160905 |                  1 |             29749 |      29705 |    12726 |                  2.3342 |
| 20160906 |                  0 |             30656 |      30634 |    12467 |                  2.4572 |
| 20160906 |                  1 |             30067 |      30046 |    12359 |                  2.4311 |
| 20160907 |                  0 |             28797 |      28772 |    11716 |                  2.4558 |
| 20160907 |                  1 |             25739 |      25726 |    11494 |                  2.2382 |
| 20160908 |                  0 |             24472 |      24449 |    10646 |                  2.2965 |
| 20160908 |                  1 |             23236 |      23217 |    10538 |                  2.2032 |
| 20160909 |                  0 |              9647 |       9630 |     5908 |                  1.6300 |
| 20160909 |                  1 |              9955 |       9945 |     6113 |                  1.6269 |
| 20160910 |                  0 |              8811 |       8791 |     4934 |                  1.7817 |
| 20160910 |                  1 |              8067 |       8050 |     4920 |                  1.6362 |
| 20160911 |                  0 |             11125 |      11105 |     6216 |                  1.7865 |
| 20160911 |                  1 |             11044 |      11028 |     6256 |                  1.7628 |
| 20160912 |                  0 |             12167 |      12146 |     7417 |                  1.6376 |
| 20160912 |                  1 |             11995 |      11982 |     7498 |                  1.5980 |
| 20160913 |                  0 |             12277 |      12238 |     7274 |                  1.6824 |
| 20160913 |                  1 |             12021 |      11989 |     7293 |                  1.6439 |
| 20160914 |                  0 |             11846 |      11835 |     7385 |                  1.6026 |
| 20160914 |                  1 |             11839 |      11821 |     7378 |                  1.6022 |
| 20160915 |                  0 |             12181 |      12153 |     7460 |                  1.6291 |
| 20160915 |                  1 |             11618 |      11602 |     7507 |                  1.5455 |
| 20160916 |                  0 |             10394 |      10363 |     6323 |                  1.6389 |
| 20160916 |                  1 |              9748 |       9733 |     6258 |                  1.5553 |
| 20160917 |                  0 |             10140 |      10110 |     5620 |                  1.7989 |
| 20160917 |                  1 |              9386 |       9362 |     5486 |                  1.7065 |
| 20160918 |                  0 |             11769 |      11728 |     6577 |                  1.7832 |
| 20160918 |                  1 |             11665 |      11640 |     6803 |                  1.7110 |
| 20160919 |                  0 |             12919 |      12894 |     7828 |                  1.6472 |
| 20160919 |                  1 |             12602 |      12572 |     7783 |                  1.6153 |
| 20160920 |                  0 |             12026 |      11991 |     7372 |                  1.6266 |
| 20160920 |                  1 |             12098 |      12075 |     7435 |                  1.6241 |
| 20160921 |                  0 |             17741 |      17699 |    10009 |                  1.7683 |
| 20160921 |                  1 |             16798 |      16788 |     9994 |                  1.6798 |
| 20160922 |                  0 |             24790 |      24776 |    11221 |                  2.2080 |
| 20160922 |                  1 |             24839 |      24820 |    11016 |                  2.2531 |
| 20160923 |                  0 |                11 |         11 |        9 |                  1.2222 |
| 20160923 |                  1 |                17 |         17 |       16 |                  1.0625 |
| 20160924 |                  0 |                 5 |          5 |        5 |                  1.0000 |
| 20160924 |                  1 |                 3 |          3 |        3 |                  1.0000 |
+----------+--------------------+-------------------+------------+----------+-------------------------+
118 rows in set (59.81 sec)

double-check (querying a day later) with all event types taken into account:


SELECT LEFT(timestamp, 8) AS date, event_popupEnabled, 
COUNT(*) AS all_events,
COUNT(DISTINCT event_pageToken) AS pageTokens,
COUNT(DISTINCT event_sessionToken) AS sessions,
COUNT(DISTINCT event_pageToken)/COUNT(DISTINCT event_sessionToken) AS page_tokens_per_session
FROM log.Popups_15777589
WHERE wiki ='huwiki'
AND event_isAnon = 1
GROUP BY date, event_popupEnabled
ORDER BY date, event_popupEnabled;

+----------+--------------------+------------+------------+----------+-------------------------+
| date     | event_popupEnabled | all_events | pageTokens | sessions | page_tokens_per_session |
+----------+--------------------+------------+------------+----------+-------------------------+
| 20160728 |                  0 |      13149 |       4849 |     1908 |                  2.5414 |
| 20160728 |                  1 |       9451 |       4867 |     1924 |                  2.5296 |
| 20160729 |                  0 |      56754 |      23153 |     9420 |                  2.4579 |
| 20160729 |                  1 |      44393 |      22024 |     9552 |                  2.3057 |
| 20160730 |                  0 |      56776 |      22471 |     7893 |                  2.8470 |
| 20160730 |                  1 |      37816 |      19123 |     7755 |                  2.4659 |
| 20160731 |                  0 |      61497 |      24967 |     9304 |                  2.6835 |
| 20160731 |                  1 |      48833 |      22801 |     9179 |                  2.4840 |
| 20160801 |                  0 |      71742 |      28486 |    10578 |                  2.6929 |
| 20160801 |                  1 |      52469 |      26542 |    10827 |                  2.4515 |
| 20160802 |                  0 |      67698 |      27017 |    10774 |                  2.5076 |
| 20160802 |                  1 |      48948 |      24800 |    10559 |                  2.3487 |
| 20160803 |                  0 |      66766 |      26126 |    10070 |                  2.5944 |
| 20160803 |                  1 |      48588 |      23767 |    10261 |                  2.3162 |
| 20160804 |                  0 |      63990 |      25475 |     9918 |                  2.5686 |
| 20160804 |                  1 |      46822 |      23260 |    10285 |                  2.2615 |
| 20160805 |                  0 |      64087 |      25097 |     9462 |                  2.6524 |
| 20160805 |                  1 |      45078 |      22883 |     9707 |                  2.3574 |
| 20160806 |                  0 |      68490 |      25766 |     9135 |                  2.8206 |
| 20160806 |                  1 |      45851 |      22293 |     8949 |                  2.4911 |
| 20160807 |                  0 |      67524 |      26050 |     9761 |                  2.6688 |
| 20160807 |                  1 |      48669 |      23376 |     9522 |                  2.4549 |
| 20160808 |                  0 |      70690 |      28034 |    11195 |                  2.5042 |
| 20160808 |                  1 |      50913 |      25178 |    10782 |                  2.3352 |
| 20160809 |                  0 |      70704 |      27879 |    11255 |                  2.4770 |
| 20160809 |                  1 |      53773 |      26652 |    11188 |                  2.3822 |
| 20160810 |                  0 |      75152 |      29133 |    11869 |                  2.4545 |
| 20160810 |                  1 |      53483 |      26385 |    11669 |                  2.2611 |
| 20160811 |                  0 |      80530 |      30943 |    11549 |                  2.6793 |
| 20160811 |                  1 |      54721 |      27114 |    11425 |                  2.3732 |
| 20160812 |                  0 |      61157 |      24373 |     9788 |                  2.4901 |
| 20160812 |                  1 |      44658 |      22258 |     9674 |                  2.3008 |
| 20160813 |                  0 |      61409 |      23367 |     8281 |                  2.8218 |
| 20160813 |                  1 |      40373 |      19675 |     8374 |                  2.3495 |
| 20160814 |                  0 |      76976 |      30293 |     9302 |                  3.2566 |
| 20160814 |                  1 |      45622 |      23218 |     9219 |                  2.5185 |
| 20160815 |                  0 |      71437 |      28613 |    11205 |                  2.5536 |
| 20160815 |                  1 |      49893 |      25630 |    11213 |                  2.2857 |
| 20160816 |                  0 |      71702 |      28448 |    11083 |                  2.5668 |
| 20160816 |                  1 |      50837 |      25264 |    10874 |                  2.3233 |
| 20160817 |                  0 |      71217 |      28331 |    11129 |                  2.5457 |
| 20160817 |                  1 |      53750 |      27139 |    11059 |                  2.4540 |
| 20160818 |                  0 |      73430 |      28492 |    11139 |                  2.5579 |
| 20160818 |                  1 |      52465 |      26425 |    11053 |                  2.3908 |
| 20160819 |                  0 |      63224 |      25195 |     9643 |                  2.6128 |
| 20160819 |                  1 |      43680 |      22131 |     9663 |                  2.2903 |
| 20160820 |                  0 |      68239 |      25702 |     9162 |                  2.8053 |
| 20160820 |                  1 |      51643 |      24324 |     9105 |                  2.6715 |
| 20160821 |                  0 |      82325 |      31477 |    10681 |                  2.9470 |
| 20160821 |                  1 |      56893 |      27059 |    10352 |                  2.6139 |
| 20160822 |                  0 |      90560 |      35598 |    13531 |                  2.6308 |
| 20160822 |                  1 |      64889 |      32387 |    13196 |                  2.4543 |
| 20160823 |                  0 |      69363 |      28345 |    11084 |                  2.5573 |
| 20160823 |                  1 |      50641 |      26131 |    11270 |                  2.3186 |
| 20160824 |                  0 |      68587 |      28004 |    10618 |                  2.6374 |
| 20160824 |                  1 |      50653 |      25713 |    10767 |                  2.3881 |
| 20160825 |                  0 |      68770 |      26998 |    10500 |                  2.5712 |
| 20160825 |                  1 |      48547 |      24463 |    10394 |                  2.3536 |
| 20160826 |                  0 |      54946 |      22465 |     9324 |                  2.4094 |
| 20160826 |                  1 |      43727 |      21806 |     9333 |                  2.3364 |
| 20160827 |                  0 |      54863 |      21709 |     7751 |                  2.8008 |
| 20160827 |                  1 |      36975 |      18223 |     7761 |                  2.3480 |
| 20160828 |                  0 |      64123 |      25319 |     9224 |                  2.7449 |
| 20160828 |                  1 |      44441 |      22116 |     9002 |                  2.4568 |
| 20160829 |                  0 |      65194 |      27118 |    11119 |                  2.4389 |
| 20160829 |                  1 |      50440 |      26078 |    11116 |                  2.3460 |
| 20160830 |                  0 |      73032 |      29237 |    11355 |                  2.5748 |
| 20160830 |                  1 |      49508 |      24648 |    11116 |                  2.2173 |
| 20160831 |                  0 |      62878 |      26233 |    10687 |                  2.4547 |
| 20160831 |                  1 |      44946 |      23407 |    10408 |                  2.2489 |
| 20160901 |                  0 |      57052 |      24290 |    10148 |                  2.3936 |
| 20160901 |                  1 |      44738 |      22203 |     9808 |                  2.2638 |
| 20160902 |                  0 |      59160 |      23935 |     9747 |                  2.4556 |
| 20160902 |                  1 |      43073 |      21999 |     9786 |                  2.2480 |
| 20160903 |                  0 |      54247 |      21977 |     8459 |                  2.5981 |
| 20160903 |                  1 |      40045 |      20099 |     8376 |                  2.3996 |
| 20160904 |                  0 |      65829 |      26587 |    10271 |                  2.5886 |
| 20160904 |                  1 |      49016 |      24582 |    10239 |                  2.4008 |
| 20160905 |                  0 |      77852 |      31966 |    12771 |                  2.5030 |
| 20160905 |                  1 |      59508 |      30133 |    12813 |                  2.3518 |
| 20160906 |                  0 |      75816 |      31134 |    12559 |                  2.4790 |
| 20160906 |                  1 |      60656 |      30474 |    12441 |                  2.4495 |
| 20160907 |                  0 |      71017 |      29258 |    11807 |                  2.4780 |
| 20160907 |                  1 |      51478 |      26120 |    11579 |                  2.2558 |
| 20160908 |                  0 |      59302 |      24871 |    10744 |                  2.3149 |
| 20160908 |                  1 |      46967 |      23546 |    10613 |                  2.2186 |
| 20160909 |                  0 |      19638 |       9781 |     5969 |                  1.6386 |
| 20160909 |                  1 |      19817 |      10105 |     6186 |                  1.6335 |
| 20160910 |                  0 |      20305 |       8929 |     4974 |                  1.7951 |
| 20160910 |                  1 |      16205 |       8162 |     4965 |                  1.6439 |
| 20160911 |                  0 |      25265 |      11310 |     6292 |                  1.7975 |
| 20160911 |                  1 |      23466 |      11201 |     6310 |                  1.7751 |
| 20160912 |                  0 |      25174 |      12324 |     7479 |                  1.6478 |
| 20160912 |                  1 |      24119 |      12146 |     7567 |                  1.6051 |
| 20160913 |                  0 |      27711 |      12438 |     7360 |                  1.6899 |
| 20160913 |                  1 |      24629 |      12148 |     7355 |                  1.6517 |
| 20160914 |                  0 |      26730 |      12046 |     7481 |                  1.6102 |
| 20160914 |                  1 |      23953 |      11993 |     7445 |                  1.6109 |
| 20160915 |                  0 |      26474 |      12340 |     7546 |                  1.6353 |
| 20160915 |                  1 |      22510 |      11760 |     7576 |                  1.5523 |
| 20160916 |                  0 |      22013 |      10528 |     6387 |                  1.6483 |
| 20160916 |                  1 |      18700 |       9843 |     6303 |                  1.5616 |
| 20160917 |                  0 |      22535 |      10264 |     5685 |                  1.8055 |
| 20160917 |                  1 |      20029 |       9516 |     5538 |                  1.7183 |
| 20160918 |                  0 |      27397 |      11910 |     6648 |                  1.7915 |
| 20160918 |                  1 |      23937 |      11812 |     6863 |                  1.7211 |
| 20160919 |                  0 |      28089 |      13092 |     7916 |                  1.6539 |
| 20160919 |                  1 |      25976 |      12757 |     7835 |                  1.6282 |
| 20160920 |                  0 |      26488 |      12197 |     7449 |                  1.6374 |
| 20160920 |                  1 |      24725 |      12269 |     7508 |                  1.6341 |
| 20160921 |                  0 |      38254 |      17969 |    10101 |                  1.7789 |
| 20160921 |                  1 |      34070 |      17024 |    10076 |                  1.6896 |
| 20160922 |                  0 |      51892 |      25163 |    11312 |                  2.2245 |
| 20160922 |                  1 |      51215 |      25186 |    11094 |                  2.2702 |
| 20160923 |                  0 |         28 |         21 |       18 |                  1.1667 |
| 20160923 |                  1 |         51 |         28 |       27 |                  1.0370 |
| 20160924 |                  0 |         16 |          6 |        6 |                  1.0000 |
| 20160924 |                  1 |          3 |          3 |        3 |                  1.0000 |
| 20160925 |                  0 |          8 |          2 |        2 |                  1.0000 |
| 20160925 |                  1 |          2 |          2 |        2 |                  1.0000 |
| 20160926 |                  0 |          9 |          9 |        6 |                  1.5000 |
| 20160926 |                  1 |          1 |          1 |        1 |                  1.0000 |
+----------+--------------------+------------+------------+----------+-------------------------+
122 rows in set (1 min 25.32 sec)
(edit: fix column name)

(I've left out the data after Sept 21 in the above charts, as it is likely affected by separate changes, cf. https://phabricator.wikimedia.org/T136746#2663210 .)

Related Objects

Event Timeline

• Tbayer created this task.Sep 26 2016, 8:42 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2016, 8:42 AM
ovasileva triaged this task as High priority.Sep 26 2016, 11:40 AM
ovasileva edited projects, added Page-Previews (2016-17-Q2-Goal), Reading-Web-Sprint-82-Xpect-Rspec; removed Page-Previews.
• bmansurov subscribed.Sep 26 2016, 2:49 PM

Currently, the pageLoaded event is being sent, but the request is being cancelled for some reason:

Screen Shot 2016-09-26 at 10.48.23.png (1×2 px, 777 KB)

MBinder_WMF set the point value for this task to 8.Sep 26 2016, 4:19 PM
MBinder_WMF moved this task from Needs Analysis to To Do on the Reading-Web-Sprint-82-Xpect-Rspec board.
• bmansurov claimed this task.Sep 26 2016, 5:35 PM
• bmansurov moved this task from To Do to Doing on the Reading-Web-Sprint-82-Xpect-Rspec board.
• Tbayer added a comment.Sep 26 2016, 6:06 PM

Applying the crude browser family detection we discussed recently, it looks like Firefox is not affected (and I guess @bmansurov's screenshot is from Chrome):

Schema:Popups pageLoaded events per day (huwiki, anons, Firefox).png (549×871 px, 51 KB)

Schema:Popups sessions per day (huwiki, anons, Firefox).png (518×791 px, 53 KB)

Interesting though that here too the gap in pageviews between Hovercards ON and OFF appears to have narrowed around Sept 8.

Data source:

SELECT LEFT(timestamp, 8) AS date, event_popupEnabled, 
COUNT(*) AS pageloaded_events,
COUNT(DISTINCT event_pageToken) AS pageTokens,
COUNT(DISTINCT event_sessionToken) AS sessions,
COUNT(DISTINCT event_pageToken)/COUNT(DISTINCT event_sessionToken) AS page_tokens_per_session
FROM log.Popups_15777589
WHERE wiki ='huwiki'
AND event_isAnon = 1
AND event_action = 'pageLoaded'
AND INSTR(userAgent,'Firefox') AND NOT INSTR(userAgent,'Seamonkey')
GROUP BY date, event_popupEnabled
ORDER BY date, event_popupEnabled;

+----------+--------------------+-------------------+------------+----------+-------------------------+
| date     | event_popupEnabled | pageloaded_events | pageTokens | sessions | page_tokens_per_session |
+----------+--------------------+-------------------+------------+----------+-------------------------+
| 20160728 |                  0 |              1320 |       1313 |      539 |                  2.4360 |
| 20160728 |                  1 |              1103 |       1098 |      537 |                  2.0447 |
| 20160729 |                  0 |              7296 |       7263 |     2923 |                  2.4848 |
| 20160729 |                  1 |              6436 |       6422 |     3080 |                  2.0851 |
| 20160730 |                  0 |              6451 |       6433 |     2341 |                  2.7480 |
| 20160730 |                  1 |              5497 |       5466 |     2315 |                  2.3611 |
| 20160731 |                  0 |              7424 |       7409 |     2742 |                  2.7020 |
| 20160731 |                  1 |              6286 |       6269 |     2757 |                  2.2738 |
| 20160801 |                  0 |              9046 |       9032 |     3253 |                  2.7765 |
| 20160801 |                  1 |              7246 |       7225 |     3372 |                  2.1426 |
| 20160802 |                  0 |              8583 |       8561 |     3369 |                  2.5411 |
| 20160802 |                  1 |              6769 |       6753 |     3353 |                  2.0140 |
| 20160803 |                  0 |              8099 |       8087 |     3155 |                  2.5632 |
| 20160803 |                  1 |              7079 |       7060 |     3221 |                  2.1919 |
| 20160804 |                  0 |              7398 |       7385 |     3090 |                  2.3900 |
| 20160804 |                  1 |              7311 |       7284 |     3238 |                  2.2495 |
| 20160805 |                  0 |              7921 |       7908 |     2912 |                  2.7157 |
| 20160805 |                  1 |              6360 |       6337 |     3026 |                  2.0942 |
| 20160806 |                  0 |              7672 |       7645 |     2727 |                  2.8034 |
| 20160806 |                  1 |              6283 |       6253 |     2740 |                  2.2821 |
| 20160807 |                  0 |              7398 |       7366 |     2911 |                  2.5304 |
| 20160807 |                  1 |              6556 |       6539 |     2798 |                  2.3370 |
| 20160808 |                  0 |              8442 |       8421 |     3454 |                  2.4380 |
| 20160808 |                  1 |              7116 |       7093 |     3302 |                  2.1481 |
| 20160809 |                  0 |              8556 |       8542 |     3514 |                  2.4308 |
| 20160809 |                  1 |              7515 |       7500 |     3498 |                  2.1441 |
| 20160810 |                  0 |              8763 |       8734 |     3692 |                  2.3657 |
| 20160810 |                  1 |              7567 |       7533 |     3567 |                  2.1119 |
| 20160811 |                  0 |              8631 |       8606 |     3570 |                  2.4106 |
| 20160811 |                  1 |              7717 |       7693 |     3511 |                  2.1911 |
| 20160812 |                  0 |              7172 |       7146 |     3033 |                  2.3561 |
| 20160812 |                  1 |              6741 |       6727 |     3049 |                  2.2063 |
| 20160813 |                  0 |              7460 |       7433 |     2543 |                  2.9229 |
| 20160813 |                  1 |              5701 |       5684 |     2528 |                  2.2484 |
| 20160814 |                  0 |              7365 |       7353 |     2690 |                  2.7335 |
| 20160814 |                  1 |              6326 |       6292 |     2723 |                  2.3107 |
| 20160815 |                  0 |              8145 |       8135 |     3383 |                  2.4047 |
| 20160815 |                  1 |              7621 |       7582 |     3541 |                  2.1412 |
| 20160816 |                  0 |              8492 |       8465 |     3436 |                  2.4636 |
| 20160816 |                  1 |              7033 |       7017 |     3341 |                  2.1003 |
| 20160817 |                  0 |              8324 |       8301 |     3503 |                  2.3697 |
| 20160817 |                  1 |              7786 |       7752 |     3539 |                  2.1904 |
| 20160818 |                  0 |              8834 |       8798 |     3564 |                  2.4686 |
| 20160818 |                  1 |              7472 |       7446 |     3435 |                  2.1677 |
| 20160819 |                  0 |              7323 |       7303 |     2998 |                  2.4360 |
| 20160819 |                  1 |              5994 |       5978 |     2929 |                  2.0410 |
| 20160820 |                  0 |              7779 |       7768 |     2639 |                  2.9435 |
| 20160820 |                  1 |              6992 |       6968 |     2642 |                  2.6374 |
| 20160821 |                  0 |              8886 |       8862 |     3163 |                  2.8018 |
| 20160821 |                  1 |              7063 |       7046 |     3006 |                  2.3440 |
| 20160822 |                  0 |             10433 |      10406 |     4111 |                  2.5313 |
| 20160822 |                  1 |              8607 |       8583 |     4004 |                  2.1436 |
| 20160823 |                  0 |              8387 |       8362 |     3498 |                  2.3905 |
| 20160823 |                  1 |              7334 |       7309 |     3482 |                  2.0991 |
| 20160824 |                  0 |              8254 |       8243 |     3305 |                  2.4941 |
| 20160824 |                  1 |              7430 |       7399 |     3432 |                  2.1559 |
| 20160825 |                  0 |              7851 |       7832 |     3243 |                  2.4150 |
| 20160825 |                  1 |              6750 |       6733 |     3235 |                  2.0813 |
| 20160826 |                  0 |              6410 |       6394 |     2869 |                  2.2287 |
| 20160826 |                  1 |              6323 |       6298 |     2829 |                  2.2262 |
| 20160827 |                  0 |              6400 |       6392 |     2299 |                  2.7803 |
| 20160827 |                  1 |              5195 |       5176 |     2330 |                  2.2215 |
| 20160828 |                  0 |              7060 |       7048 |     2598 |                  2.7129 |
| 20160828 |                  1 |              5925 |       5915 |     2623 |                  2.2551 |
| 20160829 |                  0 |              8275 |       8243 |     3358 |                  2.4547 |
| 20160829 |                  1 |              7409 |       7397 |     3393 |                  2.1801 |
| 20160830 |                  0 |              8545 |       8530 |     3328 |                  2.5631 |
| 20160830 |                  1 |              7103 |       7079 |     3437 |                  2.0596 |
| 20160831 |                  0 |              8503 |       8491 |     3330 |                  2.5498 |
| 20160831 |                  1 |              6573 |       6555 |     3148 |                  2.0823 |
| 20160901 |                  0 |              7015 |       6993 |     3184 |                  2.1963 |
| 20160901 |                  1 |              6525 |       6504 |     3050 |                  2.1325 |
| 20160902 |                  0 |              6950 |       6933 |     3070 |                  2.2583 |
| 20160902 |                  1 |              6312 |       6281 |     3002 |                  2.0923 |
| 20160903 |                  0 |              6229 |       6205 |     2516 |                  2.4662 |
| 20160903 |                  1 |              5017 |       5001 |     2419 |                  2.0674 |
| 20160904 |                  0 |              7406 |       7377 |     2962 |                  2.4905 |
| 20160904 |                  1 |              6088 |       6061 |     2886 |                  2.1001 |
| 20160905 |                  0 |              9509 |       9481 |     3912 |                  2.4236 |
| 20160905 |                  1 |              8655 |       8613 |     4024 |                  2.1404 |
| 20160906 |                  0 |              9378 |       9359 |     3842 |                  2.4360 |
| 20160906 |                  1 |              8084 |       8066 |     3829 |                  2.1066 |
| 20160907 |                  0 |              8239 |       8215 |     3664 |                  2.2421 |
| 20160907 |                  1 |              7674 |       7661 |     3602 |                  2.1269 |
| 20160908 |                  0 |              8202 |       8180 |     3464 |                  2.3614 |
| 20160908 |                  1 |              7612 |       7594 |     3463 |                  2.1929 |
| 20160909 |                  0 |              6468 |       6451 |     2966 |                  2.1750 |
| 20160909 |                  1 |              6680 |       6670 |     3096 |                  2.1544 |
| 20160910 |                  0 |              6240 |       6220 |     2604 |                  2.3886 |
| 20160910 |                  1 |              5343 |       5326 |     2463 |                  2.1624 |
| 20160911 |                  0 |              7769 |       7749 |     3202 |                  2.4200 |
| 20160911 |                  1 |              7542 |       7526 |     3164 |                  2.3786 |
| 20160912 |                  0 |              8227 |       8207 |     3880 |                  2.1152 |
| 20160912 |                  1 |              8028 |       8017 |     3904 |                  2.0535 |
| 20160913 |                  0 |              8568 |       8530 |     3930 |                  2.1705 |
| 20160913 |                  1 |              8297 |       8265 |     3905 |                  2.1165 |
| 20160914 |                  0 |              7911 |       7902 |     3766 |                  2.0982 |
| 20160914 |                  1 |              7877 |       7860 |     3735 |                  2.1044 |
| 20160915 |                  0 |              7961 |       7933 |     3811 |                  2.0816 |
| 20160915 |                  1 |              7444 |       7429 |     3777 |                  1.9669 |
| 20160916 |                  0 |              6793 |       6763 |     3091 |                  2.1880 |
| 20160916 |                  1 |              6283 |       6269 |     3058 |                  2.0500 |
| 20160917 |                  0 |              7095 |       7065 |     2930 |                  2.4113 |
| 20160917 |                  1 |              6459 |       6435 |     2794 |                  2.3031 |
| 20160918 |                  0 |              7845 |       7804 |     3254 |                  2.3983 |
| 20160918 |                  1 |              7629 |       7607 |     3290 |                  2.3122 |
| 20160919 |                  0 |              8809 |       8787 |     4138 |                  2.1235 |
| 20160919 |                  1 |              8539 |       8509 |     4123 |                  2.0638 |
| 20160920 |                  0 |              8226 |       8191 |     3935 |                  2.0816 |
| 20160920 |                  1 |              8292 |       8272 |     3955 |                  2.0915 |
| 20160921 |                  0 |              8579 |       8537 |     3831 |                  2.2284 |
| 20160921 |                  1 |              8000 |       7990 |     3925 |                  2.0357 |
| 20160922 |                  0 |              7171 |       7159 |     3435 |                  2.0841 |
| 20160922 |                  1 |              7038 |       7020 |     3385 |                  2.0739 |
| 20160923 |                  0 |                 8 |          8 |        6 |                  1.3333 |
| 20160923 |                  1 |                 3 |          3 |        3 |                  1.0000 |
| 20160924 |                  0 |                 1 |          1 |        1 |                  1.0000 |
| 20160924 |                  1 |                 2 |          2 |        2 |                  1.0000 |
| 20160925 |                  0 |                 1 |          1 |        1 |                  1.0000 |
+----------+--------------------+-------------------+------------+----------+-------------------------+
119 rows in set (52.83 sec)
• bmansurov added a comment.Sep 26 2016, 7:05 PM

Yes, the screenshot is that of Chrome.

• bmansurov added a comment.Sep 26 2016, 7:49 PM

Looking at the data, it looks majority of events are coming from Firefox, and the rest from other browsers, which may mean something si broken in Chrome, or Chrome itself is broken.

I've tried older versions of Chrome (48, 49, 50) and in all of them I see the same thing: request is cancelled. I saw the same thing on the latest Opera. Firefox doesn't show beacon requests in the developer tools [1], but I verified that the code to log is being executed. I've sent other EventLogging events, all of them are cancelled in Chrome.

@Tbayer, do we see the same picture for other events (not pageLoaded)? Or is that the only affected event?

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1045229

• bmansurov added a comment.EditedSep 26 2016, 8:17 PM
  • Chrome is sending events correctly after I disabled the uBlock Origin extension. So no problem with the browser per se.
  • Schema sampling rate hasn't changed [1]. The default rate of 0.1 [2] is still in use:

[1] https://github.com/wikimedia/operations-mediawiki-config/search?utf8=%E2%9C%93&q=SchemaPopupsSamplingRate
[2] https://github.com/wikimedia/mediawiki-extensions-Popups/search?utf8=%E2%9C%93&q=SchemaPopupsSamplingRate&type=Code

• Tbayer updated the task description. (Show Details)Sep 26 2016, 9:18 PM
• bmansurov added a comment.Sep 26 2016, 9:28 PM

@Tbayer I see the event count has picked up since 09/22 with 51892 events which is on par with pre-dip events. So things to be normal now?

Events after 09/23 seem to indicate that we started logging into the new schema [1].

[1] https://gerrit.wikimedia.org/r/#/c/310324/

• Tbayer added a comment.Sep 26 2016, 9:28 PM
In T146620#2668510, @bmansurov wrote:

...

@Tbayer, do we see the same picture for other events (not pageLoaded)? Or is that the only affected event?

It is also affecting dwelledButAbandoned events, link open events (expected for Hovercards OFF, not expected for Hovercards ON), and dismissed events. I.e. basically all of them, but not necessarily by the same factor.

BTW interesting to see a notable amount of dismissed events in the OFF group (where they should never occur) on September 9.

SELECT LEFT(timestamp, 8) AS date, event_popupEnabled, 
SUM(IF(event_action = 'dwelledButAbandoned',1,0)) AS dwelledButAbandoneds,
SUM(IF(event_action LIKE 'open%',1,0)) AS opens,
SUM(IF(event_action = 'error',1,0)) AS errors,
SUM(IF(event_action = 'dismissed',1,0)) AS dismisseds,
COUNT(DISTINCT event_sessionToken) AS sessions
FROM log.Popups_15777589
WHERE wiki ='huwiki'
AND event_isAnon = 1
GROUP BY date, event_popupEnabled
ORDER BY date, event_popupEnabled;

+----------+--------------------+----------------------+-------+--------+------------+----------+
| date     | event_popupEnabled | dwelledButAbandoneds | opens | errors | dismisseds | sessions |
+----------+--------------------+----------------------+-------+--------+------------+----------+
| 20160728 |                  0 |                 5780 |  2513 |      0 |          0 |     1908 |
| 20160728 |                  1 |                 1204 |  1036 |      5 |       2334 |     1924 |
| 20160729 |                  0 |                23066 | 10503 |      0 |          6 |     9420 |
| 20160729 |                  1 |                 6432 |  3846 |      8 |      12069 |     9552 |
| 20160730 |                  0 |                22361 | 11944 |      0 |          0 |     7893 |
| 20160730 |                  1 |                 5121 |  3561 |      5 |       9987 |     7755 |
| 20160731 |                  0 |                24371 | 12164 |      0 |          0 |     9304 |
| 20160731 |                  1 |                 7230 |  4385 |     10 |      14401 |     9179 |
| 20160801 |                  0 |                29006 | 14246 |      0 |          6 |    10578 |
| 20160801 |                  1 |                 7269 |  4693 |     24 |      13933 |    10827 |
| 20160802 |                  0 |                28102 | 12576 |      0 |          0 |    10774 |
| 20160802 |                  1 |                 6895 |  4485 |      8 |      12755 |    10559 |
| 20160803 |                  0 |                27580 | 13055 |      0 |         15 |    10070 |
| 20160803 |                  1 |                 7108 |  4454 |     10 |      13244 |    10261 |
| 20160804 |                  0 |                25631 | 12889 |      0 |          0 |     9918 |
| 20160804 |                  1 |                 7078 |  3832 |      4 |      12627 |    10285 |
| 20160805 |                  0 |                26409 | 12581 |      0 |          0 |     9462 |
| 20160805 |                  1 |                 6340 |  4254 |      3 |      11583 |     9707 |
| 20160806 |                  0 |                28526 | 14192 |      0 |          1 |     9135 |
| 20160806 |                  1 |                 6520 |  4382 |      4 |      12629 |     8949 |
| 20160807 |                  0 |                28376 | 13082 |      0 |          4 |     9761 |
| 20160807 |                  1 |                 7655 |  4305 |      4 |      13318 |     9522 |
| 20160808 |                  0 |                29280 | 13397 |      0 |          5 |    11195 |
| 20160808 |                  1 |                 7946 |  4446 |      9 |      13333 |    10782 |
| 20160809 |                  0 |                29642 | 13185 |      0 |          0 |    11255 |
| 20160809 |                  1 |                 8028 |  5107 |      9 |      13974 |    11188 |
| 20160810 |                  0 |                31875 | 14127 |      0 |          4 |    11869 |
| 20160810 |                  1 |                 8166 |  4534 |     10 |      14373 |    11669 |
| 20160811 |                  0 |                33183 | 16406 |      0 |          0 |    11549 |
| 20160811 |                  1 |                 8098 |  4753 |     12 |      14731 |    11425 |
| 20160812 |                  0 |                24994 | 11770 |      0 |         16 |     9788 |
| 20160812 |                  1 |                 6821 |  3949 |      9 |      11624 |     9674 |
| 20160813 |                  0 |                25830 | 12206 |      0 |          0 |     8281 |
| 20160813 |                  1 |                 6228 |  3497 |      2 |      10969 |     8374 |
| 20160814 |                  0 |                28094 | 18599 |      0 |          0 |     9302 |
| 20160814 |                  1 |                 6571 |  4374 |     10 |      11434 |     9219 |
| 20160815 |                  0 |                29780 | 13041 |      0 |         15 |    11205 |
| 20160815 |                  1 |                 7146 |  4492 |     12 |      12592 |    11213 |
| 20160816 |                  0 |                29045 | 14193 |      0 |          2 |    11083 |
| 20160816 |                  1 |                 7688 |  4573 |      6 |      13311 |    10874 |
| 20160817 |                  0 |                28673 | 14208 |      0 |          2 |    11129 |
| 20160817 |                  1 |                 7771 |  4903 |      9 |      13913 |    11059 |
| 20160818 |                  0 |                31210 | 13984 |      0 |          1 |    11139 |
| 20160818 |                  1 |                 7839 |  4718 |     10 |      13715 |    11053 |
| 20160819 |                  0 |                25856 | 12601 |      0 |          7 |     9643 |
| 20160819 |                  1 |                 6651 |  3854 |      4 |      11338 |     9663 |
| 20160820 |                  0 |                29475 | 13495 |      0 |          7 |     9162 |
| 20160820 |                  1 |                 8054 |  5222 |      8 |      14407 |     9105 |
| 20160821 |                  0 |                33707 | 17674 |      0 |          5 |    10681 |
| 20160821 |                  1 |                 8597 |  5455 |      9 |      16171 |    10352 |
| 20160822 |                  0 |                38082 | 17483 |      0 |          0 |    13531 |
| 20160822 |                  1 |                 9640 |  6198 |     15 |      17121 |    13196 |
| 20160823 |                  0 |                28196 | 13271 |      0 |          0 |    11084 |
| 20160823 |                  1 |                 7380 |  4571 |     12 |      12908 |    11270 |
| 20160824 |                  0 |                27519 | 13524 |      0 |          0 |    10618 |
| 20160824 |                  1 |                 7356 |  4523 |      8 |      13377 |    10767 |
| 20160825 |                  0 |                28312 | 13851 |      0 |          0 |    10500 |
| 20160825 |                  1 |                 6902 |  4453 |     11 |      13041 |    10394 |
| 20160826 |                  0 |                22321 | 10526 |      0 |          0 |     9324 |
| 20160826 |                  1 |                 6061 |  3803 |      5 |      12362 |     9333 |
| 20160827 |                  0 |                21899 | 11655 |      0 |          1 |     7751 |
| 20160827 |                  1 |                 5340 |  3223 |      9 |      10415 |     7761 |
| 20160828 |                  0 |                25964 | 13262 |      0 |          0 |     9224 |
| 20160828 |                  1 |                 6370 |  4259 |     10 |      12012 |     9002 |
| 20160829 |                  0 |                26568 | 11960 |      0 |          0 |    11119 |
| 20160829 |                  1 |                 7104 |  4720 |      8 |      12913 |    11116 |
| 20160830 |                  0 |                29771 | 14536 |      0 |          9 |    11355 |
| 20160830 |                  1 |                 7214 |  4149 |     10 |      13828 |    11116 |
| 20160831 |                  0 |                25132 | 11928 |      0 |          1 |    10687 |
| 20160831 |                  1 |                 6153 |  3577 |      9 |      12100 |    10408 |
| 20160901 |                  0 |                22876 | 10248 |      0 |          2 |    10148 |
| 20160901 |                  1 |                 6582 |  3611 |     13 |      12612 |     9808 |
| 20160902 |                  0 |                25294 | 10294 |      0 |          0 |     9747 |
| 20160902 |                  1 |                 6320 |  3726 |     11 |      11273 |     9786 |
| 20160903 |                  0 |                22429 | 10166 |      0 |          1 |     8459 |
| 20160903 |                  1 |                 5715 |  3576 |      7 |      10965 |     8376 |
| 20160904 |                  0 |                27754 | 11912 |      0 |          0 |    10271 |
| 20160904 |                  1 |                 7343 |  4112 |     11 |      13258 |    10239 |
| 20160905 |                  0 |                32891 | 13450 |      0 |          0 |    12771 |
| 20160905 |                  1 |                 9149 |  4903 |     13 |      15685 |    12813 |
| 20160906 |                  0 |                32449 | 12711 |      0 |          0 |    12559 |
| 20160906 |                  1 |                 9234 |  4927 |     14 |      16409 |    12441 |
| 20160907 |                  0 |                29750 | 12470 |      0 |          0 |    11807 |
| 20160907 |                  1 |                 8113 |  3829 |      4 |      13786 |    11579 |
| 20160908 |                  0 |                25253 |  9577 |      0 |          0 |    10744 |
| 20160908 |                  1 |                 7170 |  3500 |      9 |      13047 |    10613 |
| 20160909 |                  0 |                 8320 |  1438 |      0 |        233 |     5969 |
| 20160909 |                  1 |                 2986 |  1445 |      1 |       5428 |     6186 |
| 20160910 |                  0 |                 9772 |  1722 |      0 |          0 |     4974 |
| 20160910 |                  1 |                 2524 |  1166 |      2 |       4444 |     4965 |
| 20160911 |                  0 |                12110 |  2028 |      0 |          2 |     6292 |
| 20160911 |                  1 |                 3835 |  1712 |      8 |       6864 |     6310 |
| 20160912 |                  0 |                11209 |  1798 |      0 |          0 |     7479 |
| 20160912 |                  1 |                 3801 |  1501 |      5 |       6812 |     7567 |
| 20160913 |                  0 |                13350 |  2084 |      0 |          0 |     7360 |
| 20160913 |                  1 |                 3804 |  1683 |      2 |       7116 |     7355 |
| 20160914 |                  0 |                13001 |  1882 |      0 |          1 |     7481 |
| 20160914 |                  1 |                 3775 |  1726 |      5 |       6606 |     7445 |
| 20160915 |                  0 |                12375 |  1917 |      0 |          1 |     7546 |
| 20160915 |                  1 |                 3573 |  1329 |      4 |       5986 |     7576 |
| 20160916 |                  0 |                 9923 |  1696 |      0 |          0 |     6387 |
| 20160916 |                  1 |                 2880 |  1186 |      2 |       4884 |     6303 |
| 20160917 |                  0 |                10469 |  1926 |      0 |          0 |     5685 |
| 20160917 |                  1 |                 3190 |  1459 |      3 |       5988 |     5538 |
| 20160918 |                  0 |                12921 |  2707 |      0 |          0 |     6648 |
| 20160918 |                  1 |                 3818 |  1679 |      8 |       6761 |     6863 |
| 20160919 |                  0 |                13183 |  1985 |      0 |          2 |     7916 |
| 20160919 |                  1 |                 4369 |  1694 |      4 |       7307 |     7835 |
| 20160920 |                  0 |                12709 |  1751 |      0 |          2 |     7449 |
| 20160920 |                  1 |                 3932 |  1701 |      7 |       6986 |     7508 |
| 20160921 |                  0 |                17746 |  2764 |      0 |          3 |    10101 |
| 20160921 |                  1 |                 5500 |  2345 |      5 |       9419 |    10076 |
| 20160922 |                  0 |                23166 |  3929 |      0 |          7 |    11312 |
| 20160922 |                  1 |                 8084 |  4033 |     16 |      14238 |    11094 |
| 20160923 |                  0 |                   14 |     3 |      0 |          0 |       18 |
| 20160923 |                  1 |                   16 |     6 |      0 |         12 |       27 |
| 20160924 |                  0 |                   10 |     1 |      0 |          0 |        6 |
| 20160924 |                  1 |                    0 |     0 |      0 |          0 |        3 |
| 20160925 |                  0 |                    6 |     0 |      0 |          0 |        2 |
| 20160925 |                  1 |                    0 |     1 |      0 |          0 |        2 |
| 20160926 |                  0 |                    4 |     0 |      0 |          0 |        6 |
| 20160926 |                  1 |                    0 |     0 |      0 |          0 |        1 |
+----------+--------------------+----------------------+-------+--------+------------+----------+
122 rows in set (1 min 5.17 sec)
• Tbayer added a comment.Sep 26 2016, 9:33 PM
In T146620#2668937, @bmansurov wrote:

@Tbayer I see the event count has picked up since 09/22 with 51892 events which is on par with pre-dip events. So things to be normal now?

As I wrote in the initial task description: This is likely affected by separate changes, cf. https://phabricator.wikimedia.org/T136746#2663210 (specifically, T146035, according to JonR's remark there). Or did these code changes revert something from September 8/9?

• bmansurov added a comment.Sep 26 2016, 9:41 PM

Patchsets in that task don't seem to revert the changes made prior to September 8/9.

phuedx added a comment.Sep 27 2016, 5:28 PM

Questions from post-standup discussion:

  • are other events affected, e.g. is MobileFrontend event logging affected?
  • would (longer) edge cache lifetimes affect the rollout of our fixes across huwiki?
  • can we narrow down the version of Chrome that's affected?
  • is this a side-effect of poor quality connections?
• bmansurov added a comment.Sep 27 2016, 6:19 PM

@Tbayer can you help answering the questions above?

Here are my thoughts around the points above:

  • We could use Schema:MobileWebSearch.
  • I think time will tell. We could wait another week and compare the numbers again.
  • Yes, we would like to know what versions of chrome are effected.
  • I don't think so because the question assumes quality of connection of our users has changed. Am I wrong?
• Tbayer added a comment.EditedSep 27 2016, 11:08 PM
In T146620#2671235, @phuedx wrote:

Questions from post-standup discussion:

  • are other events affected, e.g. is MobileFrontend event logging affected?

Indeed! A screenshot from Grafana showing the total event rate for the example suggested by Baha (MobileWebSearch):

MobileWebSearch events 2016-07-29..2016-09-27 from Grafana.png (576×847 px, 113 KB)

Likewise:

  • RelatedArticles shows a similar development as well (sharp drop around Sept 8/9, and a more gradual rise back to near previous levels from Sept 21 on).
  • MobileWebLanguageSwitcher (somewhat smaller drop, but still clearly visible).
  • Edit: even less pronounced, but still visible (the maximum hourly values for Sept 9, 10 and 11 are lower than for every day since Jul 30)

Interesting! So regarding Popups, we can now assume the timing was by chance in both cases (the multiple hovers fix on Sept 8 and the ResourceLoader fix on Sept 21). A good reminder that coincidences can and do happen...

However, various other schemas don't show such a pattern, so it still doesn't seem to be an issue that affects all of EventLogging:

Also, why would this occur both on desktop and mobile web, but only for some browsers on desktop?

In any case though, I will file the above observations as a more general EL bug.

• Tbayer mentioned this in T146840: Various EventLogging schemas losing events since around September 8/9.Sep 28 2016, 12:34 AM
• Tbayer added a comment.Sep 28 2016, 5:30 AM
In T146620#2671235, @phuedx wrote:

Questions from post-standup discussion:

...

  • can we narrow down the version of Chrome that's affected?

No, it does not look like this is tied to a particular Chrome version:

pageLoaded events in the Popups schema (Chrome only, by version).png (465×977 px, 40 KB)

Data source (relying on another crudely handcrafted browser/version detection):

SELECT LEFT(timestamp, 8) AS date, 
SUBSTRING(userAgent,INSTR(userAgent,'Chrome/')+7,2) AS chrome_version,
COUNT(*) AS pageloaded_events
FROM log.Popups_15777589
WHERE wiki ='huwiki'
AND event_isAnon = 1
AND event_action = 'pageLoaded'
AND INSTR(userAgent,'Chrome')
GROUP BY date, chrome_version
ORDER BY date, chrome_version;

(versions with <= 200 events in this result are omitted in the above chart)

• bmansurov added a subscriber: ovasileva.Sep 28 2016, 11:41 AM

Thanks, @Tbayer for the detailed analysis.

@ovasileva do you think we can close this task as T146840 has been created?

ovasileva added a comment.Sep 28 2016, 12:01 PM

@bmansurov - I'm okay with that if seems nothing more we can do from our side. also, whew, at least this isn't only hovercards.

• bmansurov removed • bmansurov as the assignee of this task.Sep 28 2016, 12:02 PM
• bmansurov moved this task from Doing to Ready for Signoff on the Reading-Web-Sprint-82-Xpect-Rspec board.
ovasileva closed this task as Resolved.Sep 28 2016, 6:08 PM
ovasileva claimed this task.
Jdlrobson added subscribers: MBinder_WMF, Jdlrobson.Sep 28 2016, 6:25 PM

@MBinder_WMF should we re-estimate it? 8 seems like a lot for something which in the end only involved some investigation work.

MBinder_WMF added a comment.Sep 28 2016, 6:33 PM

@Jdlrobson appropriate. @bmansurov what do you think about for a new estimate?

• bmansurov added a comment.Sep 28 2016, 6:35 PM

I did only a small part of the work. Tilman did more. I'd say 2 or 3 points for the combined amount of work.

MBinder_WMF added a comment.Sep 28 2016, 7:56 PM

@bmansurov Well, we don't track Tilman's contribution. :) 2 points for Web Team effort?

• bmansurov changed the point value for this task from 8 to 2.Sep 28 2016, 8:22 PM

Sure, 2 sounds good.

• Tbayer unsubscribed.Oct 11 2016, 1:55 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL