Schema:Popups sends extraneous link interaction events in control condition
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Tbayer
	Mar 29 2017, 10:07 PM

Description

See this sample of 100 link interactions from the popupEnabled = 0 condition. Per definition, only one event should be sent per link interaction (token).

In most - but not all - of the below cases it appears that the same dwelledButAbandoned event was sent several times when there was a very brief interaction (mouse moving past a link rather than resting on it):

lItoken	action	totalInteractionTime
002c2586951fcc6c	dwelledButAbandoned	1226
006a3d1cb7623e69	dwelledButAbandoned	9
006f351c5f397888	dwelledButAbandoned	8
006f351c5f397888	dwelledButAbandoned	8
008c490dc502426d	dwelledButAbandoned	505
00b10a745965ff78	dwelledButAbandoned	15
01267915d7d05c66	dwelledButAbandoned	1135
0154f4fbab797b20	dwelledButAbandoned	14
0154f4fbab797b20	dwelledButAbandoned	14
0154f4fbab797b20	dwelledButAbandoned	14
015f5192a6b14326	dwelledButAbandoned	12
01647b5877f8648e	dwelledButAbandoned	32
01647b5877f8648e	dwelledButAbandoned	32
01df2dfc20290a78	dwelledButAbandoned	8484
01ea4e62332e96f8	dwelledButAbandoned	104
031fbc47a3424708	dwelledButAbandoned	30
039d0b98ae8ad55a	dwelledButAbandoned	1304
049fd826d3a3c18b	dwelledButAbandoned	400
04a41b523aaa7a81	dwelledButAbandoned	231
0507ffa49d0aa191	dwelledButAbandoned	1
0507ffa49d0aa191	dwelledButAbandoned	1
0580b7553fabbc43	dwelledButAbandoned	92
0580b7553fabbc43	dwelledButAbandoned	92
0580b7553fabbc43	dwelledButAbandoned	92
062f33fe5e422d93	dwelledButAbandoned	21
062f33fe5e422d93	dwelledButAbandoned	21
062f33fe5e422d93	dwelledButAbandoned	21
074ff6335084f687	dwelledButAbandoned	157
0779b2e2d04b55f1	dwelledButAbandoned	15
0779b2e2d04b55f1	dwelledButAbandoned	15
088ccf2166390eea	dwelledButAbandoned	6
088ccf2166390eea	dwelledButAbandoned	6
08ef3919854d1255	dwelledButAbandoned	48
08ef3919854d1255	dwelledButAbandoned	48
0957ca8f31cd5304	dwelledButAbandoned	47
0957ca8f31cd5304	dwelledButAbandoned	47
097ce08a11590ca1	dwelledButAbandoned	8
097ce08a11590ca1	dwelledButAbandoned	8
097ce08a11590ca1	dwelledButAbandoned	8
09f59d92b80ab05c	dwelledButAbandoned	82
0a22d9813b717574	opened	1319
0a3782e58ba05f8e	dwelledButAbandoned	45
0a5406f3cb80cf3d	dwelledButAbandoned	2133
0a759cb54be5b2c3	dwelledButAbandoned	160
0ad0e2586182046a	dwelledButAbandoned	1266
0b133983f65931a2	dwelledButAbandoned	663
0b37baa94f3f6f8c	dwelledButAbandoned	17
0b37baa94f3f6f8c	dwelledButAbandoned	17
0b390772bb0d0bc3	dwelledButAbandoned	10
0b72e5c65bdd38da	dwelledButAbandoned	33
0b9162d764925c3b	dwelledButAbandoned	4
0b9162d764925c3b	dwelledButAbandoned	4
0b9162d764925c3b	dwelledButAbandoned	4
0bf127f4bd3ed94a	dwelledButAbandoned	5
0bf127f4bd3ed94a	dwelledButAbandoned	5
0bf127f4bd3ed94a	dwelledButAbandoned	5
0c7b9320d8472047	dwelledButAbandoned	45
0c9bee8427969399	dwelledButAbandoned	24
0c9bee8427969399	dwelledButAbandoned	24
0cbe17f1abf25e11	dwelledButAbandoned	13
0d0bdba5f7a32f82	dwelledButAbandoned	566
0d7a6a6aa030c0fe	dwelledButAbandoned	853
0d99b8790fc889a4	dwelledButAbandoned	18
0d99b8790fc889a4	dwelledButAbandoned	18
0d99b8790fc889a4	dwelledButAbandoned	18
0d99b8790fc889a4	dwelledButAbandoned	18
0d99b8790fc889a4	dwelledButAbandoned	18
0dbf09b07609d151	dwelledButAbandoned	136
0e86b8978b753237	dwelledButAbandoned	5
0eb590f351983a48	dwelledButAbandoned	16
0eb590f351983a48	dwelledButAbandoned	16
0f133a1a96a48444	dwelledButAbandoned	16
0f3f92fff1b33319	dwelledButAbandoned	48
0fb0f62d1bc63dd0	dwelledButAbandoned	8
0fb0f62d1bc63dd0	dwelledButAbandoned	8
1040f79d22bed5fc	dwelledButAbandoned	376
113ebc2adc79862f	opened	2281
113ebc2adc79862f	dwelledButAbandoned	3208
11660779ecacb390	dwelledButAbandoned	9
11660779ecacb390	dwelledButAbandoned	9
11660779ecacb390	dwelledButAbandoned	1216
11b784205f9e837f	dwelledButAbandoned	6
11b784205f9e837f	dwelledButAbandoned	6
11b784205f9e837f	dwelledButAbandoned	6
11e1851587899c1c	dwelledButAbandoned	479
11e1851587899c1c	dwelledButAbandoned	479
1217140970ef3969	dwelledButAbandoned	18
12d03ade6d942294	dwelledButAbandoned	201
13542fa5f1be584a	dwelledButAbandoned	64
13ecdbe9a9d663c0	dwelledButAbandoned	24
13ecdbe9a9d663c0	dwelledButAbandoned	24
13ecdbe9a9d663c0	dwelledButAbandoned	24
13ecdbe9a9d663c0	dwelledButAbandoned	24
1482698f93dbc7a9	dwelledButAbandoned	30
1482698f93dbc7a9	dwelledButAbandoned	30
164b48618f47cb3c	dwelledButAbandoned	11
173296b21730f954	dwelledButAbandoned	365
178a7b099a4b32b0	dwelledButAbandoned	13
17a7ad7516f3f7cc	dwelledButAbandoned	2993
17df7dcb21091213	opened	6036

Data from today, source:

SELECT event_linkInteractionToken AS lItoken, 
event_action AS action, event_totalInteractionTime AS totalInteractionTime 
FROM log.Popups_16364296
WHERE event_popupEnabled = 0 AND event_isAnon = 1
AND event_linkInteractionToken IS NOT NULL
ORDER BY liToken, totalInteractionTime
LIMIT 100;

This is related to T159490: PagePreviews events system sends duplicate and incorrect events (but that task concerns only event_popupEnabled = 1) and to T140485: Schema:Popups logs "dwelledButAbandoned" and link open events at the same time (but that case only occurred once in the above sample).

Details

Subject	Repo	Branch	Lines +/-
Clear interaction after an event for it is logged in EL	mediawiki/extensions/Popups	master	+73 -10
eventLogging: Discard events with duplicate tokens	mediawiki/extensions/Popups	wmf/1.29.0-wmf.21	+30 -11
eventLogging: Discard events with duplicate tokens	mediawiki/extensions/Popups	wmf/1.30.0-wmf.1	+30 -11
eventLogging: Discard events with duplicate tokens	mediawiki/extensions/Popups	master	+30 -11

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dereckson	T68374 Enable Hovercards on se.wikimedia.org (Swedish chapter wiki)
Resolved	Jdlrobson	T70860 [GOAL] Graduate Page Previews feature (Popups extension) out of Beta Feature
Resolved	ovasileva	T154635 [EPIC] Deploy page previews to English and German Wikipedia
Resolved	ovasileva	T162672 Deploy page previews to 90% of users on all wikis but English and German
Resolved	• Tbayer	T158172 Analyze results of stage 0 page previews test
Resolved	pmiazga	T159490 PagePreviews events system sends duplicate and incorrect events
Resolved	• Tbayer	T161769 Schema:Popups sends extraneous link interaction events in control condition
Resolved	phuedx	T159539 [2 hours] PagePreviews `totalInteractionTime` for `opened` event is calculated incorrectly
Resolved	pmiazga	T162924 PagePreviews sending "dwelledButAbandoned" after "opened"
Resolved	phuedx	T173422 Investigate the increase in the number of requests to Swift after the Page Previews deploy

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Jdlrobson added a parent task: T159490: PagePreviews events system sends duplicate and incorrect events.Apr 5 2017, 5:27 PM

MBinder_WMF moved this task from Needs Analysis to Blocked Externally on the Reading-Web-Sprint-95 board.Apr 5 2017, 5:28 PM

ovasileva moved this task from Upcoming to 2016-17 Q3 on the Web-Team-Backlog board.Apr 5 2017, 5:53 PM

phuedx moved this task from Blocked Externally to To Do on the Reading-Web-Sprint-95 board.Apr 7 2017, 1:14 PM

In T161769#3142392, @phuedx wrote:

I suspect that the changes for T159490: PagePreviews events system sends duplicate and incorrect events will fix this as there's very little distinction between the enabled and disabled conditions in the EL-related part of the PP codebase.

As nearly all of the scenarios from T159490 are fixed and the associated fixes were deployed during yesterday's MediaWiki train deployment, now seems like a good time to verify. @HaeB: Is there enough data from yesterday evening (9 PM UTC onwards) to verify whether this is fixed?

phuedx assigned this task to HaeB.Apr 7 2017, 1:18 PM

As nearly all of the scenarios from T159490 are fixed and the associated fixes were deployed during yesterday's MediaWiki train deployment, now seems like a good time to verify. @HaeB: Is there enough data from yesterday evening (9 PM UTC onwards) to verify whether this is fixed?

This sounds like a "ready for sign off" activity and should be moved backwards into needs more work if we encounter an issue. Feel free to correct if I'm messing up the process :)

• Tbayer claimed this task.Apr 7 2017, 10:45 PM

• Tbayer removed a subscriber: HaeB.

^ Mibad. /me notes the difference between @Tbayer and @HaeB.

In T161769#3163765, @phuedx wrote:

In T161769#3142392, @phuedx wrote:

I suspect that the changes for T159490: PagePreviews events system sends duplicate and incorrect events will fix this as there's very little distinction between the enabled and disabled conditions in the EL-related part of the PP codebase.

As nearly all of the scenarios from T159490 are fixed and the associated fixes were deployed during yesterday's MediaWiki train deployment, now seems like a good time to verify. @HaeB: Is there enough data from yesterday evening (9 PM UTC onwards) to verify whether this is fixed?

The ratio of extraneous events dropped indeed a lot around that time (both for test and control). But there are still a quite a few of these occurring - I will need to take a closer look to determine if that's another still unresolved scenario or whether it can be chalked up to the general baseline of duplicate, identical events that occur in many schemas due to so far unclear but likely browser-related reason (T142667).

Data (via SWAP notebook) from:

SELECT dupesperhour.popupEnabled, dupesperhour.yearmonthdayhour, ROUND(100*dupes/totalevents,2) AS dupes_percentage FROM 
	(SELECT popupEnabled, yearmonthdayhour, SUM(copies-1) AS dupes FROM 
		(SELECT event_popupEnabled AS popupEnabled, LEFT(timestamp, 10) AS yearmonthdayhour, 
		timestamp, userAgent, event_linkInteractionToken, event_action, 
		COUNT(*) AS copies 
		FROM log.Popups_16364296
		WHERE event_isAnon = 1 
		AND event_linkInteractionToken IS NOT NULL
		GROUP BY popupEnabled, yearmonthdayhour, 
		timestamp, userAgent, event_linkInteractionToken, event_action
		HAVING copies>1) AS dupeslist 
	GROUP BY popupEnabled, yearmonthdayhour) AS dupesperhour
	JOIN
	(SELECT event_popupEnabled AS popupEnabled, LEFT(timestamp, 10) AS yearmonthdayhour, 
	COUNT(*) AS totalevents 
	FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_linkInteractionToken IS NOT NULL
	GROUP BY popupEnabled, yearmonthdayhour) AS totaleventsperhour
	ON dupesperhour.popupEnabled = totaleventsperhour.popupEnabled
        AND dupesperhour.yearmonthdayhour = totaleventsperhour.yearmonthdayhour
ORDER BY popupEnabled, yearmonthdayhour;

Below are two samples of the extraneous events that are still happening (in the control condition, from April 11 and April 10). It looks like the large majority indeed consist of the same event being recorded several times, but there are also several cases of a dwelledButAbandonedevent occuring after an opened event had already been sent (which should not happen per the state diagram).

There is various directions into which one could investigate from here - e.g. see if these cases happen more often for some browsers than others.

+------------------+---------------------+----------------------+
| token            | action              | totalInteractionTime |
+------------------+---------------------+----------------------+
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 020e04c4874c185b | dwelledButAbandoned |                   48 |
| 0239f34c7dfea51a | dwelledButAbandoned |                   32 |
| 0239f34c7dfea51a | dwelledButAbandoned |                   32 |
| 0239f34c7dfea51a | dwelledButAbandoned |                   32 |
| 023fa26eec72c715 | opened              |                  633 |
| 023fa26eec72c715 | opened              |                  633 |
| 023fa26eec72c715 | dwelledButAbandoned |                  869 |
| 023fa26eec72c715 | dwelledButAbandoned |                  869 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | opened              |                  537 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029949c69367d3be | dwelledButAbandoned |                  769 |
| 029c0604b8c1f8f2 | dwelledButAbandoned |                   16 |
| 029c0604b8c1f8f2 | dwelledButAbandoned |                   16 |
| 02af7566a1b7e5dd | opened              |                  328 |
| 02af7566a1b7e5dd | dwelledButAbandoned |                  438 |
| 02edb10b49b95b49 | dwelledButAbandoned |                   31 |
| 02edb10b49b95b49 | dwelledButAbandoned |                   31 |
| 02edb10b49b95b49 | dwelledButAbandoned |                   31 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | opened              |                  307 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0344b2e33e47a3de | dwelledButAbandoned |                  405 |
| 0498bb082d32856e | dwelledButAbandoned |                   39 |
| 0498bb082d32856e | dwelledButAbandoned |                   39 |
| 051076c0b472a756 | dwelledButAbandoned |                   24 |
| 051076c0b472a756 | dwelledButAbandoned |                   24 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 055bd09c25417101 | dwelledButAbandoned |                   21 |
| 05daee5f7202662d | dwelledButAbandoned |                   56 |
| 05daee5f7202662d | dwelledButAbandoned |                   56 |
| 05daee5f7202662d | dwelledButAbandoned |                   56 |
| 0608045031cea685 | dwelledButAbandoned |                  141 |
| 0608045031cea685 | dwelledButAbandoned |                  141 |
| 0608045031cea685 | dwelledButAbandoned |                  141 |
| 063aec90b357401d | dwelledButAbandoned |                  104 |
| 063aec90b357401d | dwelledButAbandoned |                  104 |
| 063aec90b357401d | dwelledButAbandoned |                  104 |
| 063aec90b357401d | dwelledButAbandoned |                  104 |
| 063aec90b357401d | dwelledButAbandoned |                  104 |
| 063dd3752fe9825e | dwelledButAbandoned |                  432 |
| 063dd3752fe9825e | dwelledButAbandoned |                  432 |
| 0688acd273da5b1f | dwelledButAbandoned |                   23 |
| 0688acd273da5b1f | dwelledButAbandoned |                   23 |
| 0688acd273da5b1f | dwelledButAbandoned |                   23 |
+------------------+---------------------+----------------------+
100 rows in set (2.12 sec)


SELECT dupetokenlist.event_linkInteractionToken AS token, event_action AS action, event_totalInteractionTime AS totalInteractionTime
FROM 
	(SELECT event_linkInteractionToken,	COUNT(*) AS copies 
	FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '20170411%'
	AND event_linkInteractionToken IS NOT NULL
	GROUP BY event_linkInteractionToken
	HAVING copies>1) AS dupetokenlist
JOIN 
	(SELECT * FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '20170411%'
	AND event_linkInteractionToken IS NOT NULL) AS allevents
ON dupetokenlist.event_linkInteractionToken = allevents.event_linkInteractionToken
ORDER BY token, totalInteractionTime
LIMIT 100;


+------------------+---------------------+----------------------+
| token            | action              | totalInteractionTime |
+------------------+---------------------+----------------------+
| 009d896a5f346269 | dwelledButAbandoned |                   95 |
| 009d896a5f346269 | dwelledButAbandoned |                   95 |
| 00f0bfac32177aea | opened              |                  976 |
| 00f0bfac32177aea | opened              |                  976 |
| 00f0bfac32177aea | dwelledButAbandoned |                 1058 |
| 00f0bfac32177aea | dwelledButAbandoned |                 1058 |
| 02877e97f54e0c24 | dwelledButAbandoned |                   16 |
| 02877e97f54e0c24 | dwelledButAbandoned |                   16 |
| 083ba7a51cca4198 | dwelledButAbandoned |                  297 |
| 083ba7a51cca4198 | dwelledButAbandoned |                  297 |
| 09ed55558d9d7895 | dwelledButAbandoned |                   27 |
| 09ed55558d9d7895 | dwelledButAbandoned |                   27 |
| 09ed55558d9d7895 | dwelledButAbandoned |                   27 |
| 0d42c77b105d4757 | dwelledButAbandoned |                   29 |
| 0d42c77b105d4757 | dwelledButAbandoned |                   29 |
| 11497d69635b14b1 | opened              |                  829 |
| 11497d69635b14b1 | dwelledButAbandoned |                 1000 |
| 119249c89407260c | dwelledButAbandoned |                   25 |
| 119249c89407260c | dwelledButAbandoned |                   25 |
| 120d97b17103960f | dwelledButAbandoned |                   18 |
| 120d97b17103960f | dwelledButAbandoned |                   18 |
| 1494134f0d7f77ff | dwelledButAbandoned |                   15 |
| 1494134f0d7f77ff | dwelledButAbandoned |                   15 |
| 149746b071c0ac7c | dwelledButAbandoned |                  617 |
| 149746b071c0ac7c | dwelledButAbandoned |                  617 |
| 15d305570cb0757b | dwelledButAbandoned |                   16 |
| 15d305570cb0757b | dwelledButAbandoned |                   16 |
| 16d31eb985e316f4 | opened              |                  644 |
| 16d31eb985e316f4 | dwelledButAbandoned |                  756 |
| 17d7aa268e581540 | opened              |                 1264 |
| 17d7aa268e581540 | opened              |                 1264 |
| 17d7aa268e581540 | opened              |                 1264 |
| 17d7aa268e581540 | dwelledButAbandoned |                 1352 |
| 17d7aa268e581540 | dwelledButAbandoned |                 1352 |
| 17d7aa268e581540 | dwelledButAbandoned |                 1352 |
| 1f1a852df7466b40 | dwelledButAbandoned |                   23 |
| 1f1a852df7466b40 | dwelledButAbandoned |                   23 |
| 1ff5fc93363a6fc8 | dwelledButAbandoned |                   25 |
| 1ff5fc93363a6fc8 | dwelledButAbandoned |                   25 |
| 1ff5fc93363a6fc8 | dwelledButAbandoned |                   25 |
| 26427fd7bfdb4281 | dwelledButAbandoned |                24926 |
| 26427fd7bfdb4281 | dwelledButAbandoned |                24926 |
| 26427fd7bfdb4281 | dwelledButAbandoned |                24926 |
| 2ddcc24a5d785d35 | dwelledButAbandoned |                   17 |
| 2ddcc24a5d785d35 | dwelledButAbandoned |                   17 |
| 2f146a12b4bd30f1 | dwelledButAbandoned |                    0 |
| 2f146a12b4bd30f1 | dwelledButAbandoned |                    0 |
| 2f8d1c16e84ecc45 | dwelledButAbandoned |                 1208 |
| 2f8d1c16e84ecc45 | dwelledButAbandoned |                 1208 |
| 300f8f8c0a7f003f | dwelledButAbandoned |                   16 |
| 300f8f8c0a7f003f | dwelledButAbandoned |                   16 |
| 3913a8e88f3d55da | dwelledButAbandoned |                    8 |
| 3913a8e88f3d55da | dwelledButAbandoned |                    8 |
| 3a0470c268fbe685 | dwelledButAbandoned |                   31 |
| 3a0470c268fbe685 | dwelledButAbandoned |                   31 |
| 3ceece791c467c31 | dwelledButAbandoned |                   18 |
| 3ceece791c467c31 | dwelledButAbandoned |                   18 |
| 3f5f48d2981baae3 | dwelledButAbandoned |                  449 |
| 3f5f48d2981baae3 | dwelledButAbandoned |                  449 |
| 3f5f48d2981baae3 | dwelledButAbandoned |                  449 |
| 407f0c966d97a902 | dwelledButAbandoned |                   32 |
| 407f0c966d97a902 | dwelledButAbandoned |                   32 |
| 435e2de223915837 | dwelledButAbandoned |                   12 |
| 435e2de223915837 | dwelledButAbandoned |                   12 |
| 52edd9fc311dbde5 | opened              |                  790 |
| 52edd9fc311dbde5 | dwelledButAbandoned |                 2386 |
| 544819cb48a5a557 | dwelledButAbandoned |                   16 |
| 544819cb48a5a557 | dwelledButAbandoned |                   16 |
| 56b977a870c21bb9 | dwelledButAbandoned |                   16 |
| 56b977a870c21bb9 | dwelledButAbandoned |                   16 |
| 59b5dc6cb9d7ff05 | dwelledButAbandoned |                   39 |
| 59b5dc6cb9d7ff05 | dwelledButAbandoned |                   39 |
| 623e721315bd3813 | dwelledButAbandoned |                   12 |
| 623e721315bd3813 | dwelledButAbandoned |                   12 |
| 652bdb22f08ee74d | dwelledButAbandoned |                   54 |
| 652bdb22f08ee74d | dwelledButAbandoned |                   54 |
| 652bdb22f08ee74d | dwelledButAbandoned |                   54 |
| 652f0d8ebf2494a7 | dwelledButAbandoned |                   15 |
| 652f0d8ebf2494a7 | dwelledButAbandoned |                   15 |
| 660a2713ea1aaa1c | dwelledButAbandoned |                   11 |
| 660a2713ea1aaa1c | dwelledButAbandoned |                   11 |
| 660a2713ea1aaa1c | dwelledButAbandoned |                   11 |
| 682db87cca491b64 | dwelledButAbandoned |                    0 |
| 682db87cca491b64 | dwelledButAbandoned |                    0 |
| 7298757ca0db09fb | dwelledButAbandoned |                    0 |
| 7298757ca0db09fb | dwelledButAbandoned |                    0 |
| 76ddc9e2e643e515 | dwelledButAbandoned |                   10 |
| 76ddc9e2e643e515 | dwelledButAbandoned |                   10 |
| 76ddc9e2e643e515 | dwelledButAbandoned |                   10 |
| 7793518d7310182d | opened              |                  466 |
| 7793518d7310182d | opened              |                  466 |
| 7793518d7310182d | opened              |                  466 |
| 7793518d7310182d | dwelledButAbandoned |               595009 |
| 7793518d7310182d | dwelledButAbandoned |               595009 |
| 7793518d7310182d | dwelledButAbandoned |               595009 |
| 7c2d76bda5a66830 | dwelledButAbandoned |                   20 |
| 7c2d76bda5a66830 | dwelledButAbandoned |                   20 |
| 802ec4c643dc86d5 | dwelledButAbandoned |                 1310 |
| 802ec4c643dc86d5 | dwelledButAbandoned |                 1310 |
| 81d0e6b874125b42 | opened              |                 8060 |
+------------------+---------------------+----------------------+
100 rows in set (0.64 sec)


SELECT dupetokenlist.event_linkInteractionToken AS token, event_action AS action, event_totalInteractionTime AS totalInteractionTime
FROM 
	(SELECT event_linkInteractionToken,	COUNT(*) AS copies 
	FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '20170410%'
	AND event_linkInteractionToken IS NOT NULL
	GROUP BY event_linkInteractionToken
	HAVING copies>1) AS dupetokenlist
JOIN 
	(SELECT * FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '20170410%'
	AND event_linkInteractionToken IS NOT NULL) AS allevents
ON dupetokenlist.event_linkInteractionToken = allevents.event_linkInteractionToken
ORDER BY token, totalInteractionTime
LIMIT 100;

And here is an updated chart (source as above)

• Tbayer removed a subscriber: HaeB.Apr 13 2017, 3:04 PM

In T161769#3178477, @Tbayer wrote:

there are also several cases of a dwelledButAbandonedevent occuring after an opened event had already been sent (which should not happen per the state diagram).

As discussed, this was brought up in T159490: PagePreviews events system sends duplicate and incorrect events and consequently fixed in rEPOP83e32c255d40: reducers: Make LINK_CLICK finalize interaction. That'll be fully deployed by Thursday, 27th April.

So @phuedx suggested to check if the number of events by minute shown in Grafana matches the number of rows in the MySQL table. Here is a first quick take on this for the time between 10:00-10:59 UTC on April 13, with the MySQL side restricted to the current version of the schema (Popups_16364296; there may be older versions still sending events and AIUI the Grafana chart captures all of them, but at least the number in the left column should always be less or equal than the number in the right column).

TLDR: They differ quite a lot. (And OTOH the bump after around 10:30 makes one think it is not a mere timezone issue.)

yearmonthdayminute	Popups_16364296	Grafana mouseover
201704131000	79	62
201704131001	62	69
201704131002	30	72
201704131003	15	69
201704131004	47	62
201704131005	79	54
201704131006	46	50
201704131007	39	47
201704131008	27	46
201704131009	52	47
201704131010	64	46
201704131011	17	47
201704131012	59	43
201704131013	92	39
201704131014	101	47
201704131015	49	52
201704131016	47	56
201704131017	60	59
201704131018	61	65
201704131019	78	62
201704131020	78	64
201704131021	35	61
201704131022	50	61
201704131023	36	59
201704131024	87	55
201704131025	102	53
201704131026	89	56
201704131027	131	61
201704131028	127	72
201704131029	192	89
201704131030	171	116
201704131031	140	128
201704131032	182	137
201704131033	407	141
201704131034	320	179
201704131035	157	186
201704131036	51	213
201704131037	36	217
201704131038	79	205
201704131039	16	154
201704131040	11	118
201704131041	32	69
201704131042	47	36
201704131043	26	40
201704131044	85	36
201704131045	77	34
201704131046	68	45
201704131047	59	55
201704131048	85	57
201704131049	90	65
201704131050	88	73
201704131051	33	75
201704131052	88	70
201704131053	65	73
201704131054	105	75
201704131055	49	76
201704131056	64	70
201704131057	63	72
201704131058	73	71
201704131059	47	70

Data sources:
SELECT LEFT(timestamp, 12) AS yearmonthdayminute, COUNT(*)
FROM log.Popups_16364296
WHERE timestamp LIKE '2017041310%'
GROUP BY yearmonthdayminute
ORDER BY yearmonthdayminute;

and
https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?var-schema=Popups&refresh=5m&orgId=1&from=1492077044526&to=1492082282710 - extracted manually by mouseover. If someone shows me a way to export data from Grafana in CSV form, I'll do a more systematic analysis for longer timespans.

In T161769#3178477, @Tbayer wrote:

Below are two samples of the extraneous events that are still happening (in the control condition, from April 11 and April 10). It looks like the large majority indeed consist of the same event being recorded several times, but there are also several cases of a dwelledButAbandonedevent occuring after an opened event had already been sent (which should not happen per the state diagram).

The latter has now been (discovered independently and) filed by @bmansurov in T162924 .

ovasileva moved this task from 2016-17 Q3 to 2016-17 Q4 on the Web-Team-Backlog board.Apr 13 2017, 7:10 PM

@Tbayer can we resolve this task? Does T162924 capture the remaining concern?

In T161769#3179293, @Tbayer wrote:

(Popups_16364296; there may be older versions still sending events and AIUI the Grafana chart captures all of them, but at least the number in the left column should always be less or equal than the number in the right column).

I only thought about this briefly but I'd expected to see the Grafana numbers to be consistently lower than those from MySQL:

09:34:19 <phuedx> i'd expect the grafana graph to be less than or equal to the number of events stored in mysql
09:34:45 <phuedx> as the grafana graph is a plot of an aggregate generated by statsd
09:34:53 <phuedx> and data gets into statsd via udp
09:35:05 <phuedx> which can be lossy
09:35:31 <phuedx> if there's network congestion or (statsd) server resource congestion for example

TLDR: They differ quite a lot. (And OTOH the bump after around 10:30 makes one think it is not a mere timezone issue.)

Moreover, they're consistently inconsistent!

@Tbayer: Do we have access to the raw request logs from the Varnishes? If so, then we could also verify that duplicates are being sent by the client by finding duplicate requests to the /beacon URL.

phuedx mentioned this in T163198: Track instances of duplicate Popups events being logged.Apr 18 2017, 12:00 PM

We discussed a subtask of/follow-up to this task in the weekly Page Previews Sync meeting: instrumenting the number of duplicate events seen by the client-side logging code. This is captured in T163198: Track instances of duplicate Popups events being logged.

ovasileva moved this task from Ready for Signoff to Needs QA on the Reading-Web-Sprint-95 board.Apr 18 2017, 5:05 PM

MBinder_WMF added a project: Reading-Web-Sprint-96.Apr 19 2017, 5:12 PM

MBinder_WMF moved this task from Needs Analysis to Needs QA on the Reading-Web-Sprint-96 board.

In T161769#3189508, @phuedx wrote:

@Tbayer: Do we have access to the raw request logs from the Varnishes? If so, then we could also verify that duplicates are being sent by the client by finding duplicate requests to the /beacon URL.

Yes. Here are the rows in the webrequest table corresponding to the first token from the extraneous events sample above (query: [1]).
TLDR: In that example, it's all consistent with the client sending duplicate events (11 identical ones over a timespan of 16 minutes).

dt	time_firstbyte	hostname	sequence
2017-04-11T14:57:41	5.1E-5	cp3033.esams.wmnet	3284147892
2017-04-11T15:01:09	4.1E-5	cp3033.esams.wmnet	3285196376
2017-04-11T15:01:50	4.8E-5	cp3033.esams.wmnet	3285402705
2017-04-11T15:02:50	4.5E-5	cp3033.esams.wmnet	3285696942
2017-04-11T15:03:41	5.1E-5	cp3033.esams.wmnet	3285950824
2017-04-11T15:07:12	5.1E-5	cp3033.esams.wmnet	3286999116
2017-04-11T15:07:29	5.7E-5	cp3033.esams.wmnet	3287082690
2017-04-11T15:09:46	4.4E-5	cp3033.esams.wmnet	3287765241
2017-04-11T15:11:16	4.5E-5	cp3033.esams.wmnet	3288213227
2017-04-11T15:11:51	4.3E-5	cp3033.esams.wmnet	3288382799
2017-04-11T15:13:00	5.1E-5	cp3033.esams.wmnet	3288727086
11 rows selected (154.198 seconds)

The request URL was identical in all cases (verified via [2]), and as far as I can see, all the other webrequest fields too (apart from the ts timestamp which corresponds to dt). The user agent in this case is from Firefox 52 on Windows 10.

For comparison, here are the timestamps of the corresponding entries in the Popups table (query: [3]). The timestamps all match.

+----------------+----------------------------+
| timestamp      | event_totalInteractionTime |
+----------------+----------------------------+
| 20170411145741 |                         48 |
| 20170411150109 |                         48 |
| 20170411150150 |                         48 |
| 20170411150250 |                         48 |
| 20170411150341 |                         48 |
| 20170411150712 |                         48 |
| 20170411150729 |                         48 |
| 20170411150946 |                         48 |
| 20170411151116 |                         48 |
| 20170411151151 |                         48 |
| 20170411151300 |                         48 |
+----------------+----------------------------+
11 rows in set (0.40 sec)

[1]

SELECT dt, time_firstbyte, hostname, sequence FROM wmf.webrequest
WHERE year = 2017 AND month = 4 AND day = 11
AND uri_query LIKE '%020e04c4874c185b%'
ORDER BY dt
LIMIT 100;

[2]

SELECT DISTINCT uri_query
FROM wmf.webrequest
WHERE year = 2017 AND month = 4 AND day = 11
AND uri_query LIKE '%020e04c4874c185b%'
LIMIT 100;

[3]

SELECT timestamp, event_totalInteractionTime
FROM log.Popups_16364296
WHERE timestamp LIKE '20170411%'
AND event_linkInteractionToken = '020e04c4874c185b'
ORDER BY timestamp;

In T161769#3178477, @Tbayer wrote:

...

There is various directions into which one could investigate from here - e.g. see if these cases happen more often for some browsers than others.

Almost all of them - ca. 98% - happen on Firefox (matching earlier informal observations), also with the current version (52).
On the other hand, the distribution by OS doesn't seem to deviate too much from the general distribution.

SELECT SUBSTRING( userAgent, INSTR(userAgent,'"browser_family": "')+19, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "')+19), '"' )-1 ) AS browser_family, COUNT(*) AS tokens_with_dupes FROM (
    SELECT userAgent, event_linkInteractionToken, COUNT(*) AS events 
	FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '2017041%'
	AND event_linkInteractionToken IS NOT NULL
	GROUP BY event_linkInteractionToken # userAgent should be identical if the token is
	HAVING events>1) AS dupetokenlist
GROUP BY browser_family
ORDER BY tokens_with_dupes DESC;

+----------------+-------------------+
| browser_family | tokens_with_dupes |
+----------------+-------------------+
| Firefox        |              3165 |
| Chrome         |                54 |
| Opera          |                14 |
| Yandex Browser |                 9 |
| Edge           |                 3 |
| Chromium       |                 1 |
+----------------+-------------------+
6 rows in set (1.99 sec)



SELECT 
SUBSTRING( userAgent, INSTR(userAgent,'"browser_family": "')+19, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "')+19), '"' )-1 ) AS browser_family, 
SUBSTRING( userAgent, INSTR(userAgent,'"browser_major": "')+18, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_major": "')+18), '"' )-1 ) AS browser_major, 
COUNT(*) AS tokens_with_dupes 
FROM (
    SELECT userAgent, event_linkInteractionToken, COUNT(*) AS events 
	FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '2017041%'
	AND event_linkInteractionToken IS NOT NULL
	GROUP BY event_linkInteractionToken # userAgent should be identical if the token is
	HAVING events>1) AS dupetokenlist
GROUP BY browser_family, browser_major
ORDER BY tokens_with_dupes DESC;

+----------------+---------------+-------------------+
| browser_family | browser_major | tokens_with_dupes |
+----------------+---------------+-------------------+
| Firefox        | 52            |              3061 |
| Firefox        | 51            |                46 |
| Chrome         | 57            |                38 |
| Firefox        | 42            |                13 |
| Firefox        | 47            |                12 |
| Firefox        | 50            |                10 |
| Yandex Browser | 17            |                 9 |
| Chrome         | 49            |                 7 |
| Firefox        | 45            |                 7 |
| Firefox        | 48            |                 5 |
| Opera          | 44            |                 4 |
| Chrome         | 43            |                 3 |
| Edge           | 14            |                 3 |
| Opera          | 43            |                 3 |
| Opera          | 36            |                 3 |
| Firefox        | 38            |                 2 |
| Firefox        | 49            |                 2 |
| Chrome         | 55            |                 2 |
| Firefox        | 32            |                 2 |
| Chrome         | 48            |                 2 |
| Opera          | 29            |                 2 |
| Opera          | 34            |                 2 |
| Chrome         | 51            |                 2 |
| Firefox        | 41            |                 1 |
| Chromium       | 57            |                 1 |
| Chrome         | 56            |                 1 |
| Firefox        | 44            |                 1 |
| Firefox        | 46            |                 1 |
| Firefox        | 34            |                 1 |
| Firefox        | 53            |                 1 |
+----------------+---------------+-------------------+
30 rows in set (1.38 sec)



SELECT SUBSTRING( userAgent, INSTR(userAgent,'"os_family": "')+14, INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"os_family": "')+14), '"' )-1 ) AS os_family, 
COUNT(*) AS tokens_with_dupes 
FROM (
    SELECT userAgent, event_linkInteractionToken, COUNT(*) AS events 
	FROM log.Popups_16364296
	WHERE event_isAnon = 1 
	AND event_popupEnabled = 0
	AND timestamp LIKE '2017041%'
	AND event_linkInteractionToken IS NOT NULL
	GROUP BY event_linkInteractionToken # userAgent should be identical if the token is
	HAVING events>1) AS dupetokenlist
GROUP BY os_family
ORDER BY tokens_with_dupes DESC;

+---------------+-------------------+
| os_family     | tokens_with_dupes |
+---------------+-------------------+
| Windows 7     |              1399 |
| Windows 10    |               770 |
| Windows 8.1   |               379 |
| Windows XP    |               348 |
| Mac OS X      |               128 |
| Windows 8     |               117 |
| Ubuntu        |                63 |
| Windows Vista |                34 |
| Linux         |                 8 |
+---------------+-------------------+
9 rows in set (1.20 sec)

In T161769#3179293, @Tbayer wrote:

...

If someone shows me a way to export data from Grafana in CSV form, I'll do a more systematic analysis for longer timespans.

If you click the title of a graph, a menu will appear. The left-most button of the menu is a hamburger. Click the hamburger and you'll see the following:

Screen Shot 2017-04-20 at 07.56.33.png (170×286 px, 16 KB)

!!1

phuedx mentioned this in T148850: Sampling sanity check for Schema:Popups pageview events.Apr 20 2017, 10:13 AM

Progress! Excellent! A couple of thoughts:

We're using 16 character hex tokens, which have a probably of collision of 1% in 600,000,000 tokens generated. Given the number of PVs across the projects, this seems in line with the number of duplicate tokens from other browser families (first table from T161769#3195922). It would be trivial to double the length of these tokens.
We know that there's something wrong with either the PP/EventLogging code running in Firefox or that WebKit is guarding against bad behaviour –since Firefox is an evergreen browser, I'm less concerned with the exact version number. We also know that it's not just duplicate tokens that are being sent but entire duplicate events

In T161769#3194893, @Tbayer wrote:

The request URL was identical in all cases (verified via [2]), and as far as I can see, all the other webrequest fields too (apart from the ts timestamp which corresponds to dt).

This should inform T163198: Track instances of duplicate Popups events being logged.

Jdlrobson mentioned this in T162924: PagePreviews sending "dwelledButAbandoned" after "opened".Apr 24 2017, 6:01 PM

At the very worst, @Tbayer's queries can be re-run on Friday morning after Thursday's MediaWiki train deployment.

In T161769#3197638, @phuedx wrote:

Progress! Excellent! A couple of thoughts:

We're using 16 character hex tokens, which have a probably of collision of 1% in 600,000,000 tokens generated. Given the number of PVs across the projects, this seems in line with the number of duplicate tokens from other browser families (first table from T161769#3195922). It would be trivial to double the length of these tokens.

But the number of (non-distinct) tokens generated in that example was much lower than 600,000,000 , namely 63,340 [1] . Per the linked table, that means a random collision probability between 0.0000001% and 0.0001%. In other words, even the extraneous events for non-Firefox browsers in T161769#3195922 are unlikely to have resulted from a birthday issue, and increasing the token length is probably unnecessary in this particular context. - In any case, as discussed last week, the Firefox events are the main problem, and once it is fixed, the small number of remaining duplicates for other browsers will probably be tolerable in this case.

[1]

SELECT COUNT(*) AS events_with_linkInteractionToken
FROM log.Popups_16364296
WHERE event_isAnon = 1 
AND event_popupEnabled = 0
AND timestamp LIKE '2017041%'
AND event_linkInteractionToken IS NOT NULL;
+----------------------------------+
| events_with_linkInteractionToken |
+----------------------------------+
|                            63340 |
+----------------------------------+
1 row in set (4.08 sec)

(Includes a few hours of April 19 that were not yet covered in the corresponding query at T161769#3195922, so the number is slightly higher here.)

In T161769#3214830, @phuedx wrote:

At the very worst, @Tbayer's queries can be re-run on Friday morning after Thursday's MediaWiki train deployment.

Which changes are part of that deployment?

In T161769#3216024, @Tbayer wrote:

Which changes are part of that deployment?

Per https://github.com/wikimedia/mediawiki-extensions-Popups/compare/wmf/1.29.0-wmf.20...wmf/1.29.0-wmf.21, all of the changes for T159490: PagePreviews events system sends duplicate and incorrect events and T162924: PagePreviews sending "dwelledButAbandoned" after "opened".

In T161769#3217103, @phuedx wrote:

In T161769#3216024, @Tbayer wrote:

Which changes are part of that deployment?

Per https://github.com/wikimedia/mediawiki-extensions-Popups/compare/wmf/1.29.0-wmf.20...wmf/1.29.0-wmf.21, all of the changes for T159490: PagePreviews events system sends duplicate and incorrect events and T162924: PagePreviews sending "dwelledButAbandoned" after "opened".

OK, IIRC we did not expect these to fix the majority of extraneous events recorded in the charts above (which we assume to be similar to the example in T161769#3194893 , i.e. the same event - identical beacon URL - sent multiple times), and we're mostly still waiting for the deployment of T163198: Track instances of duplicate Popups events being logged.

But I re-ran the check (data in the SWAP notebook is updated, will post charts here later), and there doesn't seem to have been a big change in those rates after Thursday's deployment. E.g. on May 1 the duplicates ratio for Popups ON reached 27.73% between 4 and 5h UTC , and 13.88% between 21 and 22h UTC.

PS: Here is another concrete example (taken from the second timespan mentioned in the preceding comment), this time querying the EL table instead of webrequest. UA is from Firefox 53 on Windows 10.

SELECT timestamp, event_action AS action, 
event_totalInteractionTime AS totalInteractionTime
FROM log.Popups_16364296
WHERE event_linkInteractionToken  = '0432265bb4341bab'
ORDER BY timestamp;
+----------------+---------------------+----------------------+
| timestamp      | action              | totalInteractionTime |
+----------------+---------------------+----------------------+
| 20170501212000 | dwelledButAbandoned |                   52 |
| 20170501212000 | dwelledButAbandoned |                   52 |
| 20170501212000 | dwelledButAbandoned |                   52 |
| 20170501212000 | dwelledButAbandoned |                   52 |
| 20170501212000 | dwelledButAbandoned |                   52 |
| 20170501212043 | dwelledButAbandoned |                   52 |
| 20170501212240 | dwelledButAbandoned |                   52 |
| 20170501212851 | dwelledButAbandoned |                   52 |
+----------------+---------------------+----------------------+
8 rows in set (3.11 sec)

In T161769#3227372, @Tbayer wrote:

OK, IIRC we did not expect these to fix the majority of extraneous events recorded in the charts above (which we assume to be similar to the example in T161769#3194893 , i.e. the same event - identical beacon URL - sent multiple times), and we're mostly still waiting for the deployment of T163198: Track instances of duplicate Popups events being logged.

Yes. The clear majority of extraneous events recorded in the charts above are caused by multiple identical beacons being sent. I'd note that after the deploy of T159490: PagePreviews events system sends duplicate and incorrect events and T162924: PagePreviews sending "dwelledButAbandoned" after "opened", the relatively low-frequency duplication of tokens in all other browsers have mostly dropped off. I say mostly because there's still some duplication [0][1].

[0]

SELECT event_linkInteractionToken, COUNT(*) AS n
FROM log.Popups_16364296
WHERE timestamp LIKE '20170501%'
AND event_linkInteractionToken IS NOT NULL
AND userAgent NOT LIKE '%Firefox%'
GROUP BY event_linkInteractionToken # userAgent should be identical if the token is
HAVING n > 1;

[1]

SELECT timestamp, event_action AS action, event_totalInteractionTime AS totalInteractionTime
FROM log.Popups_16364296
WHERE event_linkInteractionToken  = '467637990c45342b'
ORDER BY timestamp;

+----------------+--------+----------------------+
| timestamp      | action | totalInteractionTime |
+----------------+--------+----------------------+
| 20170501100615 | opened |                 1499 |
| 20170501100617 | opened |                 4204 |
| 20170501100618 | opened |                 4669 |
| 20170501100618 | opened |                 4700 |
| 20170501100618 | opened |                 4643 |
+----------------+--------+----------------------+
5 rows in set (1.36 sec)

• NHarateh_WMF added a project: Readers-Web-Kanbanana-Board-Old.May 4 2017, 5:36 PM

ovasileva moved this task from To Do to Blocked on Others on the Readers-Web-Kanbanana-Board-Old board.May 4 2017, 5:38 PM

Part of T163198: Track instances of duplicate Popups events being logged was deduplicating events on the client side. Currently, two events are considered identical if they serialise to the same value, not if they share a token. The associated logging is enabled on all wikis for the duration of 1% of all distinct browser sessions of those users who have Page Previews loaded (not necessarily enabled). By contrast, the EL-based logging is enabled for 0.1% of those browser sessions.

Since yesterday's Morning SWAT deployment, there've been a number of duplicate events enqueued – but not logged!!1 – by the PP instrumentation. This is definitely a bug in the eventLogging reducer/changeListener pair.

Deduplication events based on identity hasn't had any effect on the queries @Tbayer ran in T161769#3195922 [0], nor has it stopped identical events arriving at Varnish [1][2].

[0]

SELECT
  SUBSTRING(
    userAgent,
    INSTR(userAgent,'"browser_family": "') + 19,
    INSTR(SUBSTRING(userAgent, INSTR(userAgent,'"browser_family": "') +19 ), '"' ) - 1
  ) AS browser_family,
  COUNT(*) AS tokens_with_dupes
FROM (
  SELECT userAgent, event_linkInteractionToken, COUNT(*) AS events 
  FROM log.Popups_16364296
  WHERE timestamp LIKE '20170509%'
  AND event_linkInteractionToken IS NOT NULL
  # N.B. that this isn't restricted to logged out users with PP disabled.
  GROUP BY event_linkInteractionToken
  HAVING events > 1
) AS dupetokenlist
GROUP BY browser_family
ORDER BY tokens_with_dupes DESC;

+----------------+-------------------+
| browser_family | tokens_with_dupes |
+----------------+-------------------+
| Firefox        |               632 |
| Chrome         |                 5 |
| Chrome Mobile  |                 1 |
| Yandex Browser |                 1 |
+----------------+-------------------+

[1]

SELECT dt, time_firstbyte, hostname, sequence
FROM wmf.webrequest
WHERE year = 2017 AND month = 5 AND day = 9
AND uri_query LIKE '%5d6350a679c2b398%'
ORDER BY dt
LIMIT 100;

dt                      time_firstbyte  hostname                sequence
2017-05-09T06:55:56     4.8E-5          cp3033.esams.wmnet      64811278
2017-05-09T06:55:56     5.4E-5          cp3033.esams.wmnet      64811326
2017-05-09T06:55:56     5.2E-5          cp3033.esams.wmnet      64811377
2017-05-09T06:55:56     5.0E-5          cp3033.esams.wmnet      64811376
2017-05-09T06:55:56     5.5E-5          cp3033.esams.wmnet      64811325
2017-05-09T06:55:56     5.1E-5          cp3033.esams.wmnet      64811382
2017-05-09T06:55:56     4.6E-5          cp3033.esams.wmnet      64811277

[2]

SELECT DISTINCT uri_query
FROM wmf.webrequest
WHERE year = 2017 AND month = 5 AND day = 9
AND uri_query LIKE '%5d6350a679c2b398%'
LIMIT 100;

In T161769#3247604, @phuedx wrote:

...

Deduplication events based on identity hasn't had any effect on the queries @Tbayer ran in T161769#3195922 [0], nor has it stopped identical events arriving at Varnish [1][2].

I took a closer look at your example (the link interaction with token 5d6350a679c2b398) and inspected its context in EL, by querying all events recorded for the same pageview (PageToken = 52b0f6c07ae0e926). There are other weird things there besides the duplicates, for example:

more than 50 different (!) events recorded within one second (06:55:32).
7 different pageLoaded events (with the same page token) over the course of 2 minutes and 24 seconds (pageToken is meant to be regenerated for each new page load). Maybe that's another case of event duplication though (I didn't check if all fields are identical).

On the other hand, all 511 events from this one pageview/page token have the same source page name, as it should be. This is in the control condidtion BTW (popupEnabled = 0).

Once the pageTitleHover field become enable we will be able to do further checks on whether such sequences of events can plausibly reflect actual user behavior, but this example oozes weirdness already.

SELECT CONCAT( SUBSTRING(timestamp, 9, 2), ':', SUBSTRING(timestamp, 11, 2), ':', RIGHT(timestamp,2)) AS hhmmss,
event_linkInteractionToken AS lIaToken, 
event_action AS action,
event_totalInteractionTime AS tIaTime
FROM log.Popups_16364296 
WHERE event_PageToken ='52b0f6c07ae0e926'
AND LEFT(timestamp, 8) = '20170509'
ORDER BY hhmmss, lIaToken, action, tIaTime;

+----------+------------------+---------------------+---------+
| hhmmss   | lIaToken         | action              | tIaTime |
+----------+------------------+---------------------+---------+
| 06:53:20 | NULL             | pageLoaded          |    NULL |
| 06:53:31 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:53:31 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:53:32 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:53:33 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:53:35 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:54:32 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:54:32 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:54:32 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:54:33 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:54:42 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:54:42 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:54:42 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:54:42 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:54:42 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:54:42 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:54:42 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:54:43 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:54:43 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:54:44 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:54:45 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:54:46 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:54:46 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:54:46 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:54:49 | 82b2c3fa5e731743 | opened              |     364 |
| 06:54:49 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:54:58 | NULL             | pageLoaded          |    NULL |
| 06:54:58 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:54:58 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:54:58 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:54:58 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:54:58 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:54:58 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:54:58 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:54:58 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:54:58 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:54:58 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:54:58 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:54:58 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:54:58 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:54:58 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:54:58 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:54:58 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:54:58 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:54:58 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:54:58 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:54:58 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:54:59 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:54:59 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:54:59 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:54:59 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:54:59 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:54:59 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:54:59 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:54:59 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:54:59 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:54:59 | 82b2c3fa5e731743 | opened              |     364 |
| 06:54:59 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:55:00 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:00 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:01 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:01 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:01 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:01 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:02 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:02 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:02 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:02 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:02 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:02 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:02 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:02 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:03 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:03 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:03 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:03 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:03 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:03 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:03 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:03 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:04 | 87326a4f0a790cca | opened              |     354 |
| 06:55:04 | 87326a4f0a790cca | opened              |     354 |
| 06:55:06 | NULL             | pageLoaded          |    NULL |
| 06:55:06 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:55:06 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:55:06 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:55:06 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:55:06 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:55:06 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:55:06 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:55:06 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:55:06 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:55:06 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:55:06 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:55:06 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:55:07 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:55:07 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:55:07 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:55:07 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:55:07 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:55:07 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:55:07 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:07 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:07 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:07 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:55:07 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:55:07 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:55:07 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:07 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:55:07 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:55:07 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:07 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:07 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:07 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:55:07 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:07 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:07 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:07 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:07 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:07 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:07 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:07 | 82b2c3fa5e731743 | opened              |     364 |
| 06:55:07 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:07 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:07 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:07 | 87326a4f0a790cca | opened              |     354 |
| 06:55:07 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:07 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:07 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:07 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:07 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:07 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:07 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:55:07 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:55:07 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:55:07 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:07 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:07 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:08 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:08 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:08 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:09 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:09 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:09 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:16 | NULL             | pageLoaded          |    NULL |
| 06:55:16 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:55:16 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:55:16 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:55:16 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:55:16 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:55:16 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:55:17 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:55:17 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:17 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:55:17 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:55:17 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:55:17 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:55:17 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:55:17 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:17 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:55:17 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:55:17 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:55:17 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:17 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:55:17 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:55:17 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:55:17 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:17 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:55:17 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:17 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:17 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:17 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:17 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:17 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:17 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:17 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:17 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:55:17 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:17 | 82b2c3fa5e731743 | opened              |     364 |
| 06:55:17 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:17 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:55:17 | 87326a4f0a790cca | opened              |     354 |
| 06:55:17 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:17 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:17 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:55:17 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:17 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:17 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:17 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:55:17 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:55:17 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:55:17 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:17 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:55:17 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:17 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:17 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:17 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:17 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:17 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:55:17 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:18 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:18 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:18 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:18 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:18 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:18 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:18 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:18 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:19 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:19 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:19 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:19 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:26 | NULL             | pageLoaded          |    NULL |
| 06:55:26 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:55:26 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:55:26 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:55:26 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:55:26 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:55:26 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:55:26 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:55:26 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:55:26 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:55:26 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:55:26 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:55:26 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:55:26 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:55:27 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:55:27 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:27 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:55:27 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:55:27 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:55:27 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:55:27 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:55:27 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:27 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:55:27 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:55:27 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:55:27 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:27 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:55:27 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:55:27 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:27 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:55:27 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:27 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:27 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:27 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:27 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:27 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:27 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:27 | 82b2c3fa5e731743 | opened              |     364 |
| 06:55:27 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:27 | 87326a4f0a790cca | opened              |     354 |
| 06:55:27 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:27 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:27 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:27 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:27 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:27 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:55:27 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:55:27 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:27 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:27 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:27 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:28 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:28 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:28 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:28 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:28 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:28 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:28 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:28 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:28 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:28 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:28 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:28 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:28 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:28 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:28 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:28 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:28 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:28 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:28 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:28 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:28 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:28 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:29 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:29 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:29 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:29 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:29 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:32 | NULL             | pageLoaded          |    NULL |
| 06:55:32 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:55:32 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:32 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:55:32 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:55:32 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:55:32 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:55:32 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:55:32 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:32 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:55:32 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:55:32 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:55:32 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:32 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:55:32 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:55:32 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:55:32 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:55:32 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:32 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:55:32 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:55:32 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:55:32 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:55:32 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:32 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:32 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:32 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:32 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:32 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:32 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:55:32 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:55:32 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:32 | 82b2c3fa5e731743 | opened              |     364 |
| 06:55:32 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:32 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:55:32 | 87326a4f0a790cca | opened              |     354 |
| 06:55:32 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:32 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:55:32 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:32 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:55:32 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:32 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:32 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:32 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:55:32 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:55:32 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:55:32 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:32 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:55:32 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:32 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:32 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:55:32 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:33 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:33 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:33 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:33 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:33 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:33 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:33 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:37 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:37 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:37 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:37 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:37 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:37 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:44 | NULL             | pageLoaded          |    NULL |
| 06:55:44 | 1ce2b50ac521517d | dwelledButAbandoned |      16 |
| 06:55:44 | 23bce1a83e7db63f | dwelledButAbandoned |      16 |
| 06:55:44 | 27e9726d77b6de37 | dwelledButAbandoned |     161 |
| 06:55:44 | 45b5a0565faf5893 | dwelledButAbandoned |      49 |
| 06:55:44 | 4799349dd0eb718d | dwelledButAbandoned |      19 |
| 06:55:44 | 47eb4d848c12e726 | dwelledButAbandoned |       7 |
| 06:55:44 | 6ff56ac37e56a3c2 | dwelledButAbandoned |      26 |
| 06:55:44 | 7036d1e4228059c1 | dwelledButAbandoned |      40 |
| 06:55:44 | 87232487f5d44930 | dwelledButAbandoned |      32 |
| 06:55:44 | a3308bc2777cb448 | dwelledButAbandoned |      32 |
| 06:55:44 | a68557f49bb455f5 | dwelledButAbandoned |      16 |
| 06:55:44 | cf867e325c85581f | dwelledButAbandoned |      16 |
| 06:55:44 | d70c237fc5efc162 | dwelledButAbandoned |      13 |
| 06:55:44 | e14bbe5deb7e0839 | dwelledButAbandoned |       8 |
| 06:55:44 | e6f444fe1eb34f1e | dwelledButAbandoned |       8 |
| 06:55:45 | 002d56701451c9ef | dwelledButAbandoned |       8 |
| 06:55:45 | 00830068b8bc163c | dwelledButAbandoned |      74 |
| 06:55:45 | 0088a7c0061fbb55 | dwelledButAbandoned |      40 |
| 06:55:45 | 04b1a573cf73af17 | dwelledButAbandoned |      48 |
| 06:55:45 | 063eda3f61a47e54 | dwelledButAbandoned |      16 |
| 06:55:45 | 0ac4467b063ead9b | dwelledButAbandoned |      32 |
| 06:55:45 | 0b51102a4e702d49 | dwelledButAbandoned |      14 |
| 06:55:45 | 0de3506f847b0377 | dwelledButAbandoned |      50 |
| 06:55:45 | 18b65d3b9aa4c2b3 | dwelledButAbandoned |      16 |
| 06:55:45 | 1a91e31959dad313 | dwelledButAbandoned |       4 |
| 06:55:45 | 1af13046819520af | dwelledButAbandoned |      39 |
| 06:55:45 | 1afb48f06b222156 | dwelledButAbandoned |      56 |
| 06:55:45 | 2013829fb95286ee | dwelledButAbandoned |      50 |
| 06:55:45 | 347fbebe1de01a63 | dwelledButAbandoned |      82 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 34b423fb36b95ae4 | dwelledButAbandoned |      16 |
| 06:55:45 | 3e948c775944c1e9 | dwelledButAbandoned |      56 |
| 06:55:45 | 42278d37457ad912 | dwelledButAbandoned |      22 |
| 06:55:45 | 49451bbe21db74fe | dwelledButAbandoned |      57 |
| 06:55:45 | 5223d7735db28e35 | dwelledButAbandoned |     412 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 54e00651d94784cb | dwelledButAbandoned |      12 |
| 06:55:45 | 56cf63e8efb0938d | dwelledButAbandoned |     168 |
| 06:55:45 | 58903de4c2d6584f | dwelledButAbandoned |      10 |
| 06:55:45 | 5994b17f4254fe8c | dwelledButAbandoned |      39 |
| 06:55:45 | 5a650386acbfd8b1 | dwelledButAbandoned |     280 |
| 06:55:45 | 6a288c6f2925c00f | dwelledButAbandoned |      28 |
| 06:55:45 | 73bd3b864d62ea32 | opened              |     470 |
| 06:55:45 | 76a44eeb40221f43 | dwelledButAbandoned |      83 |
| 06:55:45 | 82b2c3fa5e731743 | opened              |     364 |
| 06:55:45 | 83d56e4ec0b7031c | dwelledButAbandoned |      13 |
| 06:55:45 | 867fb688281919a2 | dwelledButAbandoned |      22 |
| 06:55:45 | 87326a4f0a790cca | opened              |     354 |
| 06:55:45 | 88bfde52220a190f | dwelledButAbandoned |     145 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4c9f2922ab99a2d | dwelledButAbandoned |      17 |
| 06:55:45 | a4ce37a81a9f0b35 | opened              |     526 |
| 06:55:45 | a8c2c4f953c9ec65 | dwelledButAbandoned |      55 |
| 06:55:45 | ad18fd0756aad924 | dwelledButAbandoned |      16 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b4ad25d50a0a664b | dwelledButAbandoned |       3 |
| 06:55:45 | b77a9b3d8f87761f | dwelledButAbandoned |      56 |
| 06:55:45 | c046ed944e1fca55 | opened              |    1076 |
| 06:55:45 | c214a2a4e7fae4e5 | opened              |     361 |
| 06:55:45 | d83ea17fed4eac49 | dwelledButAbandoned |      66 |
| 06:55:45 | e0645760e62f222b | dwelledButAbandoned |      63 |
| 06:55:45 | e28b4ff200a67e4a | dwelledButAbandoned |      21 |
| 06:55:45 | e2d52f276518b35c | dwelledButAbandoned |     477 |
| 06:55:45 | e690e3c436b1efba | dwelledButAbandoned |      39 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e6c28bd64e14d04e | dwelledButAbandoned |      31 |
| 06:55:45 | e76a2f1cfedd1905 | dwelledButAbandoned |     129 |
| 06:55:45 | f2c9f32497285fb6 | dwelledButAbandoned |       6 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 10bb1492d4c70e62 | dwelledButAbandoned |      12 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:46 | 21b5fa2244e69303 | dwelledButAbandoned |      16 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 354fe767aa14f968 | dwelledButAbandoned |      20 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:47 | 3ddea088dc3154e1 | dwelledButAbandoned |      14 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | 81d8dc6ac33da58a | dwelledButAbandoned |      25 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:49 | f89fc36618ed1dff | dwelledButAbandoned |      34 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:50 | 24feebd3d7918d8c | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:54 | 4ea2f93c6d8607a0 | dwelledButAbandoned |      32 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
| 06:55:56 | 5d6350a679c2b398 | dwelledButAbandoned |       7 |
+----------+------------------+---------------------+---------+
511 rows in set (4.22 sec)

Change 353035 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/Popups@master] eventLogging: Discard events with duplicate tokens

https://gerrit.wikimedia.org/r/353035

gerritbot added a project: Patch-For-Review.May 10 2017, 10:36 AM

Change 353038 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/Popups@wmf/1.30.0-wmf.1] eventLogging: Discard events with duplicate tokens

https://gerrit.wikimedia.org/r/353038

Change 353035 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] eventLogging: Discard events with duplicate tokens

https://gerrit.wikimedia.org/r/353035

Change 353044 had a related patch set uploaded (by Jhernandez; owner: Jhernandez):
[mediawiki/extensions/Popups@master] Clear interaction after an event for it is logged in EL

https://gerrit.wikimedia.org/r/353044

Change 353058 had a related patch set uploaded (by Phuedx; owner: Phuedx):
[mediawiki/extensions/Popups@wmf/1.29.0-wmf.21] eventLogging: Discard events with duplicate tokens

https://gerrit.wikimedia.org/r/353058

Change 353038 merged by jenkins-bot:
[mediawiki/extensions/Popups@wmf/1.30.0-wmf.1] eventLogging: Discard events with duplicate tokens

https://gerrit.wikimedia.org/r/353038

Change 353058 merged by jenkins-bot:
[mediawiki/extensions/Popups@wmf/1.29.0-wmf.21] eventLogging: Discard events with duplicate tokens

https://gerrit.wikimedia.org/r/353058

Mentioned in SAL (#wikimedia-operations) [2017-05-10T13:19:21Z] <hashar@tin> Synchronized php-1.30.0-wmf.1/extensions/Popups: eventLogging: Discard events with duplicate tokens - T161769 T163198 (duration: 01m 08s)

Mentioned in SAL (#wikimedia-operations) [2017-05-10T13:24:21Z] <hashar@tin> Synchronized php-1.29.0-wmf.21/extensions/Popups: eventLogging: Discard events with duplicate tokens - T161769 T163198 (duration: 00m 43s)

^ We're now counting and discarding enqueued events with duplicate tokens before they can be logged. If this gets rid of the duplication on the server side, then we can be confident that the problem is solely due to one or more bugs in PP.

One thing @Jhernandez and I spotted while trying to make sense of the EventLogging Data Integrity row of the PP dashboard is that there were two abnormally high duplicate token counts at 14:36:00 and 14:37:00 (286 and 839, respectively). These were recorded within 30 minutes of kafka200{2,3} being restarted for kernel upgrades but that might be a red herring.

phuedx moved this task from Blocked on Others to Doing on the Readers-Web-Kanbanana-Board-Old board.May 11 2017, 5:07 PM

• Tbayer mentioned this in T165461: Various sanity checks for data from the rewritten Popups instrumentation.May 16 2017, 9:04 AM

Change 353044 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Clear interaction after an event for it is logged in EL

https://gerrit.wikimedia.org/r/353044

Looks like no patches to review just yet.

@Tbayer: I wasn't sure on which task to put the ReadingDepth duplicates query that I mentioned during yesterday's "Page Previews sync" meeting, so I'll put them here with a view to referencing them from re-opened or new tasks later:

SELECT dupesPerDay.ymd, nDupes, n, ROUND(100*nDupes / n, 2) AS '%' FROM
  (
    SELECT ymd, SUM(nDupes) AS nDupes FROM
    (
      SELECT LEFT(timestamp, 8) AS ymd, COUNT(*) AS nDupes 
      FROM log.ReadingDepth_16325045
      WHERE timestamp LIKE '201704%'
      AND event_pageToken IS NOT NULL
      AND event_action = 'pageLoaded'
      GROUP BY ymd, event_pageToken
      HAVING nDupes > 1
    ) AS dupesPerDayInt
    GROUP BY ymd
  ) AS dupesPerDay
  JOIN
  (
    SELECT LEFT(timestamp, 8) AS ymd, COUNT(*) AS n 
    FROM log.ReadingDepth_16325045
    WHERE timestamp LIKE '201704%'
    AND event_pageToken IS NOT NULL
    AND event_action = 'pageLoaded'
    GROUP BY ymd
  ) AS nPerDay
  ON dupesPerDay.ymd = nPerDay.ymd
ORDER BY ymd;

+----------+--------+--------+------+
| ymd      | nDupes | n      | %    |
+----------+--------+--------+------+
| 20170401 |     34 | 105733 | 0.03 |
| 20170402 |     28 | 114609 | 0.02 |
| 20170403 |     40 | 125470 | 0.03 |
| 20170404 |     32 | 136079 | 0.02 |
| 20170405 |     79 | 131136 | 0.06 |
| 20170406 |    177 | 130001 | 0.14 |
| 20170407 |    242 | 128073 | 0.19 |
| 20170408 |    126 | 114574 | 0.11 |
| 20170409 |    124 | 122567 | 0.10 |
| 20170410 |    157 | 134210 | 0.12 |
| 20170411 |    196 | 134748 | 0.15 |
| 20170412 |    231 | 123469 | 0.19 |
| 20170413 |     96 | 123061 | 0.08 |
| 20170414 |     14 | 119508 | 0.01 |
| 20170415 |     18 | 115602 | 0.02 |
| 20170416 |     10 | 119242 | 0.01 |
| 20170417 |    135 | 130094 | 0.10 |
| 20170418 |    288 | 144508 | 0.20 |
| 20170419 |    844 | 145991 | 0.58 |
| 20170420 |    916 | 138555 | 0.66 |
| 20170421 |    250 | 128556 | 0.19 |
| 20170422 |    226 | 112482 | 0.20 |
| 20170423 |    191 | 121552 | 0.16 |
| 20170424 |    218 | 131852 | 0.17 |
| 20170425 |    287 | 137491 | 0.21 |
| 20170426 |    240 | 135985 | 0.18 |
| 20170427 |    102 | 132841 | 0.08 |
| 20170428 |     43 | 129900 | 0.03 |
| 20170429 |     62 | 106822 | 0.06 |
| 20170430 |    114 | 121755 | 0.09 |
+----------+--------+--------+------+

# And with timestamp LIKE '201705%'...

+----------+--------+--------+------+
| ymd      | nDupes | n      | %    |
+----------+--------+--------+------+
| 20170501 |    624 | 145653 | 0.43 |
| 20170502 |    457 | 142432 | 0.32 |
| 20170503 |    882 | 143058 | 0.62 |
| 20170504 |    519 | 136959 | 0.38 |
| 20170505 |    744 | 122362 | 0.61 |
| 20170506 |   1402 | 106629 | 1.31 |
| 20170507 |   1109 | 116945 | 0.95 |
| 20170508 |   1086 | 132431 | 0.82 |
| 20170509 |    889 | 135161 | 0.66 |
| 20170510 |    625 | 124362 | 0.50 |
| 20170511 |     40 | 114810 | 0.03 |
| 20170512 |     24 | 117978 | 0.02 |
| 20170513 |     28 | 110706 | 0.03 |
| 20170514 |     62 | 120441 | 0.05 |
| 20170515 |     79 | 142848 | 0.06 |
| 20170516 |    154 | 141016 | 0.11 |
+----------+--------+--------+------+

Another example like T161769#3250284 : All events for the pageview that yesterday contained the link interaction (7d95d1f659fc00b7) with the most events (61). What's interesting here is that duplicates were resent (or re-recorded) for the same events at the same times.

SELECT CONCAT( SUBSTRING(timestamp, 9, 2), ':', SUBSTRING(timestamp, 11, 2), ':', RIGHT(timestamp,2)) AS hhmmss,
event_linkInteractionToken AS lIaToken, 
event_action AS action,
event_totalInteractionTime AS tIaTime
FROM log.Popups_16364296 
WHERE event_PageToken ='7db04abaed894305'
AND LEFT(timestamp, 8) = '20170518'
ORDER BY hhmmss, lIaToken, action, tIaTime;

+----------+------------------+---------------------+---------+
| hhmmss   | lIaToken         | action              | tIaTime |
+----------+------------------+---------------------+---------+
| 16:19:23 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:23 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:32 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:32 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:36 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:36 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:38 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:38 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:42 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:42 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:44 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:44 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:48 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:48 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:51 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:51 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:54 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:54 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:19:57 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:19:57 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:00 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:00 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:03 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:03 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:06 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:06 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:09 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:09 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:12 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:12 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:15 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:15 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:17 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:17 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:42 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:42 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:45 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:45 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:48 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:48 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:51 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:51 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:53 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:53 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:56 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:20:56 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:58 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:20:59 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:01 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:01 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:03 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:03 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:05 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:08 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:08 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:08 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:11 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:11 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:11 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:19 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:19 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:19 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:22 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:22 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:22 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:24 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:24 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:24 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:27 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:27 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:27 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:30 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:30 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:30 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:32 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:32 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:32 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:35 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:35 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:35 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:37 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:37 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:37 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:39 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:39 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:39 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:42 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:42 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:42 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:44 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:44 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:44 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:47 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:47 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:47 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:51 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:51 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:51 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:54 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:54 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:54 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:21:57 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:21:57 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:21:57 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:01 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:01 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:01 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:04 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:04 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:04 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:07 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:07 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:07 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:10 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:10 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:10 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:22 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:22 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:22 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:39 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:39 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:39 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:42 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:42 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:42 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:46 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:46 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:46 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:49 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:49 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:49 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:52 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:52 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:52 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:53 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:53 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:54 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:22:57 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:22:57 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:22:57 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:01 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:23:01 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:23:01 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:03 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:23:03 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:23:03 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:04 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:07 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:23:07 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:07 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:23:07 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:11 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:23:11 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:11 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:11 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:23:11 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:14 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:23:14 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:14 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:14 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:14 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:23:14 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:17 | 0d634d9bd7339e9e | dwelledButAbandoned |      32 |
| 16:23:17 | 321c842832486b59 | dwelledButAbandoned |     103 |
| 16:23:17 | 5cd30a4c3e04a0c6 | dwelledButAbandoned |      23 |
| 16:23:17 | 6b7a7eb9b8c9f8ef | dwelledButAbandoned |     128 |
| 16:23:17 | 7d95d1f659fc00b7 | dwelledButAbandoned |      63 |
| 16:23:17 | c8f5877683051362 | dwelledButAbandoned |      41 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:18 | 093ac16f0f7e5473 | dwelledButAbandoned |      12 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
| 16:23:24 | 1e8498b48f755ce4 | dwelledButAbandoned |      21 |
+----------+------------------+---------------------+---------+
488 rows in set (2.95 sec)

ovasileva removed a project: Readers-Web-Kanbanana-Board-Old.May 25 2017, 5:13 PM

Jdlrobson moved this task from 2016-17 Q4 to Epics/Goals on the Web-Team-Backlog board.May 31 2017, 4:53 PM

Jdlrobson moved this task from Epics/Goals to Product Owner Backlog on the Web-Team-Backlog board.Jun 5 2017, 5:13 PM

• Tbayer added a parent task: T158172: Analyze results of stage 0 page previews test.Jun 7 2017, 10:58 PM

phuedx mentioned this in T167391: [Spike] Isolate the source of Popups event duplication.Jun 8 2017, 9:44 AM

ovasileva moved this task from Product Owner Backlog to Needs Prioritization on the Web-Team-Backlog board.Jun 20 2017, 1:15 PM

• Tbayer mentioned this in T169581: Histogram for the number of duplicated events per browser session.Jul 3 2017, 8:39 PM

• Tbayer mentioned this in T169582: Examine several sessions which contained duplicate events.Jul 3 2017, 8:50 PM

I'm moving this to the kanban sign off column as it's being referenced and its useful not to hide it away.

Jdlrobson moved this task from Doing to Ready for Signoff on the Readers-Web-Kanbanana-Board-Old board.Jul 6 2017, 7:28 PM

Jdlrobson moved this task from Needs Prioritization to Upcoming on the Web-Team-Backlog board.

Jdlrobson moved this task from Upcoming to 2016-17 Q4 on the Web-Team-Backlog board.

phuedx mentioned this in T170018: Duplicate events sent in Firefox after back button press.Jul 12 2017, 7:16 AM

ovasileva moved this task from 2016-17 Q4 to 2017-18 Q1 on the Web-Team-Backlog board.Jul 13 2017, 5:40 PM

phuedx mentioned this in T140485: Schema:Popups logs "dwelledButAbandoned" and link open events at the same time.Jul 16 2017, 7:51 AM

MBinder_WMF added a project: Reading-analysis.Jul 17 2017, 5:09 PM

closing this. If we notice something as a part of T158172, we can open a separate bug. @Tbayer - feel free to re-open if you think we're missing something.

	F7659717: Screen Shot 2017-04-20 at 07.56.33.png
	Apr 20 2017, 7:03 AM

	F7528400: duplicate link interaction events in the Popups schema ..2017-04-13.png
	Apr 13 2017, 3:04 PM

	F7400895: duplicate link interaction events in the Popups schema.png
	Apr 9 2017, 6:16 AM

Schema:Popups sends extraneous link interaction events in control conditionClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Schema:Popups sends extraneous link interaction events in control condition
Closed, ResolvedPublic
Actions

Related Objects
Search...