Regarding the proposal "If navigator.sendBeacon is undefined do not enable Schema:PageIssues" :

That doesn't solve the problem - actually, it would exacerbate it, by extending the problem "our instrumentation sometimes sends data when it should, but sometimes not, and we don't know why" from the ReadingDepth schema to the PageIssues schema.

Change 460441 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Restrict PageIssues schema logging to browsers that support sendBeacon

https://gerrit.wikimedia.org/r/460441

The task as written will guarantee PageIssues and ReadingDepth behave consistently with one another.

The question of why sendBeacon support differs for certain browsers from published information is a separate and curious question that will need further analysis +research (feel free to open a task to investigate how that can happen)

Following up on T204143#4578937, here is a closer look at which Safari 11.x clients send ReadingDepth events in production. Both 11.1.1 and 11.1.2 occur. So we can discard the hypothesis that the discrepancy is just due to sendBeacon having been added inbetween these two releases. Also, there was one event with @Jdlrobson's exact 11.1.1 user agent during that timespan (not included in the list below, which is limited to UAs with >100 events yesterday).

Data via

SELECT user_agent, COUNT(*) AS events 
FROM wmf.webrequest WHERE month = 9 AND day = 12 
AND  user_agent_map['browser_major'] = '11'
AND  user_agent_map['browser_family'] = 'Safari'
AND uri_query  LIKE '%ReadingDepth%' 
GROUP BY user_agent
ORDER BY events DESC LIMIT 100;

Also, there was one event with @Jdlrobson's exact 11.1.1 user agent during that timespan (not included in the list below, which is limited to UAs with >100 events yesterday).

Was that me? I was fiddling with ReadingDepth yesterday and overriding the check against production.

In T204143#4582482, @Jdlrobson wrote:

Also, there was one event with @Jdlrobson's exact 11.1.1 user agent during that timespan (not included in the list below, which is limited to UAs with >100 events yesterday).

Was that me? I was fiddling with ReadingDepth yesterday and overriding the check against production.

No, it was an IP from Germany on the German Wikipedia (can send you the details offline in case they are of interest).

In T204143#4582202, @Jdlrobson wrote:

The task as written will guarantee PageIssues and ReadingDepth behave consistently with one another.

The question of why sendBeacon support differs for certain browsers from published information is a separate and curious question that will need further analysis +research (feel free to open a task to investigate how that can happen)

OK, to be a bit clearer. This is a pointless and counterproductive change, and I'm not sure why the idea is still being pursued (cf. T204143#4582135 + T204143#4582202 ). It seems to be based on some mistaken assumptions about the planned data analysis, which does not require the two schemas to "behave consistently with one another" in this sense. It does however require that the instrumentation won't unpredictably fail to send data for browsers where we expect it to work (e.g. based on sendBeacon support per browser vendor's public documentation).

In T204143#4579286, @Ryasmeen wrote:

This issue is also happening for Chrome on Android.

For the record: @Jdlrobson is pointing out that this test was done using Chrome for Android 35 (released in 2014), whereas sendBeacon support was only added in Chrome for Android 42. So that means that, fortunately, Safari remains the only browser at this point where we have encountered these inconsistencies so far.

In T204143#4579286, @Ryasmeen wrote:

This issue is also happening for Chrome on Android.

@Ryasmeen - what version of Chrome did you see this on?

@Ryasmeen - what version of Chrome did you see this on?

Rummana told me this was Chrome 35 on Android, which doesn't support sendBeacon. This is why I've not included in the bug report description. This is working as expected.

See https://phabricator.wikimedia.org/T204143#4585115

In T204143#4589495, @Jdlrobson wrote:

@Ryasmeen - what version of Chrome did you see this on?

Rummana told me this was Chrome 35 on Android, which doesn't support sendBeacon. This is why I've not included in the bug report description. This is working as expected.

See also T204143#4585115 - I assume we are all talking about the same thing here?

Sorry, I didn't phrase this correctly. I meant to ask if we did not see this behavior on later versions of Chrome (sendBeacon was working as expected for Chrome versions known to support it)

Spoke with @Ryasmeen shortly:

• Originally reported on Safari 11.1.1
• Reproduced in Chrome 35
• We still need to test a more recent version of Chrome

Change 460441 abandoned by Jdlrobson:
Restrict PageIssues schema logging to browsers that support sendBeacon

Reason:
per discussion on task this is not necessary

https://gerrit.wikimedia.org/r/460441

In standup we talked about this bug and agreed to exclude Safari 11.1.1 user agents from analysis.
Is there anything else left to do that's blocking the A/B test or can we close this task?

(If we want to spend time investigating the Safari 11.1.1 issue some more I'd suggest a new task outlining exactly that)

In T204143#4590488, @Jdlrobson wrote:

In standup we talked about this bug and agreed to exclude Safari 11.1.1 user agents from analysis.
Is there anything else left to do that's blocking the A/B test or can we close this task?

(If we want to spend time investigating the Safari 11.1.1 issue some more I'd suggest a new task outlining exactly that)

We want an extra confirmation that this is not an issue in later versions of Chrome. @Ryasmeen will be testing this today/tomorrow.

In T204143#4590612, @ovasileva wrote:

In T204143#4590488, @Jdlrobson wrote:

In standup we talked about this bug and agreed to exclude Safari 11.1.1 user agents from analysis.
Is there anything else left to do that's blocking the A/B test or can we close this task?

(If we want to spend time investigating the Safari 11.1.1 issue some more I'd suggest a new task outlining exactly that)

We want an extra confirmation that this is not an issue in later versions of Chrome. @Ryasmeen will be testing this today/tomorrow.

@ovasileva: Checked on the latest version of Chrome on Android (69.0.3497.100). This issue is not occurring there.

In T204143#4598335, @ovasileva wrote:

Perfect. Thanks @Ryasmeen! In that case, let's go ahead and close this task and confirm that we will be avoiding Safari in our analysis @Tbayer

I have added a note to the schema talk page: https://meta.wikimedia.org/wiki/Schema_talk:ReadingDepth#Likely_broken_on_Safari
(CC @Groceryheist as this will impact his upcoming work as well)

I didn't realise we were excluding all of Safari. That seems a bit extreme imo given we have seen this issue only on 11.1.1 on desktop and we could just exclude that user agent.

I hope this doesn't mean we are excluding iPhone/iPad.
If so I recommend more testing on different Safari versions to increase our confidence. We have no reason right now to believe that all safari's are bad based on 2 desktop browsers.

In T204143#4601864, @Jdlrobson wrote:

I didn't realise we were excluding all of Safari. That seems a bit extreme imo given we have seen this issue only on 11.1.1 on desktop and we could just exclude that user agent.

I hope this doesn't mean we are excluding iPhone/iPad.
If so I recommend more testing on different Safari versions to increase our confidence. We have no reason right now to believe that all safari's are bad based on 2 desktop browsers.

Remind me, did we do QA for this schema on Mobile Safari? If you and/or @Ryasmeen saw valid events on that browser, I would agree that it's reasonable to assume for now that we can use its data.

I agree it would be good to do further testing to see whether we can narrow down the issue to particular versions, but considering that (per T204143#4578937 ) almost all desktop Safari events come from 11.1 currently, it wouldn't make much of a difference right now.

(Since T153207, raw user agents are no longer available in the EL tables, so we can't efficiently narrow queries down to subversions like 11.1.1.)

Remind me, did we do QA for this schema on Mobile Safari? If you and/or @Ryasmeen saw valid events on that browser, I would agree that it's reasonable to assume for now that we can use its data.

iOS Safari sendBeacon support was only added in 11.4 (Mar 2018). Thus older versions of 11 will not have it. Are we seeing events from 11.4 Mobile Safari?

In T204143#4630961, @Jdlrobson wrote:

Remind me, did we do QA for this schema on Mobile Safari? If you and/or @Ryasmeen saw valid events on that browser, I would agree that it's reasonable to assume for now that we can use its data.

iOS Safari sendBeacon support was only added in 11.4 (Mar 2018). Thus older versions of 11 will not have it. Are we seeing events from 11.4 Mobile Safari?

Yes, see above (T204143#4578937) - but also from (clients that are logged as) Mobile Safari 11.0.

Did we do QA for this schema on Mobile Safari?

Now that the A/B test has launched, we have an additional method to check this: At the beginning of a pageview, the page issues instrumentation (from T191532) is supposed to send a pageLoaded event to both the PageIssues and ReadingDepth schemas. Because ReadingDepth depends on (in particular) support for sendBeacon and the Page Visibility API, we expect the ReadingDepth event to be missing for some browsers or browser versions, in particular Safari as discussed earlier. However, it looks like many other browsers apart from Safari are missing ReadingDepth events too ("only_pi" >> 0% in the table below). In particular, Chrome Mobile iOS, Android (stock browser) and (desktop) Chrome seem worth a closer look.

For a ReadingDepth event to fire it it must be true that:

• JavaScript is enabled
• There are no JavaScript client side errors at runtime (we'll know more about this when T205582 is live in production)
• the user is in the sample group (wgWMEReadingDepthSamplingRate) OR the user is in the page issues A/B test
  • There is NavigationTiming support (perf.timing && perf.timing.navigationStart)
  • sendBeacon support
  • navigator.doNotTrack is not set
  • mw.config.get( 'wgEventLoggingBaseUri' ) must be set
• There are no Event errors relating to the schema

For a PageIssues event to fire, it's a little less complicated. The following needs to be true:

• JavaScript is enabled
• There are no JavaScript client side errors at runtime (we'll know more about this when T205582 is live in production)
• mw.config.get( 'wgEventLoggingBaseUri' ) must be set
• The user falls within wgMinervaABSamplingRate
• The page in question has issues.
• navigator.doNotTrack is not set
• There are no Event errors relating to the schema

In cases where ReadingDepth is missing, but a PageIssues event exists, we can expect one of the following to be true:

• • There is NavigationTiming support (perf.timing && perf.timing.navigationStart)
  • sendBeacon support
• There are no JavaScript client side errors at runtime (we'll know more about this when T205582 is live in production)
• There are no Event errors relating to the schema

With regards to the first 2, I'd need more detailed information on the versions data is missing for Chrome Mobile iOS, Android (stock browser) and (desktop) Chrome. Chrome iOS is very different from Chrome for Android (one uses webkit and the other blink for rendering). For desktop, at least Chrome 39 is needed and for Android stock browser I still don't really understand why this browser is still around and I suspect it's in maintenance mode - I wouldn't be surprised if it doesn't support sendBeacon or performance.

3 actionables I can pull out of this:

• I'd suggest looking at the EventLogging errors
• Let's see what T205582 tells us about client errors
• Let's get information on browser versions for the set of events fired for page issues but not reading depth for further investigation. browser family is not enough here.

I'd suggest looking at the EventLogging errors

Quick follow up on this:
I'm seeing at least one issue in kafkacat relating to sectionNumbers being set as null which relates to this.
It occurs on a page which has no issues and I cannot replicate.

ssh stat1004.eqiad.wmnet
 -C -b kafka-jumbo1001.eqiad.wmnet -t eventlogging_EventError | grep PageIssues

I see at least one event with validation message "None is not of type 'integer'".

{"event": {"code": "validation", "message": "None is not of type 'integer'", "rawEvent": "

Decoding that with decodeURIComponent

"{"event": {"code": "validation", "message": "None is not of type 'integer'", "rawEvent": "?{"event":{"pageTitle":"شهرآورد تهران","namespaceId":0,"pageIdSource":3259010,"issuesVersion":"new2018","issuesSeverity":["DEFAULT"],"sectionNumbers":[null],"isAnon":true,"editCountBucket":"0 
....
webHost":"fa.m.wikipedia.org","wiki":"fawiki"};	cp3030.esams.wmn

The issue is sectionNumbers being set as null:

sectionNumbers":[null],

Cannot replicate myself on this page, but looks like a legit bug in the PageIssues logic which would explain the discrepancy we are seeing.
Am seeing this event quite regularly across various wikis.

User agents:

• Mozilla/5.0 (Linux; Android 9; PH-1 Build/PPR1.181005.034) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.123 Mobile Safari/537.36
• Mozilla/5.0 (Linux; Android 7.0; SM-G610F Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.137 Mobile Safari/537.36\
• Mozilla/5.0 (Linux; Android 8.0.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.91 Mobile Safari/537.36\
• Mozilla/5.0 (Linux; Android 7.1.1; TA-1032 Build/NMF26O) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Mobile Safari/537.36\

We'd need some real devices to isolate and understand this problem. I cannot replicate on any of the pages it happens, so likely to be browser specific.

In T204143#4650687, @Jdlrobson wrote:

[...]

With regards to the first 2, I'd need more detailed information on the versions data is missing for Chrome Mobile iOS, Android (stock browser) and (desktop) Chrome. Chrome iOS is very different from Chrome for Android (one uses webkit and the other blink for rendering). For desktop, at least Chrome 39 is needed and for Android stock browser I still don't really understand why this browser is still around and I suspect it's in maintenance mode - I wouldn't be surprised if it doesn't support sendBeacon or performance.

Here is the table from the task description broken down by browser version (and limited to rows with >= 100 views).

For desktop Chrome we indeed see "only_pi" drop to 0-2% with version 39. For Chrome Mobile iOS and Android, it looks much less clear.

Data via

SET hive.mapred.mode=nonstrict;
SELECT 
browser, major, minor,
ROUND(100*SUM(IF((pipageToken IS NOT NULL) AND (rdpageToken IS NOT NULL),1,0))/SUM(1),2) AS both, 
ROUND(100*SUM(IF((pipageToken IS NOT NULL) AND (rdpageToken IS NULL),1,0))/SUM(1),2) AS only_pi, 
ROUND(100*SUM(IF((pipageToken IS NULL) AND (rdpageToken IS NOT NULL),1,0))/SUM(1),2) AS only_rd, 
SUM(1) AS all_pageloads
FROM (
  SELECT 
  IF(pi.pageToken IS NOT NULL, pi.browser, rd.browser) AS browser,
  IF(pi.pageToken IS NOT NULL, pi.major, rd.major) AS major,
  IF(pi.pageToken IS NOT NULL, pi.minor, rd.minor) AS minor,
  pi.pageToken AS pipageToken, rd.pageToken AS rdpageToken
  FROM (
    SELECT useragent.browser_family AS browser,
    useragent.browser_major AS major,
    useragent.browser_minor AS minor,
    event.pageToken AS pageToken
    FROM event.pageissues 
    WHERE year = 2018 AND month = 10 AND day <=7
    AND event.action = 'pageLoaded') AS pi
  FULL OUTER JOIN (
    SELECT useragent.browser_family AS browser,
    useragent.browser_major AS major,
    useragent.browser_minor AS minor,
    event.pageToken AS pageToken
    FROM event.readingdepth
    WHERE year = 2018 AND month = 10 AND day <=7
    AND event.action = 'pageLoaded'
    AND ( event.page_issues_a_sample OR event.page_issues_b_sample )) AS rd
  ON pi.pageToken = rd.PageToken) AS alltokens
WHERE browser IN ('Chrome Mobile iOS', 'Chrome', 'Android')
GROUP BY browser, major, minor
HAVING all_pageloads >= 100
ORDER BY browser, major, minor;

Insprired by a suggestion of @Jdlrobson, here is a version of the above query by iOS version, showing a clear change at iOS 11.3, but also some oddities at earlier versions like 9.1:

SET hive.mapred.mode=nonstrict;
SELECT 
os, major, minor,
ROUND(100*SUM(IF((pipageToken IS NOT NULL) AND (rdpageToken IS NOT NULL),1,0))/SUM(1),2) AS both, 
ROUND(100*SUM(IF((pipageToken IS NOT NULL) AND (rdpageToken IS NULL),1,0))/SUM(1),2) AS only_pi, 
ROUND(100*SUM(IF((pipageToken IS NULL) AND (rdpageToken IS NOT NULL),1,0))/SUM(1),2) AS only_rd, 
SUM(1) AS all_pageloads
FROM (
  SELECT 
  IF(pi.pageToken IS NOT NULL, pi.os, rd.os) AS os,
  IF(pi.pageToken IS NOT NULL, pi.major, rd.major) AS major,
  IF(pi.pageToken IS NOT NULL, pi.minor, rd.minor) AS minor,
  pi.pageToken AS pipageToken, rd.pageToken AS rdpageToken
  FROM (
    SELECT useragent.os_family AS os,
    useragent.os_major AS major,
    useragent.os_minor AS minor,
    event.pageToken AS pageToken
    FROM event.pageissues 
    WHERE year = 2018 AND month = 10 AND day <=7
    AND event.action = 'pageLoaded') AS pi
  FULL OUTER JOIN (
    SELECT useragent.os_family AS os,
    useragent.os_major AS major,
    useragent.os_minor AS minor,
    event.pageToken AS pageToken
    FROM event.readingdepth
    WHERE year = 2018 AND month = 10 AND day <=7
    AND event.action = 'pageLoaded'
    AND ( event.page_issues_a_sample OR event.page_issues_b_sample )) AS rd
  ON pi.pageToken = rd.PageToken) AS alltokens
WHERE os = 'iOS'
GROUP BY os, major, minor
HAVING all_pageloads >= 100
ORDER BY os, INT(major), INT(minor);

I took a deep dive into this data today.
I compiled a table, cross checking browser versions with browser capabilities:
https://www.mediawiki.org/wiki/User:Jdlrobson/Page_issues_analysis

In general, the browser capabilities matched what we're seeing.

• if the majority of events were 100% only page issues (or close to 100%) it was consistent with the lack of a support for ReadingDepth (sendBeacon AND NavigationTiming support)
• if the majority of events were 100% both events (or close to 100%) that was consistent with support for ReadingDepth and PageIssues.

Where we were only seeing ReadingDepth events when we expected both, the margin of error was generally small (<= 10%).
The Android browsers proved the most problematic with discrepancies from 10-50%!

The most peculiar cases were:

• Seeing ReadingDepth events where ReadingDepth should be impossible.
  • When this happened it was generally a small fraction of our data. Where it was more problematic:
  • Chrome <= 38 on Android 4-7 (Note that sendBeacon was introduced in Chrome 38 so how sendBeacon is being used outside these browsers is not 100% clear)
  • iOS Chrome prior to 11.3 (NavigationTiming was disabled in iOS 8.1 and it's unclear when it got re-added and supported in Chrome, but our data seems to indicate 11.3 )
  • the native Android browser.
• Seeing events where JS is supposed to be disabled
  • Limited only to native Android browser.

From this analysis, I'd strongly recommend ignoring ReadingDepth data coming from Android native browser; iOS Chrome prior to 11.3 and Chrome <=38.

I have no exact answer to why we are dropping ReadingDepth events in cases where we should be sending them, but the following theories may provide answers. Given, the ReadingDepth and PageIssues events are sent at different times in the code. There are a variety of factors that could lead to only one being sent. These include

• client's web connection speed/stability
• client side error occurring in either page issues or reading depth
• event could not be decoded

One other thing that's worth pointing out - ReadingDepth will not run if navigationStart is 0. I'm not sure if that is ever true (per spec it should always be non-zero) but would also account for cases where ReadingDepth is not being sent.

None of this accounts for how ReadingDepth events can be sent without sendBeacon support.

To add to this analysis
@Nuria had this to say today:

jdlrobson: do not trust user agents 100% "android 2" could be a who-knows-bot with user agent "android 2" this happens everyday
4:42 PM jdlrobson: or also, could be a misslabeled UA, that is, parser thinks is Android 2 but it is really something else
4:42 PM jdlrobson: this does not happen a lot but it does happens
4:43 PM jdlrobson: i just run some numbers yesterday and by my early estimates 5% of our traffic labeled as "user" is really bots
4:44 PM jdlrobson: so i would not expect 100% consistancy, bots have "made up" UAs

I apologise for the non-sequitur but I mentioned that I'd follow up with the latest ReadingDepth and PageIssues server-side error rates so that the conversation was all happening in one place.

In T204143#4650715, @Jdlrobson wrote:

I'm seeing at least one issue in kafkacat relating to sectionNumbers being set as null which relates to this.

For 2018/10/09, the number of erroneous events received by the server for the PageIssues and ReadingDepth schemas are as follows:

The maximum is calculated assuming that all events that are categorised as "unknown" were actually events of that schema.

Regardless, nice investigation, y'all 💪

select
    count(*) as n
from
    event.readingdepth
where
    year = 2018 and
    month = 10 and
    day = 9
;

+-----------+
|     n     |
+-----------+
| 72358845  |
+-----------+

select
    count(*) as n
from
    event.pageissues
where
    year = 2018 and
    month = 10 and
    day = 9
;

+-----------+
|     n     |
+-----------+
| 12342261  |
+-----------+

select
    event.schema as schema,
    count(*) as n
from
    event.eventerror
where
    year = 2018 and
    month = 10 and
    day = 9 and

    event.schema in ("ReadingDepth", "PageIssues", "unknown")
group by
    event.schema
;

+---------------+-------+
|    schema     |   n   |
+---------------+-------+
| PageIssues    | 1395  |
| ReadingDepth  | 818   |
| unknown       | 2847  |
+---------------+-------+

Per T204143#4652491.

In T204143#4653502, @Jdlrobson wrote:

I took a deep dive into this data today.
I compiled a table, cross checking browser versions with browser capabilities:
https://www.mediawiki.org/wiki/User:Jdlrobson/Page_issues_analysis

[...]
Thanks again for the analysis and the recommendations!

From this analysis, I'd strongly recommend ignoring ReadingDepth data coming from Android native browser; iOS Chrome prior to 11.3 and Chrome <=38.

I guess that this was meant to read "iOS prior to 11.3", correct? (cf. above)

In T204143#4653514, @Jdlrobson wrote:

To add to this analysis
@Nuria had this to say today:

jdlrobson: do not trust user agents 100% "android 2" could be a who-knows-bot with user agent "android 2" this happens everyday
4:42 PM jdlrobson: or also, could be a misslabeled UA, that is, parser thinks is Android 2 but it is really something else
4:42 PM jdlrobson: this does not happen a lot but it does happens
4:43 PM jdlrobson: i just run some numbers yesterday and by my early estimates 5% of our traffic labeled as "user" is really bots
4:44 PM jdlrobson: so i would not expect 100% consistancy, bots have "made up" UAs

Well yes, we don't expect perfect accuracy with this kind of thing, which is why the task description already said " >> 0%" instead of "!= 0%". (BTW, there are also non-bot clients with forged user agents.)

Regarding Android 2 specifically, note that only a very small number of events were classified as coming from that OS version, see T204143#4650771. (We could double-check the full UA in the webrequest data to see if it's indeed a bug in ua-parser, but that doesn't seem worthwhile right now.)

Regarding undetected bots in general, I'm looking forward to the detection improvements planned for this fiscal year. FWIW, having noted the obvious bot UA "Baiduspider-render" in the data posted in the task description, I had also checked for *detected* bots earlier in the pageissues data, via useragent.is_bot, and their ratio was negligible.

Apropos, @Jdlrobson: The big browser+OS table at https://www.mediawiki.org/wiki/User:Jdlrobson/Page_issues_analysis#Results is great. But I got a bit confused about the "NO" in the "Matches expectations" column for many rows. E.g.

• Chrome 69 on Android 4.4: "Expected: both" seems to match the data (98.34% both)
• Chrome 36 on Android 4.4: "Expected: only_pi": seems to match the data (99.95% only_pi)

Is this because only exact 100.0% matches were marked as "YES" in "Matches expectations"?

But I got a bit confused about the "NO" in the "Matches expectations" column for many rows. E.g.

I was looking for 100% matches, yes. I've clarified the table with a "MOSTLY" value.

Jon and I talked about this some days ago, but some of what we determined isn't reflected here yet. a couple of quick thoughts and some additional analysis follows to hopefully move this forward:

Mobile Safari added support for Navigation Timing in iOS 9.0, not 10.x or 11.x. It was previously was available on iOS 8.0, but Apple removed it 8.1 due to problems with their implementation. It was back in 9.0 and has been available since.

Events from old browsers where our Grade A feature test fails are most certainly the result of User-Agent mangling. The Hive dataset being used in this task is incapable of perfection. This is by design. Being curious and scrupulous is good, but be sure to not have any expectation of it becoming perfect nor to fully understand why it isn't. It should be used to inform holistic information, not individually. Because:

1. The UA string sent by web clients can be trivially manipulated by users via their browser settings, by browser extensions modifying requests, and by headless processes such as bots, scrapers and crawlers that may or may not accurately expose their true internals via the User-Agent string.
2. The string is aggregated before it reaches Hive by the ua-parser library, which simplifies long and complex strings like Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 into something more digestible, such as Android 2.3, or maybe Kindle Fire 6.3, or maybe Mobile Safari 4.0? This is a lossy transformation and inherently produces an incorrect summary for some results. Contrary to what one might think, misclassifications or unknowns rarely end up as "Other", rather they typically end up filed under another name. This is unavoidable due to the complicated history of user-agent strings and their mixed purpose. E.g. most unknowns are insignificant variations of knowns and we want them grouped together.. except when we find them significant, which is hard to define, or at least subjective. The whole system depends on the goodwill and competence of device manufacturers and their users.

The current task title speculates about one of several possible explanations about why the schemas don't have the same number of received events. I can help narrow down the cause when more information is available, but my gut feeling tells me it is highly unlikely that it relates to availability of NavTiming or sendBeacon APIs.

Instead, I suspect the reason is that our code and the browser work fine, but our code just isn't triggered in the first place sometimes. This isn't a single reason, it's a category of reasons, all of which are likely true to some extent:

• The page can close between event A being sent and event B still awaiting asynchronous code and lazy-loading of modules. This is among the reasons I advocate against client-side event validation and against async abstractions and abstractions such as mw.eventLog.Schema. As part of T187207, I'm collaborating with Analytics to establish a much more direct and lightweight method that effectively provides a straight path to navigator.sendBeacon. Which, once reached, provides fairly strong guarantee of delivery (for as far as that is possible over the Internet). Even on slow or intermittent connections, or when the page is closed before the beacon is delivered, the browser is meant to remember beacons and send them whenever, even outside the bounds of the tab that was once open. This is why the API was created - a new primitive separate from XHR and Fetch.

• Network loss. Aside from client-side guarantees, there is also the network. The Internet doesn't provide a 100% delivery of requests and responses. Stuff happens. This is normal and expected. It's our responsibility to balance this with a compromise, or a heavy investment in complexity based on the needs and their relevant important. E.g. sending a beacons from a script without user input won't have the same guarantees as someone deciding to save an edit. The application and the user's browser can provide direct feedback and allow a user to interpret what and how much worked, and whether they are willing to try again. This is significant and usually requires user negotiation (or developer negotiation) because it may be different in subtle ways from the original (e.g. one second later is a different timestamp, potentially different IP address, potentially different cookies and their expires etc.). Even if such negotiation existed for scripts, it couldn't run after the tab is closed.

• Response to loss. If a browser tried sending it and couldn't get confirmation from the server, it doesn't know if it was delivered. It can try a second time and risk over-presenting the event. Or it may be cautious and not send again unless it know it failed, which may underrepresent the event.

Given the size of the anomaly, I'm not sure how much further we should investigate. But do let me know if you find a particular problem or have questions about something, as we should certainly make sure that anything we control works the best it can.

The UA string sent by web clients can be trivially manipulated by users via their browser settings, by browser extensions modifying requests, and by headless processes such as bots, s

+1. As I mentioned to @Jdlrobson it is very likely that up to 5% of our "user-classified" traffic is actually just not identified bots with fake UAS

I'm about to post the more detailed summary of the finding and data analysis recommendations that resulted from the above discussion, and then close this task, but just to follow up on some interesting remarks by @Krinkle:

In T204143#4671802, @Krinkle wrote:

...

Mobile Safari added support for Navigation Timing in iOS 9.0, not 10.x or 11.x. It was previously was available on iOS 8.0, but Apple removed it 8.1 due to problems with their implementation. It was back in 9.0 and has been available since.

This is a discrepancy we weren't able to resolve. (@Krinkle was referring here to the fact that based on our data in T204143#4653278, almost all iOS devices with version 11.3 and newere are sending ReadingDepth events, and almost all with older versions don't.)
We have been circumventing this issue by conservatively excluding data from all versions prior to 11.3 , see T204143#4661935 .

Events from old browsers where our Grade A feature test fails are most certainly the result of User-Agent mangling. The Hive dataset being used in this task is incapable of perfection. This is by design. Being curious and scrupulous is good, but be sure to not have any expectation of it becoming perfect nor to fully understand why it isn't.

Agreed, that's why we only focused on the larger discrepancies in this task and let many smaller ones slide (cf. T204143#4663878 ).

It should be used to inform holistic information, not individually. Because:

1. The UA string sent by web clients can be trivially manipulated by users via their browser settings, by browser extensions modifying requests, and by headless processes such as bots, scrapers and crawlers that may or may not accurately expose their true internals via the User-Agent string.

Agreed, but that seems to be something to keep in mind ever single time we rely on user agent data, not just in this particular investigation ;)

2. The string is aggregated before it reaches Hive by the ua-parser library, which simplifies long and complex strings like Mozilla/5.0 (Linux; U; Android 2.3.4; en-us; Kindle Fire Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 into something more digestible, such as Android 2.3, or maybe Kindle Fire 6.3, or maybe Mobile Safari 4.0?

True - as mentioned in T204143#4617685 , the raw user agent was removed from EL data a while ago, leaving only that ua-parser result. (That said, for recent EL events it should be possible to reconstruct the full UA from the webrequest table while it is not yet purged, but that probably would not have been worth the effort here.)

This is a lossy transformation and inherently produces an incorrect summary for some results. Contrary to what one might think, misclassifications or unknowns rarely end up as "Other", rather they typically end up filed under another name.

Indeed, see also the results of T193578 ...

[...]

Instead, I suspect the reason is that our code and the browser work fine, but our code just isn't triggered in the first place sometimes. This isn't a single reason, it's a category of reasons, all of which are likely true to some extent:

• The page can close between event A being sent and event B still awaiting asynchronous code and lazy-loading of modules. This is among the reasons I advocate against client-side event validation and against async abstractions and abstractions such as mw.eventLog.Schema. As part of T187207, I'm collaborating with Analytics to establish a much more direct and lightweight method that effectively provides a straight path to navigator.sendBeacon. Which, once reached, provides fairly strong guarantee of delivery (for as far as that is possible over the Internet).

Apropos, I noticed that T187207 was closed a couple of days ago. We can't repeat the queries above as the PageIssues schema has been deactivated, but if there are other ways to check whether T187207 has an impact on the kind of issues that had been investigated here, that would be very interesting.

[...]
Given the size of the anomaly, I'm not sure how much further we should investigate. But do let me know if you find a particular problem or have questions about something, as we should certainly make sure that anything we control works the best it can.

No, it seemed that with the findings up to that point we had already covered the most important UA segments that needed to be excluded. That said, insights about the limitations of the ReadingDepth data continue to be relevant and welcome.

It will also be wise to exclude events that are happening (for 1 entity) at too high of a rate , even if marked as user , those indicate probably automated traffic. You can set a high threshold, like events from one entity with more than say 30 requests per minute are probably automated, User Agents on those case mean very little and that type of data is just going to add more noise.

In T204143#4896637, @Nuria wrote:

It will also be wise to exclude events that are happening (for 1 entity) at too high of a rate , even if marked as user , those indicate probably automated traffic. You can set a high threshold, like events from one entity with more than say 30 requests per minute are probably automated, User Agents on those case mean very little and that type of data is just going to add more noise.

Thanks for the suggestion! I'm not going to incorporate it into the data recommendation outcomes from this task for now, considering that this potential issue sounds more like something that would affect EventLogging in general, or at least multiple schemas. (The primary purpose of this task was to determine whether this particular schema shows widespread unexpected behaviour for entire browser families or (ranges of) browser versions.)

That said, it's indeed an intriguing question and I'm inclined to run some queries to understand how this might affect EL in general. What definition of "entity" would you suggest for investigating it? How about the combination of IP and (raw) user agent?

(The ReadingDepth schema in particular also contains the page token which can be used to catch extraneous events sent during the same page view; @Groceryheist had already been excluding those for the purposes of the Reading Time investigation and IIRC they were very infrequent.)

It affects EL in general but not all events alike, many bots click around links in pages and schemas that capture clicks are affected most prominently by bots meaning that their numbers are more distorted.