There are many classes of reliability issues (e.g. failures/misconfigurations in intermediate networks) that we only find out about via direct, manual reports from users, or (for very widespread cases) we notice because traffic is 'missing' and below expected rates.
Some sort of 'external' monitoring is the usual solution to such blind spots, but of course such solutions come with their own false positive errors and other limitations (requires agreements with commercial providers; sometimes limited APIs for scraping result data; often reliability problems that are specific to the monitoring provider's infrastructure rather than something 'real'; generally such a provider's probes run within datacenters instead of on Internet edges / from within residential ISP networks; geographic distribution of provider's probes don't match userbase geographic distribution; characteristics of synthetic traffic doesn't necessarily match characteristics of real traffic; //&c. &c.//).
There's another option, which is asking browsers to send you an error report some fraction of the time when they can't fetch from your site. This is a W3C draft technical report, the [[ https://www.w3.org/TR/network-error-logging/ | Network Error Logging ]] API, part of the broader [[ https://w3c.github.io/reporting/ | Reporting ]] API. Currently the NEL API is implemented and enabled by default in [[ https://developer.mozilla.org/en-US/docs/Web/HTTP/Network_Error_Logging#Browser_compatibility | only Chrome >=71 and Edge >=79 ]], but that's still a large fraction of all traffic and users.
Asking browsers to enable NEL is implemented by serving HTTP response headers `Report-To` and `NEL`, which together define a set of endpoints that can receive reports, sampling fractions for each of failures and successes, and a TTL for this entire definition to be stored in the user's browser. See [[ https://www.w3.org/TR/network-error-logging/#sample-policy-definitions | Sample Policy Definitions ]].
==== Privacy concerns ====
//See [[ https://www.w3.org/TR/network-error-logging/#sample-network-error-reports | Sample Network Error Reports ]]//
Error reports are full of PII. They're sent from user IP addresses, contain the URL that the user was attempting to fetch, any Referer: from that original request, and in the future could optionally include specific request or response headers from the original request. They require TLS over the wire and deserve all the same protections that our logs data gets at rest.
==== v0: Minimum viable deployment: ====
* [x] Legal review
* [ ] T259160 Privacy review
* [ ] Configure [[ https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate | EventGate ]] to receive NEL reports and store them in Logstash via Kafka.
** We'll likely use the exact same EventGate instances set up to receive client-side Javascript error reports in T226986.
** [x] However, EventGate-Wikimedia will have to be modified, as NEL reports don't include EventGate-specific expected metadata fields. @Ottomata has prepared a [[ https://gerrit.wikimedia.org/r/c/eventgate-wikimedia/+/609222 | simple patch ]] that ought to allow us to set a reporting endpoint URI of something like `https://intake-logging.wikimedia.org/v1/events?schema_uri=/network/error/logging/1.0.0&stream=network.error`
** [x] ~~Determine whether or not we want additional stream processing to split apart NEL responses into their component events, as each POST made to the reporting endpoint is potentially a batch of multiple error reports.~~ EventGate does this already!
** [ ] Write a schema that matches the NEL specification and verify it validates reports generated by Chrome stable.
** [ ] Deploy an eventgate-wikimedia with all of these changes.
* [ ] Begin sending `Report-To` and `NEL` headers on our responses.
** The traffic layer seems like the right place to insert these headers. We should do a staged rollout, starting with a small fraction of traffic and with short TTLs, and expand once confident.
* [ ] Build a reasonably-nice Logstash dashboard to aggregate NELs.
==== v1: Improvements that aren't too hard ====
* [ ] T261340 Set up a "backwards GeoDNS" hostname that routes users to a faraway datacenter, or at least, a datacenter that won't be their usual primary datacenter. Use that hostname to receive error reports.
** Browsers are supposed to buffer reports and retry later if they can't send them the first time, but this will help us receive reports as outages are happening, not after they're resolved.
* [ ] Consider if we want NEL reports stored anywhere other than Logstash -- e.g. it might be useful to also have them in their own table in [[ https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive | Hive ]].
==== Harder / more open-ended future work: ====
* Attempt some sort of alerting based on receiving NELs.
* Set a nonzero `success_fraction` to also collect latency data for GeoDNS-mapping test URLs served at the edge; use that to improve our GeoDNS assignments.
* Consider setting up some off-WMF-infrastructure report collectors, taking extra care to keep them maintainable, secure, and to //not// store PII at rest there. This would all require careful planning and review, but the upside is we could get near-realtime data about user issues, despite those users being unable to reach any WMF infrastructure.