Page MenuHomePhabricator

Enable client side error logging in prod for small wiki
Closed, ResolvedPublic5 Estimated Story Points

Description

We look at haw.wikipedia.org earlier on, let's take a quick look to make sure this is a suitable site to enable error logging 100% for all pageviews: https://haw.wikipedia.org/wiki/Ka_papa_kinohi

Event Timeline

Nuria created this task.Feb 24 2020, 6:39 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFeb 24 2020, 6:39 PM
Nuria updated the task description. (Show Details)Feb 24 2020, 6:41 PM
Nuria assigned this task to Ottomata.Feb 26 2020, 5:35 PM
Nuria added a project: Analytics-Kanban.
Nuria removed a subscriber: CDanis.
Milimetric triaged this task as High priority.Mar 2 2020, 4:57 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.

Change 576120 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable client side error logging in beta

https://gerrit.wikimedia.org/r/576120

Change 576120 merged by Ottomata:
[operations/mediawiki-config@master] Enable client side error logging in beta

https://gerrit.wikimedia.org/r/576120

Change 576126 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Fix wgWMEClientErrorIntakeURL in beta

https://gerrit.wikimedia.org/r/576126

Change 576126 merged by Ottomata:
[operations/mediawiki-config@master] Fix wgWMEClientErrorIntakeURL in beta

https://gerrit.wikimedia.org/r/576126

Mentioned in SAL (#wikimedia-operations) [2020-03-02T20:01:24Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Enable Mediawiki client side error logging in beta - T246030 (duration: 00m 56s)

Change 576130 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable client side (browser) error logging for group0 wikis

https://gerrit.wikimedia.org/r/576130

Change 576130 merged by Ottomata:
[operations/mediawiki-config@master] Enable client side (browser) error logging for group0 wikis

https://gerrit.wikimedia.org/r/576130

Mentioned in SAL (#wikimedia-operations) [2020-03-02T20:22:47Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: Enable Mediawiki client side error logging on group0 wikis - T246030 (duration: 00m 57s)

Change 576145 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/WikimediaEvents@master] Set wgWMEClientErrorIntakeURL in onResourceLoaderGetConfigVars

https://gerrit.wikimedia.org/r/576145

Change 576149 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.21] Set wgWMEClientErrorIntakeURL in onResourceLoaderGetConfigVars

https://gerrit.wikimedia.org/r/576149

Change 576145 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Set wgWMEClientErrorIntakeURL in onResourceLoaderGetConfigVars

https://gerrit.wikimedia.org/r/576145

Change 576149 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.21] Set wgWMEClientErrorIntakeURL in onResourceLoaderGetConfigVars

https://gerrit.wikimedia.org/r/576149

Mentioned in SAL (#wikimedia-operations) [2020-03-03T19:19:49Z] <jforrester@deploy1001> Synchronized php-1.35.0-wmf.21/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: T246030 T226986: Set wgWMEClientErrorIntakeURL in onResourceLoaderGetConfigVars (duration: 01m 05s)

Ottomata added a comment.EditedMar 3 2020, 9:06 PM

Hm, ok navigator.sendBeacon from the browser to intake-logging.wikimedia.org works. On test.wikipedia.org in browser console:

> var event = {"$schema": "/mediawiki/client/error/1.0.0", "meta": {"stream": "mediawiki.client.error"}, "message": "test event", "type": "TEST", "url": "http://otto-test.org", "user_agent": "otto test"}

> navigator.sendBeacon( mw.config.get('wgWMEClientErrorIntakeURL'), JSON.stringify(event));

This results in a message in kafka-logging in topic eqiad.mediawiki.client.error:

{"$schema":"/mediawiki/client/error/1.0.0","meta":{"stream":"mediawiki.client.error","id":"3bf6f574-c508-4c73-bb64-53db8d23e0f7","dt":"2020-03-03T21:02:07.591Z","request_id":"4cf16df6-bfb6-46cd-a17e-b03749c03a60"},"message":"test event","type":"TEST","url":"http://otto-test.org","user_agent":"otto test","http":{"client_ip":"2604:2000:12c1:60e:3812:1ace:7867:1338","request_headers":{"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36"}}}

However, I can't seem to get this to work via mw.track('global.error', ...).

> mw.track( 'global.error', {
    errorMessage: "test event",
    url: "http://otto-test.org",
    lineNumber: 1234,
    columnNumber: 12,
    stackTrace: "test stack",
    errorObject: new Error("test error")
} );
// ... nothing happens, and the trackQueue seems to never be cleared:

> mw.trackQueue
0: {topic: "mw.deprecate", data: "jquery-migrate"}
1: {topic: "global.error", data: {}}
2: {topic: "global.error", data: {}}
3: {topic: "global.error", data: {}}

I'd also like to test triggering this from a real client error, but I'm not quite sure how to do so.

@jlinehan would appreciate some help debugging this! I think the global.error handler is just not getting called, but I don't really know how to troubleshoot.

Tgr added a comment.Mar 3 2020, 9:43 PM

Works for me as expected:

> mw.trackSubscribe( 'global.error', (...args) => console.log( 'error logged: ', ...args ) )
> mw.track( 'global.error', {
    errorMessage: "test event",
    url: "http://otto-test.org",
    lineNumber: 1234,
    columnNumber: 12,
    stackTrace: "test stack",
    errorObject: new Error("test error")
  } );
error logged:  global.error {errorMessage: "test event", url: "http://otto-test.org", lineNumber: 1234, columnNumber: 12, stackTrace: "test stack", }

I don't think the queue is ever cleared, the idea is that you can subscribe to a channel and get the events that fired before you subscribed.

(Note though that global.error is the channel, not the handler, so this doesn't strictly prove some specific handler is getting called. But subscribing as a handler does work in general.)

Nuria added a comment.EditedMar 4 2020, 3:56 AM

@Tgr This is a "mediaWiki newbies 101 question".. how can i see that wikimedia events extension is deployed to test wiki? The code that subscribes to the error channel a logs those is here: https://github.com/wikimedia/mediawiki-extensions-WikimediaEvents/blob/master/modules/ext.wikimediaEvents/clientError.js#L28 and it does not seem to be called

Deployed extensions can be seen on any wiki at /Special:Version, which on hawaiian wikipedia is at https://haw.wikipedia.org/wiki/Papa_nui:Mana. And I learned that Pepeluali is February, so Wikimedia Events was deployed at 08:37, 25 February 2020.

Tgr added a comment.Mar 4 2020, 6:16 AM

An easier although probably less educational way to interpret the page is by looking at https://haw.wikipedia.org/wiki/Papa_nui:Mana?uselang=en.

Tgr added a comment.Mar 4 2020, 6:28 AM

Error reporting is disabled for that wiki:

> mw.config.get( 'wgWMEClientErrorIntakeURL' )
< false

It's true on group0 wikis only.

Change 576540 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/extensions/WikimediaEvents@master] Fix callback parameters for client error logging

https://gerrit.wikimedia.org/r/576540

Error reporting is disabled for that wiki:

haw yes, we were testing in on test.wikipedia.org first.

THANK YOU FOR PATCH

Change 576540 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Fix callback parameters for client error logging

https://gerrit.wikimedia.org/r/576540

Change 576923 had a related patch set uploaded (by Ottomata; owner: Gergő Tisza):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.21] Fix callback parameters for client error logging

https://gerrit.wikimedia.org/r/576923

Change 576923 merged by Ottomata:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.21] Fix callback parameters for client error logging

https://gerrit.wikimedia.org/r/576923

Change 576931 had a related patch set uploaded (by Ottomata; owner: Gergő Tisza):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.22] Fix callback parameters for client error logging

https://gerrit.wikimedia.org/r/576931

Change 576931 merged by Ottomata:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.22] Fix callback parameters for client error logging

https://gerrit.wikimedia.org/r/576931

Mentioned in SAL (#wikimedia-operations) [2020-03-04T19:58:22Z] <otto@deploy1001> Synchronized php-1.35.0-wmf.22/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: SWAT: [[gerrit:576931|Fix callback parameters for client error logging (T246030)]] (duration: 01m 07s)

Change 576940 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/WikimediaEvents@master] Include required url in mediawiki/client/error event

https://gerrit.wikimedia.org/r/576940

Change 576942 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.22] Include required url in mediawiki/client/error event

https://gerrit.wikimedia.org/r/576942

Change 576940 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Include required url in mediawiki/client/error event

https://gerrit.wikimedia.org/r/576940

Change 576942 merged by Ottomata:
[mediawiki/extensions/WikimediaEvents@wmf/1.35.0-wmf.22] Include required url in mediawiki/client/error event

https://gerrit.wikimedia.org/r/576942

Change 576943 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[mediawiki/extensions/WikimediaEvents@master] Use the obj.url given by mw.errorLogger handler by default

https://gerrit.wikimedia.org/r/576943

Mentioned in SAL (#wikimedia-operations) [2020-03-04T20:29:57Z] <otto@deploy1001> Synchronized php-1.35.0-wmf.22/extensions/WikimediaEvents/modules/ext.wikimediaEvents/clientError.js: [[gerrit:576942|Include required url in mediawiki/client/error event (T246030)]] (duration: 01m 05s)

Ok, things are finally working on test.wikipedia.org! I have events! :)

Will enable in Hawaiian wiki tomorrow.

Ottomata moved this task from In Code Review to In Progress on the Analytics-Kanban board.

Change 576943 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Use obj.url given by mw.errorLogger in error event

https://gerrit.wikimedia.org/r/576943

Nuria added a comment.Mar 5 2020, 12:58 AM

@Tgr Super thanks for the help, gergo

Change 577260 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/mediawiki-config@master] Enable client side error logging on haw.wikipedia.org

https://gerrit.wikimedia.org/r/577260

Nuria added a comment.Mar 6 2020, 12:30 AM

I am a bit confused here cause i do not see things working on test wiki. The code below puts an error on the mw.trackQueue

>window.setTimeout(function() {throw new Error()}, 0);

>mw.trackQueue

[{…}]0: topic: "global.error"data: errorMessage: "Script error."url: ""lineNumber: 0columnNumber: 0stackTrace: ""errorObject: null__proto__: Object__proto__: Objectlength: 1__proto__: Array(0)

But no error seems to be send with the beacon (no request on network tab). Is the subscription from WikimediaEvents to the error topic not working? (or, likely, I am missing something else entirely here)

Nuria added a comment.Mar 6 2020, 12:35 AM

Is the mw.trackSubscribe working?

Tgr added a comment.Mar 6 2020, 6:07 AM

Script error means the browser srubbed error details because the error did not originate from code belonging to the same domain as the error logging code. Probably the error logger discards such reports as they are worthless.

I created an error gadget to make testing easer (it's towards the top of the gadget list) and tried with that; seemed to work fine.

Nuria added a comment.Mar 6 2020, 6:35 PM

@Tgr mmm, I think your last comment is incorrect. The gadget behaves the same than code like : window.setTimeout(function() {throw new Error()}, 0); they both put an event on the track queue but the subscription to the track queue that needs to fire a sendBeacon call is not happening in either case. Let's talk more about this.

Tgr added a comment.EditedMar 6 2020, 9:38 PM

I can see the network request for the mediawiki.client.error event triggered when I enable the gadget and click on the "throw error" link; I don't see it with the setTimeout code (which is expected behavior - yes it puts an event in the JS queue, the error logger subscribed to the queue then probably discards it because it has redacted data). Probably some browser issue on your side - what do you use for testing?

Change 577260 merged by Ottomata:
[operations/mediawiki-config@master] Enable client side error logging on haw.wikipedia.org

https://gerrit.wikimedia.org/r/577260

Mentioned in SAL (#wikimedia-operations) [2020-03-10T13:31:15Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Mediawiki client side error logging on hawwiki - T246030 (duration: 00m 58s)

Mentioned in SAL (#wikimedia-operations) [2020-03-10T13:41:48Z] <otto@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Mediawiki client side error logging on hawwiki (take 2) - T246030 (duration: 00m 57s)

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Mar 10 2020, 2:05 PM
fgiunchedi moved this task from Inbox to In progress on the observability board.Mar 16 2020, 2:21 PM
fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.Mar 20 2020, 1:31 PM
Nuria closed this task as Resolved.Apr 9 2020, 3:12 PM
Nuria set the point value for this task to 5.