Page MenuHomePhabricator

Automated tests for Javascript error logging
Closed, DeclinedPublic

Description

Find a way to detect when error reporting breaks.

  1. write an extension that produces various types of errors (PHP, JS module setup, JS async callback etc)
  2. ???
  3. profit

Step 2 can probably be done via Selenium, but need to think about that more.

Related Objects

StatusSubtypeAssignedTask
DeclinedNone
ResolvedTgr
ResolvedGilles
OpenNone
OpenNone
OpenNone
DeclinedNone
DeclinedTgr
ResolvedTgr
ResolvedTgr
Resolvedcsteipp
ResolvedTgr
ResolvedTgr
ResolvedAklapper
ResolvedTgr
ResolvedTgr
OpenNone
ResolvedTgr
DeclinedTgr
DeclinedTgr
DeclinedTgr
ResolvedTgr
DeclinedTgr
ResolvedTgr
ResolvedTgr
ResolvedKrinkle
DeclinedNone
OpenNone
ResolvedTgr
DeclinedNone
OpenNone
InvalidNone
DeclinedTgr
ResolvedTgr
Resolvedjcrespo
ResolvedTgr

Event Timeline

Tgr created this task.Jan 30 2015, 4:09 AM
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added a project: Sentry.
Tgr added a subscriber: Tgr.
Restricted Application added a project: Multimedia. · View Herald TranscriptJan 30 2015, 4:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot added a subscriber: gerritbot.

Change 188963 had a related patch set uploaded (by Gergő Tisza):
Initial commit

https://gerrit.wikimedia.org/r/188963

Patch-For-Review

Tgr added a comment.Feb 6 2015, 2:33 AM

Created Extension:Buggy which provides the tests needed for the report in T88399.

Tgr moved this task from Untriaged to Needs code review on the Multimedia board.Feb 6 2015, 2:33 AM
Tgr claimed this task.Feb 6 2015, 3:01 AM

Change 188963 merged by Gergő Tisza:
Initial commit

https://gerrit.wikimedia.org/r/188963

Tgr added a comment.Feb 11 2015, 8:53 PM

TODO: create jenkins config so that merges actually get merged.

Karma seems the most appropriate for this: it executes QUnit tests in an actual browser (ie. much higher level of abstraction than hand-coding JS tests in Watir), and is already included in the MediaWiki CI suite. QUnit has its own onerror handler, but that can just be disabled for the duration of these tests. sinon.js's mocking of setTimeout etc. also needs to be disabled. These tests would be integration tests rather than unit tests, making this a slight abuse of QUnit, but meh.

Tgr set Security to None.Mar 11 2015, 5:32 PM
Tgr edited a custom field.
Gilles moved this task from Next up to Untriaged on the Multimedia board.Mar 23 2015, 8:54 AM
zeljkofilipin triaged this task as Medium priority.Mar 25 2015, 1:43 PM

Backlogging. As long as we are not going to do any of the complicated callback wrapping, setting up an automated test is not really worth the effort.

zeljkofilipin edited a custom field.
zeljkofilipin added a subscriber: zeljkofilipin.

Removed patch-for-review since the patch is merged. Removed browser-tests because it looks to me it is not related. Please add the tag(s) back if needed.

Tgr added a comment.Sep 21 2015, 10:50 PM

I was hoping that browser test could provide the ??? part :)
I'm not really sure how to go about this. We could set up a browser test to check that an error gets reported when the browser visits page X which is known to have a bug, but that's not terribly interesting - error catching is done via the onerror event which we don't really expect to fail, and there are QUnit tests to make sure that the custom error reporting does not break when the error event is fired.
What's actually worth testing is that a sane amount of information is extracted (such as a stack trace). That depends on a number of interacting components (the browser's own API for error details, ResourceLoader, the Sentry extension, the 3rd party code from Raven-js, HTTP header settings) and could break easily. It does not seem easy to test though, especially if eventually we want to involve source maps, which are applied on the Sentry server. Given that browser behavior is one potential source of problems, the test would have to involve browser testing at the very least (maybe via Karma or something similar and not Watir, but tests would definitely have to run in multiple browsers), but then the errors logs should probably have to queried from the Sentry server for verification...

Tgr lowered the priority of this task from Medium to Low.Sep 21 2015, 10:52 PM

Anyway this is a lot of effort and not really worth it unless Sentry gets deployed widely. Maybe not even then since we can just use the actual live error reports to notice when error reporting doesn't work right... not sure.

Aklapper removed Tgr as the assignee of this task.Jun 19 2020, 4:30 PM

This task has been assigned to the same task owner for more than two years. Resetting task assignee due to inactivity, to decrease task cookie-licking and to get a slightly more realistic overview of plans. Please feel free to assign this task to yourself again if you still realistically work or plan to work on this task - it would be welcome!

For tips how to manage individual work in Phabricator (noisy notifications, lists of task, etc.), see https://phabricator.wikimedia.org/T228575#6237124 for available options.
(For the records, two emails were sent to assignee addresses before resetting assignees. See T228575 for more info and for potential feedback. Thanks!)

Tgr renamed this task from Automated tests for Sentry error logging to Automated tests for Javascript error logging.Sep 17 2020, 2:06 AM
Tgr removed a project: Sentry.Sep 17 2020, 2:37 AM

At the time of creating this task it was assumed that the server side implementation of error logging would be based on Sentry. We have eventually decided on a different implementation, so de-tagging.

Aklapper closed this task as Declined.Oct 4 2020, 12:00 PM

The parent tasks have been declined, hence also declining this task.
If this task should still be open, then please update its description and associate an active project - thanks!