Page MenuHomePhabricator

Automated tests for Sentry error logging
Open, LowPublic

Description

Find a way to detect when error reporting breaks.

  1. write an extension that produces various types of errors (PHP, JS module setup, JS async callback etc)
  2. ???
  3. profit

Step 2 can probably be done via Selenium, but need to think about that more.

Related Objects

StatusAssignedTask
OpenTgr
ResolvedTgr
ResolvedGilles
OpenNone
OpenNone
OpenNone
OpenNone
OpenTgr
OpenTgr
ResolvedTgr
Resolvedcsteipp
ResolvedTgr
ResolvedTgr
ResolvedAklapper
ResolvedTgr
ResolvedTgr
OpenNone
ResolvedTgr
DeclinedTgr
DeclinedTgr
StalledTgr
ResolvedTgr
StalledTgr
ResolvedTgr
OpenTgr
ResolvedKrinkle
DeclinedNone
OpenTgr
ResolvedTgr
OpenTgr
OpenNone
InvalidNone
StalledTgr
ResolvedTgr
OpenNone
Resolvedjcrespo
ResolvedTgr

Event Timeline

Tgr created this task.Jan 30 2015, 4:09 AM
Tgr updated the task description. (Show Details)
Tgr raised the priority of this task from to Needs Triage.
Tgr added a project: Sentry.
Tgr added a subscriber: Tgr.
Restricted Application added a project: Multimedia. · View Herald TranscriptJan 30 2015, 4:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot added a subscriber: gerritbot.

Change 188963 had a related patch set uploaded (by Gergő Tisza):
Initial commit

https://gerrit.wikimedia.org/r/188963

Patch-For-Review

Tgr added a comment.Feb 6 2015, 2:33 AM

Created Extension:Buggy which provides the tests needed for the report in T88399.

Tgr moved this task from Untriaged to Needs code review on the Multimedia board.Feb 6 2015, 2:33 AM
Tgr claimed this task.Feb 6 2015, 3:01 AM

Change 188963 merged by Gergő Tisza:
Initial commit

https://gerrit.wikimedia.org/r/188963

Tgr added a comment.Feb 11 2015, 8:53 PM

TODO: create jenkins config so that merges actually get merged.

Karma seems the most appropriate for this: it executes QUnit tests in an actual browser (ie. much higher level of abstraction than hand-coding JS tests in Watir), and is already included in the MediaWiki CI suite. QUnit has its own onerror handler, but that can just be disabled for the duration of these tests. sinon.js's mocking of setTimeout etc. also needs to be disabled. These tests would be integration tests rather than unit tests, making this a slight abuse of QUnit, but meh.

Tgr set Security to None.Mar 11 2015, 5:32 PM
Tgr edited a custom field.
Gilles moved this task from Next up to Untriaged on the Multimedia board.Mar 23 2015, 8:54 AM
zeljkofilipin triaged this task as Normal priority.Mar 25 2015, 1:43 PM

Backlogging. As long as we are not going to do any of the complicated callback wrapping, setting up an automated test is not really worth the effort.

zeljkofilipin edited a custom field.
zeljkofilipin added a subscriber: zeljkofilipin.

Removed patch-for-review since the patch is merged. Removed browser-tests because it looks to me it is not related. Please add the tag(s) back if needed.

Tgr added a comment.Sep 21 2015, 10:50 PM

I was hoping that browser test could provide the ??? part :)
I'm not really sure how to go about this. We could set up a browser test to check that an error gets reported when the browser visits page X which is known to have a bug, but that's not terribly interesting - error catching is done via the onerror event which we don't really expect to fail, and there are QUnit tests to make sure that the custom error reporting does not break when the error event is fired.
What's actually worth testing is that a sane amount of information is extracted (such as a stack trace). That depends on a number of interacting components (the browser's own API for error details, ResourceLoader, the Sentry extension, the 3rd party code from Raven-js, HTTP header settings) and could break easily. It does not seem easy to test though, especially if eventually we want to involve source maps, which are applied on the Sentry server. Given that browser behavior is one potential source of problems, the test would have to involve browser testing at the very least (maybe via Karma or something similar and not Watir, but tests would definitely have to run in multiple browsers), but then the errors logs should probably have to queried from the Sentry server for verification...

Tgr lowered the priority of this task from Normal to Low.Sep 21 2015, 10:52 PM

Anyway this is a lot of effort and not really worth it unless Sentry gets deployed widely. Maybe not even then since we can just use the actual live error reports to notice when error reporting doesn't work right... not sure.