Page MenuHomePhabricator

Create a test for end-to-end event logging data verification of happy path with Special:Homepage and suggested edits
Open, Needs TriagePublic

Description

It's happened in the past that we've had incorrect event logging causing us to lose insights into how users are interacting with features. (Recent example: T301486: Add an Image: back/next buttons in onboarding screens are not instrumented)

Validation of events on the server-side helps us catch problems once they've reach production, because we can monitor our event logging validation logstash dashboard. But unfortunately that is reactive, it means the problem is already in production and then it takes some time to fix, backport, etc.

Validation also does not help us if expected events are missing entirely.

It would be nice to have an end-to-end test that asserted the expected events are generated, with proper validation, for the happy path and most important scenario for GrowthExperiments which is visiting Special:Homepage, clicking a task, and saving an edit.

We already have browser tests that do most of this work; what is missing is the code to assert that the correct events are generated and have passed validation.

To do that, I think it would be easiest and most robust to have the eventlogging devserver running in CI, and the code making assertions could parse the JSON logs for the devserver to assert the correct sequence of events. Parsing the JSON logs would also tell us if any of the events failed validation.

Event Timeline