Change Details

It's happened in the past that we've had incorrect event logging causing us to lose insights into how users are interacting with features. Validation of events on the server-side helps us catch problems once they've reach production, because we can monitor our [event logging validation logstash dashboard](https://logstash.wikimedia.org/app/dashboards#/view/41cae700-bfbe-11eb-85b7-9d1831ce7631). But unfortunately that is reactive, it means the problem is already in production and then it takes some time to fix, backport, etc. Validation also does not help us if expected events are missing entirely. It would be nice to have an end-to-end test that asserted the expected events are generated, with proper validation, for the happy path and most important scenario for #growthexperiments which is visiting Special:Homepage, clicking a task, and saving an edit. We already have browser tests that do most of this work; what is missing is the code to assert that the correct events are generated and have passed validation. To do that, I think it would be easiest and most robust to have the eventlogging devserver running in CI, and the code making assertions could parse the JSON logs for the devserver to assert the correct sequence of events. Parsing the JSON logs would also tell us if any of the events failed validation.

It's happened in the past that we've had incorrect event logging causing us to lose insights into how users are interacting with features. (Recent example: {T301486}) Validation of events on the server-side helps us catch problems once they've reach production, because we can monitor our [event logging validation logstash dashboard](https://logstash.wikimedia.org/app/dashboards#/view/41cae700-bfbe-11eb-85b7-9d1831ce7631). But unfortunately that is reactive, it means the problem is already in production and then it takes some time to fix, backport, etc. Validation also does not help us if expected events are missing entirely. It would be nice to have an end-to-end test that asserted the expected events are generated, with proper validation, for the happy path and most important scenario for #growthexperiments which is visiting Special:Homepage, clicking a task, and saving an edit. We already have browser tests that do most of this work; what is missing is the code to assert that the correct events are generated and have passed validation. To do that, I think it would be easiest and most robust to have the eventlogging devserver running in CI, and the code making assertions could parse the JSON logs for the devserver to assert the correct sequence of events. Parsing the JSON logs would also tell us if any of the events failed validation.

It's happened in the past that we've had incorrect event logging causing us to lose insights into how users are interacting with features. (Recent example: {T301486}) Validation of events on the server-side helps us catch problems once they've reach production, because we can monitor our [event logging validation logstash dashboard](https://logstash.wikimedia.org/app/dashboards#/view/41cae700-bfbe-11eb-85b7-9d1831ce7631). But unfortunately that is reactive, it means the problem is already in production and then it takes some time to fix, backport, etc. Validation also does not help us if expected events are missing entirely. It would be nice to have an end-to-end test that asserted the expected events are generated, with proper validation, for the happy path and most important scenario for #growthexperiments which is visiting Special:Homepage, clicking a task, and saving an edit. We already have browser tests that do most of this work; what is missing is the code to assert that the correct events are generated and have passed validation. To do that, I think it would be easiest and most robust to have the eventlogging devserver running in CI, and the code making assertions could parse the JSON logs for the devserver to assert the correct sequence of events. Parsing the JSON logs would also tell us if any of the events failed validation.