Page MenuHomePhabricator

[Metrics Platform] QA/testing privleged mode: eventlogging-display-web, a QA user group, or something else?
Open, MediumPublic

Description

For QA and testing it is extremely handy to have access to the protected state of the EventLogging extension. However, for many events, such as session_tick, the events may be firing from many different browser windows and tabs, which makes it modifying local state a tricky proposition.

Things that are useful in such a mode include:

  • Inspecting the loaded stream configs
  • Inspecting the state of the output queue
  • Displaying events that will be sent as JSON in the console or in the browser as a pop-up
  • Always being in-sample

What is the best mix of behavior here. Whatever it is it should be specified to make the task of QA easier.

Event Timeline

Adding a note here that it is probably not a good idea to be able to gain this access using a setting that is not restricted or regulated in some way, i.e. if the user or agent is able to obtain this state themselves. This kind of unregulated access is what we have today with EventLogging's 'debugMode' via the hidden user setting 'eventlogging-display-web', but importantly this has not affected the core operation of the library nor the events that hit the wire, only what is displayed locally to the user. In a recent patch we also made it cause the user agent to be in-sample no matter what the stream config is. This is a slippery slope and we should back it out as soon as practical. Altered sampling rate and/or the ability to inspect the internal state (such as we might desire for e.g. unit testing), actually change the behavior in a way that is deeper than just what is being displayed locally, and hence should likely not be controlled by something as simple as a hidden option.

LGoto triaged this task as Medium priority.Mar 24 2021, 6:18 PM
  • Inspecting the loaded stream configs

This would be particularly useful when reporting bugs in instruments either in QA or in production. The JavaScript Metrics Platform Client (MPC) already provides the MetricsClient::getStreamConfig() method. It'd be trivial to add a MetricsClient::getStreamConfigs method. Like MetricsClient::getStreamConfig(), ::getStreamConfigs() must not allow modifying the stream configs themselves.

  • Inspecting the state of the output queue
  • Displaying events that will be sent as JSON in the console or in the browser as a pop-up

The latter is already implemented in both the JavaScript EventLogging client the JavaScript MPC. The queue itself is covered in unit tests and battle tested (it backs both EventLogging and statsv ). Is it enough to know that an event has been enqueued?

  • Always being in-sample

With the changes for T309306: Allow a stream to register interest in multiple events more easily landed, this is possible for logged-in QA Engineers using Metrics Platform Clients as they are without modifying or conditionally ignoring stream configs:

We add all QA Engineers to the wmf-qa group. We then define a stream, e.g. "test.qa.all", which 1) is interested in all events; 2) filters events where the performer is in the wmf-qa group, i.e.

$wgEventStreams = [
  [
    'stream' => 'test.qa.all',
    'schema_title' => '/analytics/mediawiki/client/metrics_event',
    'destination_event_service' => 'eventgate-analytics-external',
    'producers' => [
      'metrics_platform_client' => [
        'events' => [ '' ],
        'provide_values' => [
          'performer_groups',
        ],
        'curation' => [
          'performer_groups' => [
            'contains' => 'wmf-qa',
          ],
        ],
      ],
    ],
  ],
];