Page MenuHomePhabricator

Increase QuickSurveys' observability
Open, MediumPublic

Description

As a surveyor
I want to understand how my survey is performing soon after deploying it
So that I can correct course as necessary

As a maintainer
I want to understand how QuickSurveys is performing
So that I can respond as necessary


Background

QuickSurveys is observable but not immediately so:

  1. In order to know how many surveys are currently enabled, you must either:
    • Search for wgQuickSurveysConfig in operations/mediawiki-config
    • Navigate to a wiki and type mw.config.get( 'wgEnabledQuickSurveys', [] ) in your browser's console
  2. In order to determine how many users have seen a survey, you must query the event.QuickSurveysInitiation Hive table
  3. In order to determine how many users have engaged with a survey, you must query the event.QuickSurveysResponses Hive table

Overall Observability

The following information should be available in one place:

  1. How many surveys are enabled
  2. How quickly surveys are rendering
  3. How many surveys are seen
  4. How many surveys are being engaged with

Per-survey Observability

The above information should also be available on a per-survey basis.

TODO: Should the above information also be available on a per-country basis?

Post-deployment Tasks

  • Create a dashboard in Grafana
  • Update mw:Extension:QuickSurveys
    • … at the very least, link to the dashboard
  • Update the description of the QuickSurveys Phabricator project

Event Timeline

ovasileva triaged this task as Medium priority.Feb 28 2020, 10:53 AM
ovasileva moved this task from Incoming to Triaged but Future on the Web-Team-Backlog board.

If a <= 24 hour lag is acceptable, it would be fast and cheap to add these queries to the list of daily analytics scripts, summarizing each survey and pushing to Graphite or another public dashboard.

Change 764336 had a related patch set uploaded (by Phuedx; author: Phuedx):

[mediawiki/extensions/QuickSurveys@master] Add broad and per-survey observability

https://gerrit.wikimedia.org/r/764336

Hey @phuedx did you need a review from web team here? I'm out for the next week, but if so please feel free to reach out to Olga.

Hey @phuedx did you need a review from web team here? I'm out for the next week, but if so please feel free to reach out to Olga.

Thanks for asking/setting expectations, @Jdlrobson. Merging the above change isn't super-urgent. I'll reach out to Olga so that she's aware of it.

The solution proposed by @phuedx is to utilize statsd to increment a per survey counter for every survey response we get. Once that's done I presume web team will help setup the Grafana dashboards.

The solution proposed by @phuedx is to utilize statsd to increment a per survey counter for every survey response we get. Once that's done I presume web team will help setup the Grafana dashboards.

That's certainly a valid solution, but I wanted to point out that we're already collecting much richer data covering the same events, through the more modern event platform. If you prefer grafana dashboards you can write a Report Updater query which will aggregate events and can export to Graphite. The event schema allows us to report on the edit count bucket of respondents, among other interesting details that can be added in a custom report. CC'ing @mforns who has helped me understand this pipeline.

Amazing! I'm more than happy to abandon the patch and learn more about Report Updater.

Change 764336 abandoned by Phuedx:

[mediawiki/extensions/QuickSurveys@master] Implement simple observability

Reason:

Abandoning this is favour of using ReportUpdater <https://wikitech.wikimedia.org/wiki/Analytics/Systems/Reportupdater> to generate reports and export them to Graphite.

https://gerrit.wikimedia.org/r/764336