Tracking WMDE mw metrics passed to Graphite
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	GoranSMilovanovic
	Jan 12 2018, 11:46 PM

Description

Create a sub-page of the WMDE Analytics Wikitech page to document the mw metrics that get passed to Graphite for our Grafana dashboards. The essential information that the page will need to represent is the name of the metric as it will appear in Graphite queries from Grafana, *or* the name of the variable/object as instantiated from PHP/JS such that its regular transcription to a name as it will appear in Graphite/Grafana exists.

*Example*. In the PHP/JS code, there exists a variable/object called mw.measure.content.numObs. Option A: exactly the same name is used in Graphite/Grafana. Option B: in Graphite/Grafana, the PHP/JS variable or object called mw.measure.content.numObs becomes something like <ArbitraryString1>.mw.measure.content.numObs<ArbitraryString2, or HeyGraphiteGrabbedThis.mw.measure.content.numObs, or whatever form that obeys a set of deterministic rules that transform the original PHP/JS name to a Graphite/Grafana name.

@GoranSMilovanovic to try to collect all such metrics that are relevant for the Technical Wishlist tickets that are currently in focus;

Ask @Addshore for a review; iterate (1 - 2) until the process becomes a routine.

Once the process is established, in coordination with @Lea_WMDE and @Addshore introduce all WMDE developers that need anything sent to Graphite to the documentation process on the respective Wikitech page.

Event Timeline

GoranSMilovanovic created this task.Jan 12 2018, 11:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 12 2018, 11:46 PM

GoranSMilovanovic triaged this task as High priority.Jan 12 2018, 11:47 PM

GoranSMilovanovic moved this task from Technical Wishlist to Prioritized on the User-GoranSMilovanovic board.

@Addshore @Lea_WMDE

The Wikitech page is ready.

@Addshore Please let me try to document the TwoColConflict metrics. As soon as I have something, I will ask you for a review.
@Lea_WMDE I will try to update the respective Grafana dashboard along the way, metric by metric.

@Addshore @Lea_WMDE The TwoColConflicts Grafana dashboard is now documented on WMDE_Anayltics_Graphite_Metrics.

I propose that we have all our future Grafana dashboards documented in this way. Please pay attention to the underlined text there; it tells where some information is missing. I developed this documentation in order to propose that we adopt the following workflow for all future dashboards or new components of the existing dashboards:

@Lea_WMDE and/or other colleagues from her team provide(s) descriptions (the examples are given on the Wikitech page);
@Addshore and/or other WMDE software developer(s) provide(s) metrics and explain their semantics where necessary;
@GoranSMilovanovic (which is "analytics" on the Wikitech page, currently one person only) provides Graphite queries and explains the implementation of the new dashboard or new dashboard component.

Your feedback on the proposed workflow would be highly appreciated. The terms description, metric, semantics, and implementation are used on the Wikitech page, and you will easily figure out what means what from there. To put it more concisely and in less formal terms:

@Lea_WMDE provides a description: this is the new indicator that we need on some dashboard, or we need a whole new dashboard with the following set of indicators (describing each indicator in that case);
@Addshore provides the names of the metrics that need to be used to compute each indicator and explains their semantics to @GoranSMilovanovic where necessary (i.e. when something previously unseen on the dashboard needs to be used);
@GoranSMilovanovic picks up from there, writes out the Graphite queries, implements the indicator on the dashboard, and reports back to @Lea_WMDE.

I will now try to develop the new dashboard components by looking for the information that is currently scattered across the Phab tickets and Gerrit patches, and than ask for review and help.

I think that if we systematize the work on the Grafana dashboards in this or similar fashion, the bottleneck that these tasks are causing will disappear in several weeks from now - in fact, in as much time as we need to get used to the routine.

Addshore renamed this task from Tracking mw metrics passed to Graphite to Tracking WMDE mw metrics passed to Graphite.Jan 15 2018, 4:41 PM

@Addshore @Lea_WMDE If I recall correctly, the decision in respect to this is to have all Graphite metrics used on Grafana documented in the respective Gerrit repositories.

If I need a separate documentation, I will take care of it somewhere, most probably on https://wikitech.wikimedia.org/wiki/Analytics/WMDE/WMDE_Anayltics_Graphite_Metrics

Closing the task.

GoranSMilovanovic closed this task as Resolved.Feb 8 2018, 2:51 PM

Addshore moved this task from Incoming to Done on the WMDE-Analytics-Engineering board.May 8 2018, 1:27 PM

Tracking WMDE mw metrics passed to GraphiteClosed, ResolvedPublicActions

Description

Event Timeline

Tracking WMDE mw metrics passed to Graphite
Closed, ResolvedPublic
Actions