Page MenuHomePhabricator

[Analytics] [Request] Migrate all WMDE Grafana dashboards to Prometheus
Open, Needs TriagePublic

Description

Wikidata Analytics Request

This task was generated using the Wikidata Analytics request form. Please use the task template linked on our project page to create tasks for the team. Thank you!

Purpose

Please provide as much context as possible as well as what the produced insights or services will be used for.

As a part of T371616 we need to migrate all Grafana dashboards to Prometheus to assure that the new data is being visualized.

Specific Results

Please detail the specific results that the task should deliver.

Make a copy of all dashboards that used to have Graphite backends and create a new version that has Prometheus as the backend source. The new dashboard should be placed to the right of the old one so that the data can be read from left to right.

Desired Outputs

Please list the desired outputs of this task.

Migrate

All dashboards having new versions that reflect the Prometheus data and removing the Graphite Deprecation Notice (identified because they have this notice):

Wikidata

NOTE: Some of these were already migrated to Airflow and now have data in the private data lake.

WMDE Wikibase Product Platform

WMDE Technical Wishes

Deprecate (?)

Dashboards that are not needed anymore are deleted after communicating with the team:

Wikidata

WMDE Wikibase Product Platform

WMDE Technical Wishes

Deadline

Please make the time sensitivity of this request clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.

As soon as possible for all work in this task


Information below this point is filled out by the task assignee.

Assignee Planning

Sub Tasks

A full breakdown of the steps to complete this task.

  • Subtasks are above for the various dashboards as they're found and listed

Estimation

Estimate: 5 days
Actual:

Notes

Things that came up during the completion of this task, questions to be answered and follow up tasks.

  • Note

Event Timeline

Make a copy of all dashboards that used to have Graphite backends and create a new version that has Prometheus as the backend source. The new dashboard should be placed to the right of the old one so that the data can be read from left to right.

Is this referring to dashboards or panels? IMHO we should update dashboards in-place rather than copy them – many of them are linked to from various places and we don’t want those links to point to stale Graphite versions forever.

Thanks for the feedback here, @Lucas_Werkmeister_WMDE! I'm thinking in terms of keeping the historical data around, but then a good way forward would maybe be to copy the dashboard and change the original to Prometheus instead of the other way around? We can then put the "new" Graphite one to the left of the Prometheus one for the period when Graphite is read only?

Into In Progress as the Wikidata Alerts dashboard needs to be migrated today :)

And post a Mattermost discussion the thoughts on this are:

  • We can have a "Historical data" section at the end of dashboards
  • This section will have panels that are copies of the original Graphite panels
  • The original Graphite panels will then be switched to Prometheus to assure that all links to them are pointing to the up-to-date data

Moved to In Progress as this is the current priority :)