Page MenuHomePhabricator

Regularly & Automatically backup WMDE metrics stored in graphite
Open, Stalled, LowPublic

Description

We store most of our numbers in the WMF graphite instance.
As has been mentioned many times if ops need to delete data to get graphite working again they might just do that. (see T85451)
For this reason we should periodically grab the key metrics that we care about and store them somewhere else (Could just be on labs, locally or some machine with some space)...

See https://github.com/wmde/graphite-backup for a script that does everything we need..

#!/bin/bash

php ~/graphite-backup/run.php daily.wikidata.api.wbgetclaims.properties.*
php ~/graphite-backup/run.php daily.wikidata.site_stats.user_groups.*
php ~/graphite-backup/run.php daily.wikidata.site_stats.*
php ~/graphite-backup/run.php daily.wikidata.social.email.*.subscribers
php ~/graphite-backup/run.php daily.wikidata.social.*.*
php ~/graphite-backup/run.php daily.wikidata.entity_usage.*.*

as a start

Event Timeline

Addshore created this task.Feb 1 2016, 5:43 PM
Addshore raised the priority of this task from to High.
Addshore updated the task description. (Show Details)
Addshore added a subscriber: Addshore.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2016, 5:43 PM
Addshore renamed this task from Regularly backup WMDE metrics stored in graphite to Regularly & Automatically backup WMDE metrics stored in graphite.Mar 29 2016, 2:59 PM
Addshore changed the task status from Open to Stalled.Feb 28 2018, 1:08 PM
Addshore lowered the priority of this task from High to Low.

This has clearly not been a high prio, so setting to low & stalled.

Operations is it still the case that data in graphite might be deleted if necessary, so that we should consider creating backups?

Tagging with operations so that we can try to get an answer.

Addshore added a project: Graphite.
Addshore moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Addshore board.
Addshore added a subscriber: fgiunchedi.

@fgiunchedi Any idea if there is any sort of regular / scheduled backups of the disks for graphite nodes?

If not I'll try to put something in place for the few metrics we would like to not loose for now :)

@fgiunchedi Any idea if there is any sort of regular / scheduled backups of the disks for graphite nodes?
If not I'll try to put something in place for the few metrics we would like to not loose for now :)

We're not backing up graphite data because the whole dataset would be too many files. To your previous question re: if deleting data would be a possibility, that is unlikely at this point because the landscape has changed significantly since 2016: Prometheus is the preferred way to interact with metrics and Graphite is considered deprecated. As a result there's much less pressure/demand for disk space in Graphite now, I don't foresee any unplanned data deletion. HTH!