Regularly & Automatically backup WMDE metrics stored in graphite
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	Addshore
	Feb 1 2016, 5:43 PM

Description

We store most of our numbers in the WMF graphite instance.
As has been mentioned many times if ops need to delete data to get graphite working again they might just do that. (see T85451)
For this reason we should periodically grab the key metrics that we care about and store them somewhere else (Could just be on labs, locally or some machine with some space)...

See https://github.com/wmde/graphite-backup for a script that does everything we need..

#!/bin/bash

php ~/graphite-backup/run.php daily.wikidata.api.wbgetclaims.properties.*
php ~/graphite-backup/run.php daily.wikidata.site_stats.user_groups.*
php ~/graphite-backup/run.php daily.wikidata.site_stats.*
php ~/graphite-backup/run.php daily.wikidata.social.email.*.subscribers
php ~/graphite-backup/run.php daily.wikidata.social.*.*
php ~/graphite-backup/run.php daily.wikidata.entity_usage.*.*

as a start

Related Objects

Mentioned Here: T85451: scale graphite deployment (tracking)

Event Timeline

Addshore created this task.Feb 1 2016, 5:43 PM

Addshore raised the priority of this task from to High.

Addshore updated the task description. (Show Details)

Addshore added a project: WMDE-Analytics-Engineering.

Addshore subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2016, 5:43 PM

Addshore moved this task from Incoming to ToDo on the WMDE-Analytics-Engineering board.Feb 1 2016, 5:43 PM

Addshore renamed this task from Regularly backup WMDE metrics stored in graphite to Regularly & Automatically backup WMDE metrics stored in graphite.Mar 29 2016, 2:59 PM

This has clearly not been a high prio, so setting to low & stalled.

GoranSMilovanovic subscribed.Feb 28 2018, 1:10 PM

SRE is it still the case that data in graphite might be deleted if necessary, so that we should consider creating backups?

Tagging with operations so that we can try to get an answer.

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:30 PM

Addshore moved this task from ToDo to Watching / External on the WMDE-Analytics-Engineering board.Jan 29 2019, 9:14 AM

Addshore added a project: User-Addshore.Dec 20 2019, 8:52 AM

@fgiunchedi Any idea if there is any sort of regular / scheduled backups of the disks for graphite nodes?

If not I'll try to put something in place for the few metrics we would like to not loose for now :)

In T125408#5811525, @Addshore wrote:

@fgiunchedi Any idea if there is any sort of regular / scheduled backups of the disks for graphite nodes?

If not I'll try to put something in place for the few metrics we would like to not loose for now :)

We're not backing up graphite data because the whole dataset would be too many files. To your previous question re: if deleting data would be a possibility, that is unlikely at this point because the landscape has changed significantly since 2016: Prometheus is the preferred way to interact with metrics and Graphite is considered deprecated. As a result there's much less pressure/demand for disk space in Graphite now, I don't foresee any unplanned data deletion. HTH!

Addshore closed this task as Declined.Jul 29 2020, 7:49 AM

Regularly & Automatically backup WMDE metrics stored in graphiteClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

Regularly & Automatically backup WMDE metrics stored in graphite
Closed, DeclinedPublic
Actions