Page MenuHomePhabricator

Certain graphite data directories should be backed up
Closed, ResolvedPublic

Description

Graphite isn't backed up right now (due to the large number of files that change often). It probably should be, though, or at least certain paths should.

  • /var/lib/carbon/whisper/coal/*.wsp

Event Timeline

Dzahn triaged this task as Medium priority.May 11 2018, 3:01 AM

Change 432547 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] graphite: add backup::host and backup::set

https://gerrit.wikimedia.org/r/432547

Change 432547 merged by Dzahn:
[operations/puppet@production] graphite: add backup of /var/lib/carbon/whisper/coal/

https://gerrit.wikimedia.org/r/432547

Change 432610 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] bacula: remove unused fileset var-lib-carbon-whisper

https://gerrit.wikimedia.org/r/432610

Change 432610 merged by Dzahn:
[operations/puppet@production] bacula: remove unused fileset var-lib-carbon-whisper

https://gerrit.wikimedia.org/r/432610

I went to confirm on Bacula that the backups exist:

  • ssh helium.eqiad.wmnet
  • sudo bconsole
  • restore
  • 5: Select the most recent backup for a client
  • 44: graphite1001.eqiad.wmnet-fd
Select the Client (1-126): 44
Automatically selected FileSet: var-lib-carbon-whisper
No Full backup before 2018-05-11 22:16:02 found.
*

So it's there and the file set is there.. just didn't run the full backup yet. Let's check again in a little while and once we actually see the files we can close this.

[helium:~] $ sudo bconsole
Connecting to Director helium.eqiad.wmnet:9101
..
When at a prompt, entering a period cancels the command.

You have messages.
*run
Automatically selected Catalog: production
..
A job name must be specified.
..
    39: graphite1001.eqiad.wmnet-Monthly-1st-Thu-production-var-lib-carbon-whisper-coal
    40: graphite1003.eqiad.wmnet-Monthly-1st-Sun-production-var-lib-carbon-whisper-coal
    41: graphite2001.codfw.wmnet-Monthly-1st-Fri-production-var-lib-carbon-whisper-coal
    42: graphite2002.codfw.wmnet-Monthly-1st-Thu-production-var-lib-carbon-whisper-coal
..
Select Job resource (1-99): 39
Run Backup job
JobName:  graphite1001.eqiad.wmnet-Monthly-1st-Thu-production-var-lib-carbon-whisper-coal
Level:    Incremental
Client:   graphite1001.eqiad.wmnet-fd
FileSet:  var-lib-carbon-whisper-coal
Pool:     production (From Job resource)
Storage:  helium-FileStorage1 (From Pool resource)
When:     2018-05-12 01:00:45
Priority: 10
OK to run? (yes/mod/no): yes
Job queued. JobId=92584
When at a prompt, entering a period cancels the command.

*status
Status available for:
     1: Director
     2: Storage
     3: Client
     4: Scheduled
     5: All
3
The defined Client resources are:
..
    33: graphite1001.eqiad.wmnet-fd
..
 JobId  Level    Files      Bytes   Status   Finished        Name 
======================================================================
 92584  Full         29    176.7 M  OK       12-May-18 01:03 graphite1001.eqiad.wmnet-Monthly-1st-Thu-production-var-lib-carbon-whisper-coal
====
..
*restore
..
Automatically selected FileSet: var-lib-carbon-whisper-coal
..
Building directory tree for JobId(s) 92584 ...  
28 files inserted into the tree.
..

You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored.
.
$ cd coal/ 
cwd is: /var/lib/carbon/whisper/coal/
$ ls
connectEnd.wsp
connectStart.wsp
dnsLookup.wsp
...
Vvjjkkii renamed this task from Certain graphite data directories should be backed up to t6caaaaaaa.Jul 1 2018, 1:10 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed Dzahn as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii edited subscribers, added: Dzahn; removed: gerritbot, Aklapper.
CommunityTechBot renamed this task from t6caaaaaaa to Certain graphite data directories should be backed up.Jul 2 2018, 5:53 AM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to Dzahn.
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot edited subscribers, added: gerritbot, Aklapper; removed: Dzahn.