Page MenuHomePhabricator

Setup backups for es6, es7 and archive old read only backups
Open, In Progress, Needs TriagePublic

Description

Setup regular backups of new active read-write hosts, and once the old ones are in read-only mode, send them to the backup archival, where they are not refreshed regularly.

Steps:

  • Setup new config (including grants)
  • Add new sections to monitoring (icinga, pampinus)
  • Remove old one
  • Archival of old section data

Event Timeline

Change #1025714 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] [WIP]dbbackups: Add backups for es6 and es7

https://gerrit.wikimedia.org/r/1025714

jcrespo changed the task status from Open to In Progress.Tue, Apr 30, 10:42 AM
jcrespo updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2024-05-09T08:53:41Z] <jynus> deploy new grants for es6, es7 backups T363812

Change #1025714 merged by Jcrespo:

[operations/puppet@production] dbbackups: Add backups for es6 and es7

https://gerrit.wikimedia.org/r/1025714

@Marostegui es6 and es7 backups are enabled, and a first run was done here. They seem mostly empty, though:

image.png (558×2 px, 141 KB)

Thanks Jaime - yeah, those hosts only have the table schemas.

Change #1031387 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Start monitoring es6, es7 for regular backups produced

https://gerrit.wikimedia.org/r/1031387

Change #1031387 merged by Jcrespo:

[operations/puppet@production] dbbackups: Start monitoring es6, es7 for regular backups produced

https://gerrit.wikimedia.org/r/1031387

Change #1031397 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Update the list of valid sections to check for WMFbackups

https://gerrit.wikimedia.org/r/1031397

Change #1031397 merged by Jcrespo:

[operations/puppet@production] dbbackups: Update the list of valid sections to check for WMFbackups

https://gerrit.wikimedia.org/r/1031397

TODO:

  • Stop es4 and es5 backups
  • Generate a full clusterX and clusterY last backup
  • Archive it into long term backups
  • Remove dump user