Page MenuHomePhabricator

Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave
Closed, ResolvedPublic

Description

We will generate on es2002 (and maybe somewhere else) an offline copy of es2 and es3 content (es1 is already at es2003 and es2004) as a one-time copy, in case of a huge disaster. This will unblock pending maintenance work and server until a proper solution is in place for proper automatic incremental backups.

Event Timeline

jcrespo created this task.Dec 16 2016, 11:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2016, 11:56 AM
jcrespo moved this task from Triage to Next on the DBA board.Dec 16 2016, 11:57 AM
jcrespo claimed this task.Apr 4 2018, 3:21 PM
jcrespo moved this task from Next to In progress on the DBA board.

I am going to start doing some tests onto es2002.

Change 423949 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es2015 for maintenance

https://gerrit.wikimedia.org/r/423949

Change 423949 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es2015 for maintenance

https://gerrit.wikimedia.org/r/423949

jcrespo updated the task description. (Show Details)Apr 4 2018, 4:01 PM
jcrespo added a subscriber: ayounsi.Apr 4 2018, 4:25 PM

@ayounsi We are performing this 5.4TB backup by saturating the es2015 and es2002 host links, it will take 12 hours.

Change 424227 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es2015, depool es2019

https://gerrit.wikimedia.org/r/424227

Change 424227 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es2015, depool es2019

https://gerrit.wikimedia.org/r/424227

Mentioned in SAL (#wikimedia-operations) [2018-04-05T08:30:20Z] <jynus> starting backup of es2019, it may create lag T153440

jcrespo closed this task as Resolved.Apr 6 2018, 5:04 AM

es1 is in es2003/es2003 as a binary copy.
es2 and es3 is in es2002 as logical copies.

Change 424745 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/424745

Change 424745 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/424745