Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave
Closed, ResolvedPublic
Actions

Description

We will generate on es2002 (and maybe somewhere else) an offline copy of es2 and es3 content (es1 is already at es2003 and es2004) as a one-time copy, in case of a huge disaster. This will unblock pending maintenance work and server until a proper solution is in place for proper automatic incremental backups.

Details

Related Gerrit Patches:

operations/mediawiki-config : master	mariadb: Repool es2019 after maintenance
operations/mediawiki-config : master	mariadb: Repool es2015, depool es2019
operations/mediawiki-config : master	mariadb: Depool es2015 for maintenance

Related Objects
Search...

Status	Assigned	Task
Open	None	T106386 Compress data at external storage
Open	None	T106388 Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in the text table (and fix up if wrong)
Open	None	T106363 Migrate Flow content to new separate logical External Store in production
Open	None	T138049 Dry run of Flow External Store migration in production
Open	Tgr	T107610 Setup separate logical External Store for Flow in production
Resolved	jcrespo	T153440 Create a full backup of all external storage records that would be easy to restore/setup a temporary delayed slave
Open	Daimona	T34478 AbuseFilter not setting utf-8 flag

Event Timeline

jcrespo created this task.Dec 16 2016, 11:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 16 2016, 11:56 AM

jcrespo moved this task from Triage to Next on the DBA board.Dec 16 2016, 11:57 AM

I am going to start doing some tests onto es2002.

Change 423949 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es2015 for maintenance

https://gerrit.wikimedia.org/r/423949

gerritbot added a project: Patch-For-Review.Apr 4 2018, 3:32 PM

Change 423949 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es2015 for maintenance

https://gerrit.wikimedia.org/r/423949

jcrespo updated the task description. (Show Details)Apr 4 2018, 4:01 PM

@ayounsi We are performing this 5.4TB backup by saturating the es2015 and es2002 host links, it will take 12 hours.

Change 424227 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es2015, depool es2019

https://gerrit.wikimedia.org/r/424227

Change 424227 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es2015, depool es2019

https://gerrit.wikimedia.org/r/424227

Mentioned in SAL (#wikimedia-operations) [2018-04-05T08:30:20Z] <jynus> starting backup of es2019, it may create lag T153440

es1 is in es2003/es2003 as a binary copy.
es2 and es3 is in es2002 as logical copies.

Anomie mentioned this in T107610: Setup separate logical External Store for Flow in production.Apr 6 2018, 3:18 PM

Change 424745 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es2019 after maintenance

https://gerrit.wikimedia.org/r/424745

Change 424745 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es2019 after maintenance