Page MenuHomePhabricator

Finish dbstore2002 migration to multi-instance
Closed, ResolvedPublic

Description

After dbstore2002 has been added more than one instance (x1 and s2) - complete it with as many instances as possible using GTID (probably 7 will not fit, but we could have 4 or 5), and start using GTID on them.

This will validate the multi-instance beyond just 2 hosts.

Also delete the multi-source main instance it still holds.

  • s1
  • s2
  • s3
  • s4
  • x1
  • ...

Details

Related Gerrit Patches:
operations/mediawiki-config : masterdb-codfw.php: Depool db2074
operations/software : masters3.hosts: dbstore2002 is now replicating s3
operations/puppet : productionmariadb: dbstore2002 has now 5 shards replicating
operations/puppet : productionmariadb: Add s3 to dbstore2002
operations/mediawiki-config : masterdb-codfw.php: Depool db2073
operations/software : masters4.hosts: Add dbstore2002
operations/puppet : productionmariadb: Add instance s4 to dbstore2002
operations/software : masterdblists: Update manual database lists for dbstore2002 changes
operations/puppet : productionAdd s1 instance to dbstore2002 (imported from db2072)

Event Timeline

jcrespo created this task.Jul 21 2017, 4:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2017, 4:21 PM
jcrespo updated the task description. (Show Details)Jul 21 2017, 4:21 PM

Change 366865 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Add s1 instance to dbstore2002 (imported from db2072)

https://gerrit.wikimedia.org/r/366865

Mentioned in SAL (#wikimedia-operations) [2017-07-21T17:02:27Z] <jynus> now that db2072 is compressed and fixed, stop it to finally clone it to dbstore2002 T171321

Change 366865 merged by Jcrespo:
[operations/puppet@production] Add s1 instance to dbstore2002 (imported from db2072)

https://gerrit.wikimedia.org/r/366865

Once all the possible shards that fit in this host are done, we better copy them to dbstore2001 as part of: T168409 - see this for context: T168354#3464488

Change 367358 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/software@master] dblists: Updates to manual database lists for dbstore2002 changes

https://gerrit.wikimedia.org/r/367358

Change 367358 merged by Jcrespo:
[operations/software@master] dblists: Update manual database lists for dbstore2002 changes

https://gerrit.wikimedia.org/r/367358

Marostegui moved this task from Triage to In progress on the DBA board.Jul 24 2017, 1:57 PM
Marostegui added a comment.EditedAug 3 2017, 2:36 PM

I am planning to either s3, s4 or s5 to dbstore2002
Current state of dbstore2002

root@dbstore2002:~# df -hT /srv/
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   6.6T  1.5T  5.1T  23% /srv

We will see which 2 would fit together

Change 370155 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2073

https://gerrit.wikimedia.org/r/370155

Change 370155 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2073

https://gerrit.wikimedia.org/r/370155

Mentioned in SAL (#wikimedia-operations) [2017-08-04T07:47:10Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2073 - T171321 (duration: 00m 47s)

Mentioned in SAL (#wikimedia-operations) [2017-08-04T07:47:22Z] <marostegui> Stop MySQL on db2073 to copy its data to dbstore2002 - https://phabricator.wikimedia.org/T171321

Change 370156 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add instance s4 to dbstore2002

https://gerrit.wikimedia.org/r/370156

Change 370156 merged by Marostegui:
[operations/puppet@production] mariadb: Add instance s4 to dbstore2002

https://gerrit.wikimedia.org/r/370156

s4 is now replicating with gtid

Marostegui updated the task description. (Show Details)Aug 4 2017, 9:17 AM

Change 370169 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s4.hosts: Add dbstore2002

https://gerrit.wikimedia.org/r/370169

Change 370169 merged by jenkins-bot:
[operations/software@master] s4.hosts: Add dbstore2002

https://gerrit.wikimedia.org/r/370169

Mentioned in SAL (#wikimedia-operations) [2017-08-04T09:22:27Z] <marostegui> Add dbstore2002 to tendril - T171321

Mentioned in SAL (#wikimedia-operations) [2017-08-07T06:27:09Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2073 - T171321 (duration: 00m 47s)

Change 370440 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2074

https://gerrit.wikimedia.org/r/370440

Change 370440 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2074

https://gerrit.wikimedia.org/r/370440

Mentioned in SAL (#wikimedia-operations) [2017-08-07T06:37:39Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Depool db2074 - T171321 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-08-07T06:38:26Z] <marostegui> Stop MySQL on db2074 - T171321

Change 370441 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Add s3 to dbstore2002

https://gerrit.wikimedia.org/r/370441

Change 370441 merged by Marostegui:
[operations/puppet@production] mariadb: Add s3 to dbstore2002

https://gerrit.wikimedia.org/r/370441

Marostegui updated the task description. (Show Details)Aug 7 2017, 9:49 AM

Change 370448 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s3.hosts: dbstore2002 is now replicating s3

https://gerrit.wikimedia.org/r/370448

Change 370449 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: dbstore2002 has now 5 shards replicating

https://gerrit.wikimedia.org/r/370449

Change 370449 merged by Marostegui:
[operations/puppet@production] mariadb: dbstore2002 has now 5 shards replicating

https://gerrit.wikimedia.org/r/370449

Change 370448 merged by jenkins-bot:
[operations/software@master] s3.hosts: dbstore2002 is now replicating s3

https://gerrit.wikimedia.org/r/370448

Mentioned in SAL (#wikimedia-operations) [2017-08-07T10:02:14Z] <marostegui> Add dbstore2002:3313 to tendril - T171321

As we spoke, probably it is a good idea to leave dbstore2002 with the current 5 shards, so it has room to grow and we'd not have to revisit it in the next 6 months :-)

root@dbstore2002:~# df -hT /srv
Filesystem            Type  Size  Used Avail Use% Mounted on
/dev/mapper/tank-data xfs   6.6T  4.0T  2.6T  61% /srv

So I believe we can consider this resolved?

Marostegui closed this task as Resolved.Aug 7 2017, 12:16 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-07T12:26:23Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Repool db2074 - T171321 (duration: 00m 45s)