Page MenuHomePhabricator

Reclone db1068 to become master in s4
Closed, ResolvedPublic

Description

db1068 will replace current s4 master as part of: T162133
It needs to be recloned.
It is running Debian Jessie 8.4 and 10.0.23

I would reclone it from one of the large servers to be sure replication doesn't break.

If we are going to reclone it from the large servers, I would strongly recommend db1081 (10.0.23 same as db1068) as it has been migrated to file per table (T161088) and the others are still not running file per table.

Event Timeline

As per T162593#3175313 db1081 looks healthy to be used to clone db1068 and it would have the same data as the current master.

Change 349218 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1081 and db1068

https://gerrit.wikimedia.org/r/349218

Change 349218 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1081 and db1068

https://gerrit.wikimedia.org/r/349218

Mentioned in SAL (#wikimedia-operations) [2017-04-20T13:58:04Z] <marostegui> Stop MySQL on db1068 and db1081 for maintenance - T163110

Change 349220 had a related patch set uploaded (by Jcrespo):
[operations/puppet@production] mariadb: promote db1064 as the new s4 master on eqiad

https://gerrit.wikimedia.org/r/349220

Change 349220 merged by Jcrespo:
[operations/puppet@production] mariadb: promote db1068 as the new s4 master on eqiad

https://gerrit.wikimedia.org/r/349220

Server is recloned.
Catching up
SSL enabled
GTID using: slave_pos

Change 349238 had a related patch set uploaded (by Jcrespo):
[operations/puppet@production] prometheus-myqsld-exporter: Promote db1068 to the s4 master

https://gerrit.wikimedia.org/r/349238

Change 349240 had a related patch set uploaded (by Jcrespo):
[operations/software@master] s4.hosts: set db1068 is the new s4 master

https://gerrit.wikimedia.org/r/349240

Change 349240 merged by Jcrespo:
[operations/software@master] s4.hosts: set db1068 is the new s4 master

https://gerrit.wikimedia.org/r/349240

db1068 is now serving as a master see: T162133

Marostegui renamed this task from Reclone db1068 to become a slave in s4 to Reclone db1068 to become master in s4.Apr 20 2017, 6:24 PM
Marostegui assigned this task to jcrespo.
Marostegui updated the task description. (Show Details)

Change 349238 merged by Jcrespo:
[operations/puppet@production] prometheus-myqsld-exporter: Promote db1068 to the s4 master

https://gerrit.wikimedia.org/r/349238

For the record, I ran:

./repl.pl --switch-child-to-sibling --parent=db1040.eqiad.wmnet --child=dbstore1002.eqiad.wmnet --child-set="default_master_connection='s4'"
/repl.pl --switch-child-to-sibling --parent=db1040.eqiad.wmnet --child=db1069.eqiad.wmnet:3314

dbstore1001 is another story because it is delated and it can start replication without manual intervention, so using this will not work and it would corrupt its data anyway.