Page MenuHomePhabricator

Physically move es1017 from D to C row
Closed, ResolvedPublic

Description

I made the mistake to chose the same row for 2 servers of the same shard: T105843#1578897

Please help ups solve our mistake by moving es1017 to row C, anywhere possible. db1056 is scheduled for decom, so it should at least fit in its place on top of C3: T193736

Please coordinate with us, this can happen at any time, but we will need to depool the server in advance and reconfigure the new ip for mediawiki. This would be, however, a blocker for T183585 (we will failover the es3 master there)

Event Timeline

jcrespo created this task.Jun 13 2018, 8:54 AM
Restricted Application added a project: Operations. · View Herald TranscriptJun 13 2018, 8:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jcrespo triaged this task as Medium priority.Jun 13 2018, 9:06 AM
Marostegui moved this task from Triage to Next on the DBA board.Jun 13 2018, 9:18 AM
Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Jun 20 2018, 7:35 AM

@Cmjohnson do you have an estimate date more or less when we can do this movement? Asking just to organize ourselves (within DBA)

@Marostegui next week sometime is the best I can do right now.

Sounds good! Just let us know with a day in advance so we can organize it :)
Thanks!

Vvjjkkii renamed this task from Physically move es1017 from D to C row to 34aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
Marostegui renamed this task from 34aaaaaaaa to Physically move es1017 from D to C row.Jul 1 2018, 8:12 PM
Marostegui lowered the priority of this task from High to Medium.
Marostegui updated the task description. (Show Details)

We'll attempt to get this moved tomorrow around this time.

Change 443538 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es1017.yaml: Disable notifications

https://gerrit.wikimedia.org/r/443538

Change 443538 merged by Marostegui:
[operations/puppet@production] es1017.yaml: Disable notifications

https://gerrit.wikimedia.org/r/443538

Change 443550 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es1017 for maintenance

https://gerrit.wikimedia.org/r/443550

Change 443550 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es1017 for maintenance

https://gerrit.wikimedia.org/r/443550

@Cmjohnson this host has been depooled. Let us know the IP in advance so we can change it before rebooting it.
Thanks

Change 443602 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Changing DNS es1017 moving to row C

https://gerrit.wikimedia.org/r/443602

Change 443602 merged by Jcrespo:
[operations/dns@master] Changing DNS es1017 moving to row C

https://gerrit.wikimedia.org/r/443602

Change 443616 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Change es1017 IP and rack

https://gerrit.wikimedia.org/r/443616

Change 443616 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Change es1017 IP and rack

https://gerrit.wikimedia.org/r/443616

Mentioned in SAL (#wikimedia-operations) [2018-07-03T14:08:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Change es1017 IP and rack T197072 (duration: 00m 50s)

The move was successful.
We need to still repool the server once the buffer pool is ready and then this can be considered done

I am going to guess it was not replaced by db1056 (?) so that should still happen, but not a dependency anymore: T193736

Change 443650 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Increase es1017 weight after warmup

https://gerrit.wikimedia.org/r/443650

Change 443650 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Increase es1017 weight after warmup

https://gerrit.wikimedia.org/r/443650

Anything left after repooling the host?

jcrespo closed this task as Resolved.Jul 4 2018, 4:54 AM
jcrespo assigned this task to Cmjohnson.

I don't think so.