Page MenuHomePhabricator
Create Task
Maniphest T197072

Physically move es1017 from D to C row
Closed, ResolvedPublic
Actions

Description

I made the mistake to chose the same row for 2 servers of the same shard: T105843#1578897

Please help ups solve our mistake by moving es1017 to row C, anywhere possible. db1056 is scheduled for decom, so it should at least fit in its place on top of C3: T193736

Please coordinate with us, this can happen at any time, but we will need to depool the server in advance and reconfigure the new ip for mediawiki. This would be, however, a blocker for T183585 (we will failover the es3 master there)

Details

SubjectRepoBranchLines +/-
mariadb: Increase es1017 weight after warmupoperations/mediawiki-configmaster+3 -3
db-eqiad.php: Change es1017 IP and rackoperations/mediawiki-configmaster+1 -1
mariadb: Depool es1017 for maintenanceoperations/mediawiki-configmaster+2 -2
es1017.yaml: Disable notificationsoperations/puppetproduction+1 -0
Changing DNS es1017 moving to row Coperations/dnsmaster+2 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Jun 13 2018, 8:54 AM
Restricted Application added a project: SRE. · View Herald TranscriptJun 13 2018, 8:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
jcrespo added a parent task: T183585: Rack/cable/configure asw2-b-eqiad switch stack.Jun 13 2018, 8:55 AM
jcrespo added a subtask: T193736: Decommission db1056.Jun 13 2018, 9:01 AM
jcrespo edited parent tasks, added: T197073: switchover es1014 to es1017; removed: T183585: Rack/cable/configure asw2-b-eqiad switch stack.Jun 13 2018, 9:04 AM
jcrespo triaged this task as Medium priority.Jun 13 2018, 9:06 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.Jun 13 2018, 9:18 AM
• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Jun 20 2018, 7:35 AM
Marostegui added a comment.Jun 26 2018, 7:45 AM

@Cmjohnson do you have an estimate date more or less when we can do this movement? Asking just to organize ourselves (within DBA)

• Cmjohnson added a comment.Jun 26 2018, 2:32 PM

@Marostegui next week sometime is the best I can do right now.

Marostegui added a comment.Jun 26 2018, 3:00 PM

Sounds good! Just let us know with a day in advance so we can organize it :)
Thanks!

• Vvjjkkii renamed this task from Physically move es1017 from D to C row to 34aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
Marostegui renamed this task from 34aaaaaaaa to Physically move es1017 from D to C row.Jul 1 2018, 8:12 PM
Marostegui lowered the priority of this task from High to Medium.
Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
Marostegui updated the task description. (Show Details)
Marostegui added a comment.Jul 2 2018, 12:45 PM

We'll attempt to get this moved tomorrow around this time.

gerritbot subscribed.Jul 3 2018, 4:44 AM

Change 443538 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] es1017.yaml: Disable notifications

https://gerrit.wikimedia.org/r/443538

gerritbot added a project: Patch-For-Review.Jul 3 2018, 4:44 AM

Change 443538 merged by Marostegui:
[operations/puppet@production] es1017.yaml: Disable notifications

https://gerrit.wikimedia.org/r/443538

gerritbot added a comment.Jul 3 2018, 6:50 AM

Change 443550 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es1017 for maintenance

https://gerrit.wikimedia.org/r/443550

gerritbot added a comment.Jul 3 2018, 6:54 AM

Change 443550 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es1017 for maintenance

https://gerrit.wikimedia.org/r/443550

Marostegui added a comment.Jul 3 2018, 7:53 AM

@Cmjohnson this host has been depooled. Let us know the IP in advance so we can change it before rebooting it.
Thanks

gerritbot added a comment.Jul 3 2018, 12:21 PM

Change 443602 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Changing DNS es1017 moving to row C

https://gerrit.wikimedia.org/r/443602

gerritbot added a comment.Jul 3 2018, 12:55 PM

Change 443602 merged by Jcrespo:
[operations/dns@master] Changing DNS es1017 moving to row C

https://gerrit.wikimedia.org/r/443602

gerritbot added a comment.Jul 3 2018, 1:21 PM

Change 443616 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Change es1017 IP and rack

https://gerrit.wikimedia.org/r/443616

gerritbot added a comment.Jul 3 2018, 2:07 PM

Change 443616 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Change es1017 IP and rack

https://gerrit.wikimedia.org/r/443616

Stashbot subscribed.Jul 3 2018, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2018-07-03T14:08:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Change es1017 IP and rack T197072 (duration: 00m 50s)

Marostegui added a comment.Jul 3 2018, 2:10 PM

The move was successful.
We need to still repool the server once the buffer pool is ready and then this can be considered done

jcrespo added a comment.Jul 3 2018, 3:06 PM

I am going to guess it was not replaced by db1056 (?) so that should still happen, but not a dependency anymore: T193736

jcrespo removed a subtask: T193736: Decommission db1056.Jul 3 2018, 4:29 PM
gerritbot added a comment.Jul 3 2018, 4:30 PM

Change 443650 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Increase es1017 weight after warmup

https://gerrit.wikimedia.org/r/443650

gerritbot added a comment.Jul 3 2018, 4:41 PM

Change 443650 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Increase es1017 weight after warmup

https://gerrit.wikimedia.org/r/443650

Marostegui added a comment.Jul 4 2018, 4:50 AM

Anything left after repooling the host?

jcrespo closed this task as Resolved.Jul 4 2018, 4:54 AM
jcrespo assigned this task to • Cmjohnson.

I don't think so.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits