Page MenuHomePhabricator
Create Task
Maniphest T286042

Move db1124 and db1125 to misc services temporarily
Closed, ResolvedPublic
Actions

Description

Due to a network maintenance on which the entire row A will be affected (T286032), we need to temporarily failover misc services to hosts outside of row A.
We can use db1124 and db1125 (testing hosts) and place them temporarily as m1 and m2 masters.

  • db1125 to replace db1159
  • db1124 to replace db1107

Details

SubjectRepoBranchLines +/-
mariadb: Move db1125 to m1operations/puppetproduction+6 -6
mariadb: Move db1124 to m2.operations/puppetproduction+5 -7
db1124, db1125: Enable notifications.operations/puppetproduction+0 -2
dbproxy1013,dbproxy1015: Add db1124 as failoveroperations/puppetproduction+4 -4
dbproxy1012,dbproxy1014: Add db1125 as m1 failoveroperations/puppetproduction+4 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jul 2 2021, 1:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 2 2021, 1:11 PM
Marostegui triaged this task as High priority.Jul 2 2021, 1:11 PM
Marostegui moved this task from Triage to Ready on the DBA board.
gerritbot added a comment.Jul 5 2021, 12:48 PM

Change 703207 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1125 to m1

https://gerrit.wikimedia.org/r/703207

gerritbot added a project: Patch-For-Review.Jul 5 2021, 12:48 PM

Change 703207 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1125 to m1

https://gerrit.wikimedia.org/r/703207

Stashbot added a comment.Jul 5 2021, 12:50 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-05T12:50:21Z] <marostegui> Stop MySQL on db1117:3321 to clone db1125 T286042

Maintenance_bot removed a project: Patch-For-Review.Jul 5 2021, 1:10 PM
Marostegui updated the task description. (Show Details)Jul 5 2021, 1:24 PM

db1125 is now serving in m1.
Pending: place db1125 into haproxy (let's give it 24h to make sure it runs smoothly)

gerritbot added a comment.Jul 6 2021, 4:42 AM

Change 703269 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1124 to m2.

https://gerrit.wikimedia.org/r/703269

gerritbot added a project: Patch-For-Review.Jul 6 2021, 4:42 AM
gerritbot added a comment.Jul 6 2021, 9:17 AM

Change 703269 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1124 to m2.

https://gerrit.wikimedia.org/r/703269

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2021, 10:10 AM
Marostegui added a comment.Jul 6 2021, 10:52 AM

db1124 is now in m2, leaving it for 24h before adding it to the proxy.

Marostegui updated the task description. (Show Details)Jul 6 2021, 10:53 AM
gerritbot added a comment.Jul 8 2021, 5:02 AM

Change 703634 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1013,dbproxy1015: Add db1124 as failover

https://gerrit.wikimedia.org/r/703634

gerritbot added a project: Patch-For-Review.Jul 8 2021, 5:02 AM
gerritbot added a comment.Jul 8 2021, 5:04 AM

Change 703634 merged by Marostegui:

[operations/puppet@production] dbproxy1013,dbproxy1015: Add db1124 as failover

https://gerrit.wikimedia.org/r/703634

Marostegui added a comment.Jul 8 2021, 5:05 AM

db1124 has been added as m2 failover host. dbproxy1013 and dbproxy1015 reloaded.

gerritbot added a comment.Jul 8 2021, 5:08 AM

Change 703635 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1012,dbproxy1014: Add db1125 as m1 failover

https://gerrit.wikimedia.org/r/703635

gerritbot added a comment.Jul 8 2021, 5:09 AM

Change 703635 merged by Marostegui:

[operations/puppet@production] dbproxy1012,dbproxy1014: Add db1125 as m1 failover

https://gerrit.wikimedia.org/r/703635

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2021, 5:10 AM
Marostegui added a comment.Jul 8 2021, 5:11 AM

db1125 has been added as m1 failover. dbproxy1012, dbproxy1014 reloaded.

gerritbot added a comment.Jul 8 2021, 5:12 AM

Change 703636 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1124, db1125: Enable notifications.

https://gerrit.wikimedia.org/r/703636

gerritbot added a project: Patch-For-Review.Jul 8 2021, 5:12 AM

Change 703636 merged by Marostegui:

[operations/puppet@production] db1124, db1125: Enable notifications.

https://gerrit.wikimedia.org/r/703636

Marostegui closed this task as Resolved.Jul 8 2021, 5:15 AM

Notifications enabled. All done.
Once the maintenance is done, these hosts need to go back to the test section.

Marostegui changed the status of subtask T286329: Move db1124 and db1125 back to test-cluster section from Open to Stalled.Jul 8 2021, 5:20 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 8 2021, 6:10 AM
Marostegui mentioned this in T286032: Switch buffer re-partition - Eqiad Row A.Jul 27 2021, 3:17 PM
Marostegui changed the status of subtask T286329: Move db1124 and db1125 back to test-cluster section from Stalled to Open.Jul 30 2021, 4:54 AM
Marostegui closed subtask T286329: Move db1124 and db1125 back to test-cluster section as Resolved.Aug 2 2021, 10:52 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits