Page MenuHomePhabricator
Create Task
Maniphest T231403

Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC
Closed, ResolvedPublic
Actions

Description

db1063, the current m1 master is having broken disk quite often already (1 per week) and Chris has let us know that there are two more about to fail (T231199#5443385)
We need to find a date to swap this host with db1135.

m1 currently holds the following active databases:

bacula
etherpadlite
librenms
puppet (not in use - to be deleted)
racktables
rt

We don't have much dates without switchovers scheduled already, so I am proposing the following day:
Tuesday 10th September at 16:00 UTC
@akosiaris @Dzahn @ayounsi you've helped in the past to verify/restart services affected by this failover (mostly restarting services).
The failover shouldn't take longer than a few seconds (it is a matter of restarting an HAProxy to point to the new host).

Details

SubjectRepoBranchLines +/-
mariadb: Promote db1135 to m1 masteroperations/puppetproduction+8 -8
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Aug 28 2019, 5:15 AM
Restricted Application added a subscriber: • OldUser02. · View Herald TranscriptAug 28 2019, 5:15 AM
Marostegui triaged this task as Medium priority.Aug 28 2019, 5:15 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.

@akosiaris @Dzahn @jcrespo @ayounsi let me know if that proposed day and time would work for you.
Thanks!

• ayounsi added a comment.Aug 28 2019, 5:34 AM

👍

Dzahn added a comment.Aug 28 2019, 10:37 AM

Fine with me, speaking for RT and racktables. Note that RT is separate from OTRS which is more critical.

Marostegui added a comment.Aug 28 2019, 10:38 AM
In T231403#5445157, @Dzahn wrote:

Fine with me, speaking for RT and racktables. Note that RT is separate from OTRS which is more critical.

Correct! My bad, sorry! Amending...
OTRs is in m2 :)

Marostegui removed a project: Znuny.Aug 28 2019, 10:38 AM
Marostegui updated the task description. (Show Details)
Marostegui removed subscribers: Krenair, • OldUser02.
akosiaris added a comment.Aug 29 2019, 10:47 AM

Sep 10 16:00UTC sounds okish to me as far as bacula goes. We will probably have a couple of full backups (it's the start of the month, when full backups happen and they might not have finished by then) failing, but we can pick them out from the list, and reschedule them manually.

Note that the puppet db can be dropped now that servermon has been killed.

Marostegui added a comment.Aug 29 2019, 10:50 AM

Thanks @akosiaris!
As spoken on IRC, no need to re-schedule because of bacula.

Marostegui renamed this task from Switchover m1 primary master: db1063 to db1135 to Switchover m1 primary master: db1063 to db1135: Tuesday 10th September at 16:00 UTC.Aug 29 2019, 10:54 AM
Marostegui mentioned this in T231539: Drop puppet database from m1.Aug 29 2019, 11:04 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.Sep 3 2019, 2:56 PM
gerritbot added a comment.Sep 4 2019, 8:19 AM

Change 534386 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1135 to m1 master

https://gerrit.wikimedia.org/r/534386

gerritbot added a project: Patch-For-Review.Sep 4 2019, 8:19 AM
Stashbot added a comment.Sep 4 2019, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-04T08:26:06Z] <marostegui> Reboot db1135 to pick up new kernel - T231403

Trizek-WMF added a project: User-notice.Sep 4 2019, 9:06 AM
Trizek-WMF subscribed.

Added for Tech News, since Etherpad service is quite used, and 16:00 UTC is a common meetings hour.

Trizek-WMF moved this task from To Triage to In current Tech/News draft on the User-notice board.Sep 4 2019, 9:06 AM
Marostegui added a comment.Sep 4 2019, 9:08 AM
In T231403#5464235, @Trizek-WMF wrote:

Added for Tech News, since Etherpad service is quite used, and 16:00 UTC is a common meetings hour.

Oh - thank you!

Agusbou2015 subscribed.Sep 5 2019, 1:51 PM

Will enwiki the only wiki affected to this failover?

Marostegui added a comment.Sep 5 2019, 1:52 PM
In T231403#5468025, @Agusbou2015 wrote:

Will enwiki the only wiki affected to this failover?

enwiki will not be affected by this failover. No wikis will be affected in fact.

Marostegui added a comment.Sep 6 2019, 5:04 AM

I have reserved the window on the Deployments page.

Marostegui updated the task description. (Show Details)Sep 6 2019, 5:37 AM
Stashbot added a comment.Sep 10 2019, 3:37 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-10T15:37:05Z] <marostegui> Start pre-switchover for m1 steps T231403

gerritbot added a comment.Sep 10 2019, 3:42 PM

Change 534386 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1135 to m1 master

https://gerrit.wikimedia.org/r/534386

Stashbot added a comment.Sep 10 2019, 4:10 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-10T16:10:18Z] <marostegui> Failover m1 from db1063 to db1135 - T231403

Maintenance_bot removed a project: Patch-For-Review.Sep 10 2019, 4:10 PM
Marostegui added a comment.Sep 10 2019, 4:20 PM

This was done.
Read-only starts: Tue Sep 10 16:10:39 UTC 2019
Read-only stops: Tue Sep 10 16:10:45 UTC 2019

Total read-only time: 6 seconds

Marostegui closed this task as Resolved.Sep 10 2019, 4:22 PM

Thanks everyone who helped out, closing this!

Marostegui mentioned this in T232564: Decommission db1063.eqiad.wmnet.Sep 11 2019, 5:13 AM
Johan moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Sep 11 2019, 1:47 PM
Marostegui mentioned this in T244238: Upgrade and restart m1 master (db1135).Feb 4 2020, 2:23 PM
Ladsgroup edited projects, added User-notice-archive; removed User-notice.Aug 13 2022, 1:54 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 13 2022, 2:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits