Page MenuHomePhabricator
Create Task
Maniphest T272964

Restart m2 database master (db1107)
Closed, ResolvedPublic
Actions

Description

In order to be able to enable report_host variable on db1107 we'd need to issue a mariadb daemon restart for db1107.
We expect this to take around 1 minute.

Affected databases:

[x] debmonitor
[x] mwaddlink
[x] otrs
[x] recommendationapi
[x] sockpuppet
[x] xhgui

When: Wednesday 3th Feb at 09:00AM UTC
Impact: All these databases will be on read-only for around 1 minute.

Related Objects
Search...

Event Timeline

Marostegui created this task.Jan 26 2021, 10:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 26 2021, 10:54 AM
Marostegui triaged this task as Medium priority.Jan 26 2021, 10:57 AM
Marostegui moved this task from Triage to Ready on the DBA board.
Marostegui added a project: Research.
Marostegui added subscribers: leila, Krenair.

@akosiaris @hnowlan @MoritzMuehlenhoff @kostajh Performance-Team Research I would like to propose Wednesday 5th Feb at 09:00AM UTC to restart the above host.
Would that be ok?

hnowlan added a comment.Jan 26 2021, 10:59 AM

No objections from us on Sockpuppet.

kostajh added a comment.Jan 26 2021, 11:03 AM
In T272964#6776773, @Marostegui wrote:

@akosiaris @hnowlan @MoritzMuehlenhoff @kostajh Performance-Team Research I would like to propose Wednesday 5th Feb at 09:00AM UTC to restart the above host.
Would that be ok?

sounds good, thanks!

akosiaris added a comment.Jan 26 2021, 11:03 AM

Fine by me, I 'll keep an eye on OTRS and recommendation api.

MoritzMuehlenhoff added a comment.Jan 26 2021, 11:08 AM
In T272964#6776773, @Marostegui wrote:

@akosiaris @hnowlan @MoritzMuehlenhoff @kostajh Performance-Team Research I would like to propose Wednesday 5th Feb at 09:00AM UTC to restart the above host.
Would that be ok?

Sounds good, I'll be around.

Marostegui added a subscriber: • dpifke.EditedJan 26 2021, 11:11 AM

Thank you guys for the fast responses!
Going to schedule it for Wednesday 3th Feb at 09:00AM UTC then - @dpifke if this doesn't work for xhgui let me know!

Marostegui updated the task description. (Show Details)Jan 26 2021, 11:12 AM
Marostegui updated the task description. (Show Details)

I checked the 2020 calendar (I wonder why.....), Wednesday is 3rd of Feb, not 5th :)

Marostegui updated the task description. (Show Details)Jan 26 2021, 11:16 AM
Marostegui added a comment.Jan 26 2021, 11:18 AM

Maintenance window booked on the deployment calendar

Krinkle updated the task description. (Show Details)Jan 26 2021, 3:20 PM
• dpifke updated the task description. (Show Details)Jan 26 2021, 3:58 PM

No objection here.

• dpifke moved this task from Inbox, needs triage to Radar on the Performance-Team board.Jan 26 2021, 4:00 PM
• dpifke edited projects, added Performance-Team (Radar); removed Performance-Team.
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Jan 26 2021, 5:05 PM
Marostegui added a parent task: T272568: Add m* and es4/es5 sections to Orchestrator.Jan 28 2021, 9:40 AM
Marostegui added a comment.EditedJan 28 2021, 11:46 AM

Procedure:

Pre restart

  • Silence m2 hosts
  • buffer pool dump + disablement in advance to make the restart faster

Restart

  • !log m2 master restart - T272964
  • db1107: restart mysql
  • verify report_host is enabled
  • verify read_only is OFF
  • Once mysql is back: reload haproxy on dbproxy1013 (active) and dbproxy1015 (passive)
  • check everything is ok
  • close this task
Marostegui moved this task from Ready to In progress on the DBA board.Feb 1 2021, 11:20 AM
LSobanski subscribed.Feb 1 2021, 5:26 PM
Marostegui added a comment.Feb 3 2021, 8:19 AM

pre-restart steps are done

Marostegui added a comment.Feb 3 2021, 9:07 AM

This has been delayed a little bit as there's an unrelated incident going on.

Stashbot added a comment.Feb 3 2021, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2021-02-03T09:57:53Z] <marostegui> m2 master restart - T272964

Marostegui closed this task as Resolved.Feb 3 2021, 10:03 AM

This was done.
RO start: 09:58:05
RO stop: 09:58:48

All the services recovered without human intervention.

akosiaris awarded a token.Feb 3 2021, 2:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits