Page MenuHomePhabricator

Improve database master switchover script
Open, LowPublic

Description

The master switchover script worked nicely for T197073, but it needs some extra features:

  • Have it fully documented on wikitech so it can be used by any op
  • have a dry-run feature where it goes over each step, but doesn't really change anything
  • Provide timestamps for each step executed
    • Total read_only time on a MySQL level
  • More pre-flight checks such as
    • is pt-heartbeat running on the current master?
  • Make heartbeat migration more robust until it is migrated to a systemd service or moved remotely (so it is automatic and etcd-dependent)
  • It alters or checks some master-related variables automatically (pt-config-diff h=localhost /etc/my.cnf ?):
    • Alter expire_log_days variable
    • Alter gtid mode automatically
    • Alter semi-sync automatically

Event Timeline

jcrespo moved this task from Triage to Backlog on the DBA board.

Stevie: I believe you have implemented bullet point #5 at https://gerrit.wikimedia.org/r/665324 (thank you!).

Check the other things that were pending (some years ago) just FYI and feel free to take ownership of this, or move it elsewhere (e.g. to a TODO on Wikitech or wmfmariadbpy) or close it, etc. as you better organize it as you want. But I think it is a nice read of "things we wanted improved years ago but never had the time". CC @LSobanski as our "task wizard"