Page MenuHomePhabricator
Create Task
Maniphest T254556

Upgrade m1 to Buster and Mariadb 10.4
Closed, ResolvedPublic
Actions

Description

I want to have m1 upgraded to Buster and Mariadb 10.4

m1 has:

  • db1135 master
  • Provision db1097 as future master.
  • db1117 slave (across all miscs)
  • db2132
  • db2078 (already migrated)

The idea is to grab one of the hosts at T253217 and migrate one of these as they have smaller disk (but still enough for misc) and keep the current master as a rollback plan just in case.
Once migrated, move the "old" m1 master to the list of pooleables hosts and move those to sXX as they have more disk space (4.4T vs 3.2T)

Details

SubjectRepoBranchLines +/-
switchover.py: Change zarcillo instanceoperations/software/wmfmariadbpymaster+1 -1
mariadb: Promote db1097 to m1 masteroperations/puppetproduction+6 -7
dbproxy1012,1014: Place db1097 as standby host.operations/puppetproduction+4 -4
db1117: Enable notificationsoperations/puppetproduction+0 -1
db2132: Enable notificationsoperations/puppetproduction+0 -1
db2132: Disable notificationsoperations/puppetproduction+1 -0
install_server: Reimage db2132 with busteroperations/puppetproduction+0 -1
db1097: Enable notificationsoperations/puppetproduction+1 -2
mariadb: Move db1097 to m1operations/puppetproduction+11 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jun 5 2020, 8:39 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 5 2020, 8:39 AM
Marostegui triaged this task as Medium priority.Jun 5 2020, 8:40 AM
Marostegui moved this task from Triage to Pending comment on the DBA board.
Marostegui updated the task description. (Show Details)Jun 9 2020, 7:17 AM
gerritbot added a comment.Jun 9 2020, 7:17 AM

Change 603871 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1097 to m1

https://gerrit.wikimedia.org/r/603871

gerritbot added a project: Patch-For-Review.Jun 9 2020, 7:17 AM
gerritbot added a comment.Jun 9 2020, 7:19 AM

Change 603871 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1097 to m1

https://gerrit.wikimedia.org/r/603871

ops-monitoring-bot added a comment.Jun 9 2020, 7:22 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1097.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006090722_marostegui_243699.log.

ops-monitoring-bot added a comment.Jun 9 2020, 7:40 AM

Completed auto-reimage of hosts:

['db1097.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jun 9 2020, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-09T08:01:43Z] <marostegui> stop m1 on db1117 to clone db1097 (this will trigger an haproxy irc alert) - T254556

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 8:10 AM
Marostegui updated the task description. (Show Details)Jun 9 2020, 8:46 AM
gerritbot added a comment.Jun 9 2020, 8:46 AM

Change 603900 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1097: Enable notifications

https://gerrit.wikimedia.org/r/603900

gerritbot added a project: Patch-For-Review.Jun 9 2020, 8:46 AM
Marostegui mentioned this in T253217: Relocate "old" s4 hosts.Jun 9 2020, 8:46 AM
gerritbot added a comment.Jun 9 2020, 8:47 AM

Change 603900 merged by Marostegui:
[operations/puppet@production] db1097: Enable notifications

https://gerrit.wikimedia.org/r/603900

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 9:10 AM
Marostegui moved this task from Pending comment to In progress on the DBA board.Jun 9 2020, 10:40 AM
gerritbot added a comment.Jun 9 2020, 11:42 AM

Change 603978 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Reimage db2132 with buster

https://gerrit.wikimedia.org/r/603978

gerritbot added a project: Patch-For-Review.Jun 9 2020, 11:42 AM

Change 603978 merged by Marostegui:
[operations/puppet@production] install_server: Reimage db2132 with buster

https://gerrit.wikimedia.org/r/603978

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 12:10 PM
gerritbot added a comment.Jun 19 2020, 6:18 AM

Change 606558 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2132: Disable notifications

https://gerrit.wikimedia.org/r/606558

gerritbot added a project: Patch-For-Review.Jun 19 2020, 6:18 AM

Change 606558 merged by Marostegui:
[operations/puppet@production] db2132: Disable notifications

https://gerrit.wikimedia.org/r/606558

Stashbot added a comment.Jun 19 2020, 6:20 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-19T06:19:59Z] <marostegui> Stop mysql on db2132 to reimage m1 codfw master - T254556

ops-monitoring-bot added a comment.Jun 19 2020, 6:20 AM

Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts:

['db2132.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006190620_marostegui_16910.log.

ops-monitoring-bot added a comment.Jun 19 2020, 6:44 AM

Completed auto-reimage of hosts:

['db2132.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Jun 19 2020, 6:50 AM
gerritbot added a comment.Jun 19 2020, 6:51 AM

Change 606634 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2132: Enable notifications

https://gerrit.wikimedia.org/r/606634

gerritbot added a comment.Jun 19 2020, 6:51 AM

Change 606634 merged by Marostegui:
[operations/puppet@production] db2132: Enable notifications

https://gerrit.wikimedia.org/r/606634

Maintenance_bot removed a project: Patch-For-Review.Jun 19 2020, 7:10 AM
ops-monitoring-bot added a comment.Jun 22 2020, 7:19 AM

Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts:

['db1117.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006220719_marostegui_6426.log.

ops-monitoring-bot added a comment.Jun 22 2020, 7:36 AM

Completed auto-reimage of hosts:

['db1117.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jun 22 2020, 7:41 AM

Change 606950 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1117: Enable notifications

https://gerrit.wikimedia.org/r/606950

gerritbot added a project: Patch-For-Review.Jun 22 2020, 7:41 AM
gerritbot added a comment.Jun 22 2020, 7:47 AM

Change 606950 merged by Marostegui:
[operations/puppet@production] db1117: Enable notifications

https://gerrit.wikimedia.org/r/606950

Marostegui updated the task description. (Show Details)Jun 22 2020, 7:47 AM
Marostegui added subscribers: ayounsi, jcrespo, akosiaris.Jun 22 2020, 7:52 AM

@jcrespo @akosiaris @ayounsi I would like to switchover the master to the new master that runs Buster and MariaDB 10.4.
m1 holds:

bacula
librenms (which in previous switchovers has shown that no action is required)
etherpad (which we probably need to restart, as done in previous reimages).

The switchover shouldn't take more than a few seconds as we just need to run the script to change replication and reload the proxies.

@jcrespo @akosiaris - I would like to do this maybe Thursday 25th at 08:00 AM UTC. Is that ok? Let me know if you prefer any other day/time

akosiaris added a comment.Jun 22 2020, 7:54 AM
In T254556#6243442, @Marostegui wrote:

@jcrespo @akosiaris - I would like to do this maybe Thursday 25th at 08:00 AM UTC. Is that ok? Let me know if you prefer any other day/time

Sure, fine by me.

Marostegui added a comment.Jun 22 2020, 7:56 AM

Thank you, I have sent a calendar invite to you and to @jcrespo

gerritbot added a comment.Jun 22 2020, 8:00 AM

Change 606953 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1097 to m1 master

https://gerrit.wikimedia.org/r/606953

jcrespo added a comment.Jun 22 2020, 8:19 AM

ok.

Marostegui added a project: User-notice.Jun 23 2020, 7:03 AM

Adding the User-notice tag as Etherpad will be on read-only for a few seconds Thursday 25th at 08:00 AM UTC
Will email wikitech-l shortly.

RhinosF1 subscribed.Jun 23 2020, 7:07 AM
taavi subscribed.Jun 23 2020, 7:13 AM
gerritbot added a comment.Jun 23 2020, 9:28 AM

Change 607249 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1012,1014: Place db1097 as standby host.

https://gerrit.wikimedia.org/r/607249

gerritbot added a comment.Jun 23 2020, 9:30 AM

Change 607249 merged by Marostegui:
[operations/puppet@production] dbproxy1012,1014: Place db1097 as standby host.

https://gerrit.wikimedia.org/r/607249

Stashbot added a comment.Jun 23 2020, 9:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-23T09:32:17Z] <marostegui> Reload haproxy on dbproxy1012 and dbproxy1014 to test db1097 as secondary for 24h T254556

Johan removed a project: User-notice.EditedJun 25 2020, 1:52 AM
Johan subscribed.

Since next Tech News won't go out until Monday anyway (and I don't think we need to be too concerned about a few seconds of Etherpad read-only) – do re-instate if the update for some reason goes wrong and there are lingering problems the communities should know about.

Johan unsubscribed.Jun 25 2020, 1:52 AM
Stashbot added a comment.Jun 25 2020, 7:08 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-25T07:08:50Z] <marostegui> Start pre switchover steps on m1 T254556

gerritbot added a comment.Jun 25 2020, 7:30 AM

Change 606953 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1097 to m1 master

https://gerrit.wikimedia.org/r/606953

Stashbot added a comment.Jun 25 2020, 7:52 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-25T07:52:08Z] <jynus> stop bacula-director on backup1001 for db maintenance T254556

Stashbot added a comment.Jun 25 2020, 8:03 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-25T08:03:05Z] <marostegui> Failover m1 from db1135 to db1097 - T254556

Maintenance_bot removed a project: Patch-For-Review.Jun 25 2020, 8:10 AM
gerritbot added a comment.Jun 25 2020, 8:29 AM

Change 607725 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software/wmfmariadbpy@master] switchover.py: Change zarcillo instance

https://gerrit.wikimedia.org/r/607725

gerritbot added a project: Patch-For-Review.Jun 25 2020, 8:29 AM

Change 607725 merged by Marostegui:
[operations/software/wmfmariadbpy@master] switchover.py: Change zarcillo instance

https://gerrit.wikimedia.org/r/607725

Marostegui mentioned this in rOSMD8ee0e032d404: switchover.py: Change zarcillo instance.Jun 25 2020, 8:32 AM
Marostegui added a comment.Jun 25 2020, 8:44 AM

This is done.
I am going to leave db1135 replicating for 24h (so we can also see if basic 10.4 -> 10.1 replication works) and then I will move db1135 somewhere else.

Marostegui closed this task as Resolved.Jun 25 2020, 8:44 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 25 2020, 9:10 AM
Marostegui mentioned this in rOSWB8ee0e032d404: switchover.py: Change zarcillo instance.Sep 1 2020, 8:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL