Page MenuHomePhabricator
Create Task
Maniphest T288803

Upgrade s4 to Buster + MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

Given that the switch back will be in September, let's upgrade s4 to fully Buster and MariaDB 10.4

  • Available backup source on the standby DC (db1150 - all sections have sources everywhere now)
  • Switchover backup generation standby DC (waiting for DBAs to merge) https://gerrit.wikimedia.org/r/c/operations/puppet/+/714704
  • Candidate master on the standby DC
  • Master on the standby DC
  • Candidate master on the primary DC
  • Available backup source on the primary DC (db2139 - all sections have sources everywhere now)
  • Switchover backup generation Primary DC (waiting for switchover date https://gerrit.wikimedia.org/r/c/operations/puppet/+/715919 )
  • Switchover on the primary DC to promote a Buster+10.4 host to master: T289650
  • Upgrade the old master and make it a candidate master, pool it
  • Cleanup (remove) old backup sources from both DCs

Please read the doc about procedure for more details.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Sep 2 2021, 11:28 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T11:28:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 10%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17161 and previous config saved to /var/cache/conftool/dbconfig/20210902-112812-root.json

Stashbot added a comment.Sep 2 2021, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T11:43:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 25%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17162 and previous config saved to /var/cache/conftool/dbconfig/20210902-114315-root.json

Stashbot added a comment.Sep 2 2021, 11:58 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T11:58:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17163 and previous config saved to /var/cache/conftool/dbconfig/20210902-115819-root.json

Stashbot added a comment.Sep 2 2021, 12:13 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T12:13:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17164 and previous config saved to /var/cache/conftool/dbconfig/20210902-121323-root.json

Stashbot added a comment.Sep 2 2021, 12:28 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T12:28:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2106 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17165 and previous config saved to /var/cache/conftool/dbconfig/20210902-122826-root.json

Stashbot added a comment.Sep 2 2021, 12:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T12:44:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2119 (re)pooling @ 10%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17167 and previous config saved to /var/cache/conftool/dbconfig/20210902-124434-root.json

Stashbot added a comment.Sep 2 2021, 12:59 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T12:59:38Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2119 (re)pooling @ 25%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17168 and previous config saved to /var/cache/conftool/dbconfig/20210902-125937-root.json

Stashbot added a comment.Sep 2 2021, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T13:14:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2119 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17169 and previous config saved to /var/cache/conftool/dbconfig/20210902-131441-root.json

Stashbot added a comment.Sep 2 2021, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T13:29:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2119 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17171 and previous config saved to /var/cache/conftool/dbconfig/20210902-132945-root.json

Stashbot added a comment.Sep 2 2021, 1:44 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T13:44:49Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2119 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17172 and previous config saved to /var/cache/conftool/dbconfig/20210902-134448-root.json

Stashbot added a comment.Sep 2 2021, 2:03 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T14:03:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17174 and previous config saved to /var/cache/conftool/dbconfig/20210902-140357-root.json

Stashbot added a comment.Sep 2 2021, 2:19 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T14:19:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17175 and previous config saved to /var/cache/conftool/dbconfig/20210902-141901-root.json

Stashbot added a comment.Sep 2 2021, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T14:34:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17176 and previous config saved to /var/cache/conftool/dbconfig/20210902-143405-root.json

Stashbot added a comment.Sep 2 2021, 2:49 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T14:49:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17177 and previous config saved to /var/cache/conftool/dbconfig/20210902-144908-root.json

Stashbot added a comment.Sep 2 2021, 3:04 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-02T15:04:13Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17178 and previous config saved to /var/cache/conftool/dbconfig/20210902-150412-root.json

Stashbot added a comment.Sep 3 2021, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:11:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 10%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17193 and previous config saved to /var/cache/conftool/dbconfig/20210903-051124-root.json

Stashbot added a comment.Sep 3 2021, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:11:50Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 10%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17194 and previous config saved to /var/cache/conftool/dbconfig/20210903-051149-root.json

Marostegui added a comment.Sep 3 2021, 5:12 AM

All hosts ready

Stashbot added a comment.Sep 3 2021, 5:26 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:26:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 25%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17195 and previous config saved to /var/cache/conftool/dbconfig/20210903-052628-root.json

Stashbot added a comment.Sep 3 2021, 5:26 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:26:54Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 25%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17196 and previous config saved to /var/cache/conftool/dbconfig/20210903-052653-root.json

Stashbot added a comment.Sep 3 2021, 5:41 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:41:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17198 and previous config saved to /var/cache/conftool/dbconfig/20210903-054131-root.json

Stashbot added a comment.Sep 3 2021, 5:41 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:41:57Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 50%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17199 and previous config saved to /var/cache/conftool/dbconfig/20210903-054157-root.json

Stashbot added a comment.Sep 3 2021, 5:56 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:56:35Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17200 and previous config saved to /var/cache/conftool/dbconfig/20210903-055635-root.json

Stashbot added a comment.Sep 3 2021, 5:57 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T05:57:01Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 75%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17201 and previous config saved to /var/cache/conftool/dbconfig/20210903-055700-root.json

Stashbot added a comment.Sep 3 2021, 6:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T06:11:39Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3312 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17202 and previous config saved to /var/cache/conftool/dbconfig/20210903-061138-root.json

Stashbot added a comment.Sep 3 2021, 6:12 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-03T06:12:04Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2138:3314 (re)pooling @ 100%: Slowly repool after reimage T288803', diff saved to https://phabricator.wikimedia.org/P17203 and previous config saved to /var/cache/conftool/dbconfig/20210903-061204-root.json

Marostegui added a comment.Sep 6 2021, 4:26 AM

s4 codfw backup is on-going now, once it is finished I will merge the backup patch.

gerritbot added a comment.Sep 6 2021, 4:30 AM

Change 718938 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2090: Disable notifications

https://gerrit.wikimedia.org/r/718938

gerritbot added a comment.Sep 6 2021, 4:31 AM

Change 718938 merged by Marostegui:

[operations/puppet@production] db2090: Disable notifications

https://gerrit.wikimedia.org/r/718938

Stashbot added a comment.Sep 6 2021, 5:07 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-06T05:07:47Z] <marostegui> Stop replication on db2090 (old s4 master) T289650 T288803

Stashbot mentioned this in T289650: Switchover s4 from db2090 to db2110.Sep 6 2021, 5:07 AM
gerritbot added a comment.Sep 6 2021, 5:08 AM

Change 718939 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2090 to Buster

https://gerrit.wikimedia.org/r/718939

gerritbot added a comment.Sep 6 2021, 5:10 AM

Change 718939 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2090 to Buster

https://gerrit.wikimedia.org/r/718939

Marostegui closed subtask T289650: Switchover s4 from db2090 to db2110 as Resolved.Sep 6 2021, 5:12 AM
Marostegui updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 6 2021, 5:19 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2090.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202109060519_marostegui_3862.log.

ops-monitoring-bot added a comment.Sep 6 2021, 5:42 AM

Completed auto-reimage of hosts:

['db2090.codfw.wmnet']

and were ALL successful.

Marostegui added a comment.Sep 6 2021, 5:54 AM

db2090 reimaged, checking its tables now

gerritbot added a comment.Sep 6 2021, 7:58 AM

Change 715919 merged by Marostegui:

[operations/puppet@production] dbbackups: Migrate s4 generation from db2097 (stretch) to db2139 (buster)

https://gerrit.wikimedia.org/r/715919

Marostegui updated the task description. (Show Details)Sep 6 2021, 7:58 AM
Marostegui added a comment.Sep 7 2021, 5:00 AM

db2090 came back clean - replication started

Stashbot added a comment.Sep 7 2021, 6:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T06:47:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 5%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17228 and previous config saved to /var/cache/conftool/dbconfig/20210907-064711-root.json

Stashbot added a comment.Sep 7 2021, 7:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T07:02:16Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 10%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17229 and previous config saved to /var/cache/conftool/dbconfig/20210907-070215-root.json

Stashbot added a comment.Sep 7 2021, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T07:17:19Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 25%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17232 and previous config saved to /var/cache/conftool/dbconfig/20210907-071719-root.json

Stashbot added a comment.Sep 7 2021, 7:32 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T07:32:23Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 50%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17234 and previous config saved to /var/cache/conftool/dbconfig/20210907-073222-root.json

Stashbot added a comment.Sep 7 2021, 7:34 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T07:34:36Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Start to pool db2090 into API T288803', diff saved to https://phabricator.wikimedia.org/P17235 and previous config saved to /var/cache/conftool/dbconfig/20210907-073436-marostegui.json

Stashbot added a comment.Sep 7 2021, 7:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T07:47:26Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 75%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17237 and previous config saved to /var/cache/conftool/dbconfig/20210907-074726-root.json

Stashbot added a comment.Sep 7 2021, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T07:49:02Z] <marostegui@cumin1001> dbctl commit (dc=all): 'More weight for db2090 into API T288803', diff saved to https://phabricator.wikimedia.org/P17238 and previous config saved to /var/cache/conftool/dbconfig/20210907-074901-marostegui.json

Stashbot added a comment.Sep 7 2021, 8:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T08:02:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'db2090 (re)pooling @ 100%: Slowly repool T288803', diff saved to https://phabricator.wikimedia.org/P17240 and previous config saved to /var/cache/conftool/dbconfig/20210907-080230-root.json

Stashbot added a comment.Sep 7 2021, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-07T08:29:52Z] <marostegui@cumin1001> dbctl commit (dc=all): 'More weight for db2090 into API T288803', diff saved to https://phabricator.wikimedia.org/P17241 and previous config saved to /var/cache/conftool/dbconfig/20210907-082952-marostegui.json

Marostegui reassigned this task from Marostegui to • jcrespo.Sep 7 2021, 8:30 AM
Marostegui updated the task description. (Show Details)

@jcrespo all yours, only pending backup clean up

• jcrespo added a comment.Sep 7 2021, 8:33 AM

Leaving it up, but idle, until next week.

• jcrespo mentioned this in T288244: Upgrade s7 to Debian Buster and MariaDB 10.4.Sep 7 2021, 9:47 AM
gerritbot added a comment.Sep 13 2021, 2:29 PM

Change 720766 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove s4 and s7 stretch backup source instances

https://gerrit.wikimedia.org/r/720766

gerritbot added a comment.Sep 13 2021, 2:29 PM

Change 720767 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Reenable notifications on stretch hosts after maintenance

https://gerrit.wikimedia.org/r/720767

gerritbot added a comment.Sep 13 2021, 4:58 PM

Change 720766 merged by Jcrespo:

[operations/puppet@production] dbbackups: Remove s4 and s7 stretch backup source instances

https://gerrit.wikimedia.org/r/720766

gerritbot added a comment.Sep 13 2021, 6:42 PM

Change 720767 merged by Jcrespo:

[operations/puppet@production] dbbackups: Reenable notifications on stretch hosts after maintenance

https://gerrit.wikimedia.org/r/720767

• jcrespo added a comment.Sep 13 2021, 6:44 PM

I've cleaned up puppet, icinga and config files of:

  • db1116
  • db1139
  • db2097
  • db2100

However, it is getting late, and I still have to cleanup:

  • datadir
  • tendril
  • zarcillo
  • orchestrator
  • (final checks)

And I don't trust myself so late in the day- sorry if it creates some noise there (e.g. prometheus scrapping errors/tendril red status/orchestrator alerts) until I finish all tasks.

Marostegui moved this task from In progress to Done on the DBA board.Sep 15 2021, 4:47 AM
• jcrespo updated the task description. (Show Details)Sep 15 2021, 10:14 AM
• jcrespo closed this task as Resolved.Sep 15 2021, 11:26 AM
• jcrespo reassigned this task from • jcrespo to Marostegui.

Based on prometheus and tendril, I think this is done, but please do not doubt to ping me if there is any unintended leftover.

Marostegui added a comment.Sep 15 2021, 11:30 AM

Thank you!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits