Now that dbctl is in place, we should build some automation around automatically upgrading databases (mysql, kernel...).
Ideally the script should:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T327300 Cookbook for rack depool | |||
| In Progress | FCeratto-WMF | T384810 MariaDB lifetime management system | |||
| Open | FCeratto-WMF | T239814 Automate DB upgrades | |||
| Resolved | None | T374191 upgrade clone cookbook | |||
| Declined | ABran-WMF | T376712 mysql_legacy: SQL query quote escape | |||
| Resolved | ABran-WMF | T376701 mysql_legacy data_directory getter | |||
| Resolved | ABran-WMF | T377129 mariadb: systemctl status accessor in mysql_legacy |
Random note: With auto_schema, it provides a simple interface to do this kind of work. It only needs to have proper support of multiinstance hosts but otherwise it's good to be at least tested. I wrote a POC for mysql upgrades in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/+/master/dbtools/auto_schema/upgrade_mysql.py
Change 748720 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/software@master] auto_schema: Rework upgrade_mysql a bit to reuse code
Change 749176 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/cookbooks@master] Add MySQL upgrade cookbook
Change 749195 had a related patch set uploaded (by Jbond; author: jbond):
[operations/cookbooks@master] Add MySQL upgrade cookbook
Change 749176 merged by jenkins-bot:
[operations/cookbooks@master] Add MySQL upgrade cookbook
Change 751157 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/cookbooks@master] sre.myql.upgrade: Fix missing argument
Change 751157 merged by jenkins-bot:
[operations/cookbooks@master] sre.myql.upgrade: Fix missing argument
Change 751161 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/cookbooks@master] sre.mysql.upgrade: Fix argparse
Change 751161 merged by jenkins-bot:
[operations/cookbooks@master] sre.mysql.upgrade: Fix argparse
Change 751225 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/cookbooks@master] sre.mysql.upgrade: Fix calling icinga with list
Change 751225 merged by jenkins-bot:
[operations/cookbooks@master] sre.mysql.upgrade: Fix calling icinga with list
Change 751226 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/cookbooks@master] sre.mysql.upgrade: Fix the icinga, second try
Change 751226 merged by jenkins-bot:
[operations/cookbooks@master] sre.mysql.upgrade: Fix the icinga, second try
Change 751228 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/cookbooks@master] sre.mysql.upgrade: Add logger object
Change 751228 merged by jenkins-bot:
[operations/cookbooks@master] sre.mysql.upgrade: Add logger object
I just did an upgrade using a cookbook: P18336
It doesn't depool yet but I let auto_schema handle that part.
Change 748720 merged by jenkins-bot:
[operations/software@master] auto_schema: Rework upgrade_mysql to reuse code and cookbooks
Change 752700 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[operations/software@master] auto_schema: Force depool in codfw for mysql upgrades
Change 752700 merged by jenkins-bot:
[operations/software@master] auto_schema: Force depool in codfw for mysql upgrades
Change 754872 had a related patch set uploaded (by Volans; author: Volans):
[operations/cookbooks@master] sre.mysql.upgrade: various improvements
Mentioned in SAL (#wikimedia-operations) [2022-01-19T13:35:15Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1100 (T239814)', diff saved to https://phabricator.wikimedia.org/P18864 and previous config saved to /var/cache/conftool/dbconfig/20220119-133514-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-19T17:16:41Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1100 (T239814)', diff saved to https://phabricator.wikimedia.org/P18885 and previous config saved to /var/cache/conftool/dbconfig/20220119-171640-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-19T18:01:55Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1100 (T239814)', diff saved to https://phabricator.wikimedia.org/P18888 and previous config saved to /var/cache/conftool/dbconfig/20220119-180154-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-19T18:08:40Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1110 (T239814)', diff saved to https://phabricator.wikimedia.org/P18889 and previous config saved to /var/cache/conftool/dbconfig/20220119-180840-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-19T18:16:24Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1110 (T239814)', diff saved to https://phabricator.wikimedia.org/P18890 and previous config saved to /var/cache/conftool/dbconfig/20220119-181623-ladsgroup.json
Mentioned in SAL (#wikimedia-operations) [2022-01-19T19:01:38Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1110 (T239814)', diff saved to https://phabricator.wikimedia.org/P18893 and previous config saved to /var/cache/conftool/dbconfig/20220119-190137-ladsgroup.json
I ran the automatic upgrades in two eqiad hosts in s5, for one the db simply just refused to get back but that seemed to be the issue of the host being old/etc. but a powercycle seemed to fix the issue. The second one went fully smoothly and no issues so far.
We are currently upgrading all of ours hosts to bullseye which requires a different cookbook so we can't focus on this work for a while. Moving back the task to reflect the reality and we will pick this up and finish it (including support for multiinstance and hosts that have replicas) maybe in Q4
Change 749195 abandoned by Jbond:
[operations/cookbooks@master] Add MySQL upgrade cookbook
Reason:
Change 754872 abandoned by Volans:
[operations/cookbooks@master] sre.mysql.upgrade: various improvements
Reason:
Abandoning after 2 years, owners not interested
What's exactly blocked? The migration to 10.6.19 or the idea of having a script to upgrade all databases automatically. If it is the latter, I don't think a possible bug (which I have been unable to reproduce (T375652#10295853)) should block the efforts to DB upgrades automation.
it was blocked because of the unknown nature of the bug we encountered so we chose to freeze upgrade until it was at list partially identified. I don't think we should consider it blocked anymore.
I've updated what is already done on the cookbook sre.mysql.upgrade.
I'd like to get @FCeratto-WMF involved here and see if we can get two critical steps done:
These two would give us the first step to get automatic upgrades done. There was a stab done in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1080718 but I am not sure if we want to continue that approach, resume it, or start a new one.
I'd appreciate @Volans opinion here!
I think it's ok to resume that patch that was mainly doing two things:
Adding comment just for visibility - work is being resumed in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1080718
I updated the task summary: the cookbook is doing depooling, replication lag checks, icinga checks and repooling and was tested on different hosts.
Thank you. So I think what's left is the first item. To be on the safe side, we should just check that we are not upgrading more than 1 host from the same section and that should be good enough for now.
What do you think?
As discussed in a meeting I'll add a lock on the section to update one host at a time and show it on https://zarcillo.wikimedia.org/ui/locks