Page MenuHomePhabricator
Create Task
Maniphest T257540

Upgrade m2 to Buster and Mariadb 10.4
Closed, ResolvedPublic
Actions

Description

After m1, let's now upgrade m2 to Buster and MariaDB 10.4

  • db2133
  • db2078
  • db1132 (to be replaced with db1107)
  • db1107
  • db1117

Involved active databases:

xhgui
recommendationsapi
otrs
debmonitor

Details

SubjectRepoBranchLines +/-
db1132: Disable notificationsoperations/puppetproduction+1 -0
mariadb: Promote db1107 to m2 masteroperations/puppetproduction+8 -7
dbproxy1013,1015: Test db1107 in haproxyoperations/puppetproduction+4 -4
db1107: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Move db1107 from s1 to m2operations/puppetproduction+11 -4
db1084: Enable notificationsoperations/puppetproduction+0 -1
install_server: Format db1084operations/puppetproduction+1 -1
mariadb: Move db1084 from s4 to m2operations/puppetproduction+9 -3
db1084: Remove it from dbctloperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jul 9 2020, 5:20 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 9 2020, 5:20 AM
Marostegui triaged this task as Medium priority.Jul 9 2020, 5:21 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jul 9 2020, 5:22 AM

Change 610597 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1084: Remove it from dbctl

https://gerrit.wikimedia.org/r/610597

gerritbot added a project: Patch-For-Review.Jul 9 2020, 5:22 AM

Change 610597 merged by Marostegui:
[operations/puppet@production] db1084: Remove it from dbctl

https://gerrit.wikimedia.org/r/610597

gerritbot added a comment.Jul 9 2020, 5:31 AM

Change 610598 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1084 from s4 to m2

https://gerrit.wikimedia.org/r/610598

gerritbot added a comment.Jul 9 2020, 6:34 AM

Change 610598 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1084 from s4 to m2

https://gerrit.wikimedia.org/r/610598

ops-monitoring-bot added a comment.Jul 9 2020, 6:37 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1084.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007090637_marostegui_10724.log.

ops-monitoring-bot added a comment.Jul 9 2020, 6:58 AM

Completed auto-reimage of hosts:

['db1084.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jul 9 2020, 6:58 AM

Change 610675 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Format db1084

https://gerrit.wikimedia.org/r/610675

gerritbot added a comment.Jul 9 2020, 6:59 AM

Change 610675 merged by Marostegui:
[operations/puppet@production] install_server: Format db1084

https://gerrit.wikimedia.org/r/610675

ops-monitoring-bot added a comment.Jul 9 2020, 7:03 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1084.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007090703_marostegui_13834.log.

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2020, 7:10 AM
ops-monitoring-bot added a comment.Jul 9 2020, 7:24 AM

Completed auto-reimage of hosts:

['db1084.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 9 2020, 7:59 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-09T07:59:41Z] <marostegui> Stop db1117:3322 to clone db1084, this will trigger haproxy alerts - T257540

Marostegui updated the task description. (Show Details)Jul 9 2020, 9:31 AM
• jcrespo added a subtask: T257928: Create a parallel OTRS database with a frozen snapshot of the production one.Jul 14 2020, 2:20 PM
• jcrespo subscribed.Jul 14 2020, 2:23 PM

Given 90% of m2 is OTRS database, I will setup db1077 with buster/MariaDB 10.4 on db1077 at T257928, and that will allow testing of the eventual upgrade of the primary instance to it. If that works correctly, I think the upgrade should cause no issue/could happen shortly after.

• jcrespo closed subtask T257928: Create a parallel OTRS database with a frozen snapshot of the production one as Resolved.Jul 16 2020, 7:23 AM
gerritbot added a comment.Jul 17 2020, 5:06 AM

Change 613466 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1084: Enable notifications

https://gerrit.wikimedia.org/r/613466

gerritbot added a project: Patch-For-Review.Jul 17 2020, 5:06 AM

Change 613466 merged by Marostegui:
[operations/puppet@production] db1084: Enable notifications

https://gerrit.wikimedia.org/r/613466

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2020, 5:10 AM
Marostegui updated the task description. (Show Details)Jul 22 2020, 6:28 AM
Stashbot added a comment.Jul 23 2020, 8:16 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T08:16:50Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1107 to move it to m2 T257540', diff saved to https://phabricator.wikimedia.org/P12024 and previous config saved to /var/cache/conftool/dbconfig/20200723-081650-marostegui.json

gerritbot added a comment.Jul 23 2020, 8:19 AM

Change 615669 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1107 from s1 to m2

https://gerrit.wikimedia.org/r/615669

gerritbot added a project: Patch-For-Review.Jul 23 2020, 8:20 AM
gerritbot added a comment.Jul 23 2020, 8:24 AM

Change 615669 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1107 from s1 to m2

https://gerrit.wikimedia.org/r/615669

Stashbot added a comment.Jul 23 2020, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T08:26:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1107 from s1 T257540', diff saved to https://phabricator.wikimedia.org/P12025 and previous config saved to /var/cache/conftool/dbconfig/20200723-082647-marostegui.json

ops-monitoring-bot added a comment.Jul 23 2020, 8:27 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1107.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202007230827_marostegui_25483.log.

ops-monitoring-bot added a comment.Jul 23 2020, 8:46 AM

Completed auto-reimage of hosts:

['db1107.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Jul 23 2020, 8:59 AM

Mentioned in SAL (#wikimedia-operations) [2020-07-23T08:59:00Z] <marostegui> transfer --type=xtrabackup from db1117:3322 to db1107 T257540

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2020, 9:10 AM
Marostegui updated the task description. (Show Details)Jul 23 2020, 10:53 AM
Marostegui added a comment.Jul 23 2020, 12:49 PM

The only thing pending is the master switchover.
Going to wait a few days to make sure db1107 works ok and then I will go ahead and get a day scheduled

gerritbot added a comment.Jul 24 2020, 5:01 AM

Change 615926 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1107: Enable notifications

https://gerrit.wikimedia.org/r/615926

gerritbot added a project: Patch-For-Review.Jul 24 2020, 5:01 AM

Change 615926 merged by Marostegui:
[operations/puppet@production] db1107: Enable notifications

https://gerrit.wikimedia.org/r/615926

Ailbeve subscribed.Jul 26 2020, 10:38 AM
Adamant.pwn subscribed.EditedJul 27 2020, 1:15 AM

@Marostegui hi there! I want to analyse some categorylinks data (namely, collect all categories which are reachable from given category), but it's hierarchical and traversing it usually means utilising recursive common table expressions (CTE). It's only supported for MariaDB 10.2+, which is not the case for replica server (which is 10.1.44 AFAIK). So I wanted to ask when such upgrade will be available. Thanks.

MBH subscribed.Jul 27 2020, 2:57 AM
Marostegui added a comment.Jul 27 2020, 5:08 AM

@Adamant.pwn I assume you need wikireplicas service for that and not m2?
If that is the case, we are not yet sure when wikireplicas will be upgraded, but it will take a few months still for them to be running mariadb 10.4.

Thanks

Marostegui updated the task description. (Show Details)Jul 27 2020, 6:51 AM
Marostegui added subscribers: • dpifke, Volans, akosiaris, leila.Jul 27 2020, 6:58 AM

@jcrespo @akosiaris @Volans @dpifke Recommendation-API Research I would like to failover the master Tuesday 4th Aug at 08:00 AM UTC
This means around 1 minute of read only time.

Marostegui added projects: Recommendation-API, Research.Jul 27 2020, 6:58 AM
Volans added a comment.Jul 27 2020, 7:06 AM

@Marostegui: green light from me for Debmonitor, usually no action is needed as it reconnects automatically but I can be around just in case.

akosiaris added a project: Znuny.Jul 27 2020, 7:10 AM
In T257540#6336427, @Marostegui wrote:

@jcrespo @akosiaris @Volans @dpifke Recommendation-API Research I would like to failover the master Tuesday 4th Aug at 08:00 AM UTC
This means around 1 minute of read only time.

Fine by me.

akosiaris moved this task from Incoming to Pending patch / update on the Znuny board.Jul 27 2020, 7:11 AM
• dpifke added a comment.Jul 27 2020, 5:19 PM

No objection here.

gerritbot added a comment.Jul 30 2020, 9:30 AM

Change 617402 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1013,1015: Test db1107 into haproxy

https://gerrit.wikimedia.org/r/617402

gerritbot added a comment.Jul 30 2020, 10:37 AM

Change 617402 merged by Marostegui:
[operations/puppet@production] dbproxy1013,1015: Test db1107 in haproxy

https://gerrit.wikimedia.org/r/617402

gerritbot added a comment.Aug 3 2020, 7:17 AM

Change 617997 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1107 to m2 master

https://gerrit.wikimedia.org/r/617997

Stashbot added a comment.Aug 4 2020, 7:27 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-04T07:27:47Z] <marostegui> Start topology changes on m2 - T257540

gerritbot added a comment.Aug 4 2020, 7:29 AM

Change 617997 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1107 to m2 master

https://gerrit.wikimedia.org/r/617997

MBH unsubscribed.Aug 4 2020, 7:32 AM
Stashbot added a comment.Aug 4 2020, 8:00 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-04T08:00:23Z] <marostegui> Failover m2 from db1132 to db1107 -T257540

Marostegui added a comment.Aug 4 2020, 8:06 AM

Failover done successfully

gerritbot added a comment.Aug 4 2020, 8:14 AM

Change 618236 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1132: Disable notifications

https://gerrit.wikimedia.org/r/618236

gerritbot added a comment.Aug 4 2020, 8:15 AM

Change 618236 merged by Marostegui:
[operations/puppet@production] db1132: Disable notifications

https://gerrit.wikimedia.org/r/618236

Marostegui closed this task as Resolved.Aug 4 2020, 8:31 AM

Resolving this - db1132 will be moved to m3 which will be tracked at T253217

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits