Page MenuHomePhabricator
Create Task
Maniphest T254870

Upgrade analytics dbstore databases to Buster and Mariadb 10.4
Closed, ResolvedPublic
Actions

Description

I would like to upgrade dbstore1003, dbstore1004 and dbstore1005 to Buster and MariaDB 10.4
We already have lots of hosts in production running this setup (including multi-instance hosts like dbstore100X).

I would do one at the time and will leave a few days between upgrading them.
Normally, upgrading a host takes around 15-30 minutes, so MySQL won't be accessible during this time.
@Nuria @elukey any objection from your side? Any preference from which host I should start from?

  • dbstore1003 (s1, s5, s7)
  • dbstore1004 (s2, s3, s4)
  • dbstore1005 (s6, s8, x1, staging database)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -2mariadb: Reimage dbstore1004 to Buster
operations/puppetproduction+0 -1dbstore1005: Enable notifications
operations/puppetproduction+1 -1install_server: Remiage dbstore1005 to Buster
operations/puppetproduction+0 -1dbstore1004: Enable notifications
operations/puppetproduction+1 -2mariadb: Reimage dbstore1003 with Buster
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Jun 9 2020, 12:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 9 2020, 12:34 PM
Marostegui triaged this task as Medium priority.Jun 9 2020, 12:34 PM
Marostegui moved this task from Triage to Pending comment on the DBA board.
Marostegui mentioned this in T254871: Upgrade x1 databases to Buster and Mariadb 10.4.Jun 9 2020, 12:42 PM
elukey added a comment.Jun 9 2020, 12:45 PM

No preference! I think that given the limited downtime it can be done anytime, no problems from my side. Do you require help in doing anything?

Marostegui added a comment.Jun 9 2020, 12:46 PM
In T254870#6205693, @elukey wrote:

No preference! I think that given the limited downtime it can be done anytime, no problems from my side. Do you require help in doing anything?

Thank you! I will start tomorrow then with dbstore1003. No help is needed, thank you! :)

gerritbot added a comment.Jun 11 2020, 5:08 AM

Change 604551 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage dbstore1003 with Buster

https://gerrit.wikimedia.org/r/604551

gerritbot added a project: Patch-For-Review.Jun 11 2020, 5:08 AM
gerritbot added a comment.Jun 11 2020, 7:05 AM

Change 604551 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage dbstore1003 with Buster

https://gerrit.wikimedia.org/r/604551

Stashbot added a comment.Jun 11 2020, 7:07 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-11T07:07:52Z] <marostegui> Stop MySQL on dbstore1003 for reimage - T254870

Maintenance_bot removed a project: Patch-For-Review.Jun 11 2020, 7:10 AM
ops-monitoring-bot added a comment.Jun 11 2020, 7:12 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbstore1003.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006110712_marostegui_199022.log.

ops-monitoring-bot added a comment.Jun 11 2020, 7:30 AM

Completed auto-reimage of hosts:

['dbstore1003.eqiad.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Jun 11 2020, 7:44 AM
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.Jun 15 2020, 3:54 PM
gerritbot added a comment.Jun 16 2020, 4:40 AM

Change 605731 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage dbstore1004 to Buster

https://gerrit.wikimedia.org/r/605731

gerritbot added a project: Patch-For-Review.Jun 16 2020, 4:40 AM

Change 605731 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage dbstore1004 to Buster

https://gerrit.wikimedia.org/r/605731

ops-monitoring-bot added a comment.Jun 16 2020, 4:49 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['dbstore1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006160449_marostegui_211436.log.

ops-monitoring-bot added a comment.Jun 16 2020, 5:10 AM

Completed auto-reimage of hosts:

['dbstore1004.eqiad.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2020, 5:10 AM
Marostegui updated the task description. (Show Details)Jun 16 2020, 5:12 AM
gerritbot added a comment.Jun 16 2020, 6:50 AM

Change 605809 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbstore1004: Enable notifications

https://gerrit.wikimedia.org/r/605809

gerritbot added a project: Patch-For-Review.Jun 16 2020, 6:50 AM

Change 605809 merged by Marostegui:
[operations/puppet@production] dbstore1004: Enable notifications

https://gerrit.wikimedia.org/r/605809

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2020, 7:10 AM
gerritbot added a comment.Jun 22 2020, 6:03 AM

Change 606851 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbstore1005: Disable notifications

https://gerrit.wikimedia.org/r/606851

gerritbot added a project: Patch-For-Review.Jun 22 2020, 6:03 AM
gerritbot added a comment.Jun 22 2020, 6:05 AM

Change 606851 merged by Marostegui:
[operations/puppet@production] install_server: Remiage dbstore1005 to Buster

https://gerrit.wikimedia.org/r/606851

Stashbot added a comment.Jun 22 2020, 6:06 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-22T06:06:09Z] <marostegui> Stop MySQL on dbstore1005 for reimage to Buster - T254870

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2020, 6:10 AM
ops-monitoring-bot added a comment.Jun 22 2020, 6:11 AM

Script wmf-auto-reimage was launched by marostegui on cumin2001.codfw.wmnet for hosts:

['dbstore1005.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006220611_marostegui_27761.log.

ops-monitoring-bot added a comment.Jun 22 2020, 6:30 AM

Completed auto-reimage of hosts:

['dbstore1005.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.Jun 22 2020, 6:38 AM

Change 606853 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbstore1005: Enable notifications

https://gerrit.wikimedia.org/r/606853

gerritbot added a project: Patch-For-Review.Jun 22 2020, 6:38 AM
gerritbot added a comment.Jun 22 2020, 6:41 AM

Change 606853 merged by Marostegui:
[operations/puppet@production] dbstore1005: Enable notifications

https://gerrit.wikimedia.org/r/606853

Marostegui closed this task as Resolved.Jun 22 2020, 7:00 AM
Marostegui claimed this task.
Marostegui updated the task description. (Show Details)

All done!

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2020, 7:10 AM
Kormat mentioned this in T256966: dbstore1005 s8 mariadb instance crashed.Jul 2 2020, 12:27 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL