Page MenuHomePhabricator
Create Task
Maniphest T275112

Migrate codfw sanitarium hosts (db2094/db2095) to Buster and 10.4
Closed, ResolvedPublic
Actions

Description

Once we've disconnected the old labsdb* hosts from the old sanitariums in eqiad (db1124/db1125) we should go ahead and migrate db2094/db2095 in codfw to Buster and 10.4
db1124/db1125 will not need to be migrated as they'll be repurposed somewhere else (T258361)

Details

SubjectRepoBranchLines +/-
db2095: Disable notificationsoperations/puppetproduction+1 -0
db2094: Disable notificationsoperations/puppetproduction+1 -0
install_server: Reimage db2094,db2095 to busteroperations/puppetproduction+0 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Feb 18 2021, 9:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 18 2021, 9:10 AM
Marostegui changed the task status from Open to Stalled.Feb 18 2021, 9:10 AM
Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to Blocked on the DBA board.
Marostegui added a comment.Mar 11 2021, 3:22 PM

This can happen after 15th April

Marostegui changed the task status from Stalled to Open.Apr 15 2021, 5:54 AM
Marostegui moved this task from Blocked to Ready on the DBA board.
Marostegui moved this task from Ready to In progress on the DBA board.Apr 16 2021, 5:28 AM
gerritbot added a comment.Apr 16 2021, 5:30 AM

Change 680162 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2094,db2095 to buster

https://gerrit.wikimedia.org/r/680162

gerritbot added a project: Patch-For-Review.Apr 16 2021, 5:30 AM

Change 680162 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2094,db2095 to buster

https://gerrit.wikimedia.org/r/680162

ops-monitoring-bot added a comment.Apr 16 2021, 5:35 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2094.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104160534_marostegui_9954.log.

ops-monitoring-bot added a comment.Apr 16 2021, 5:57 AM

Completed auto-reimage of hosts:

['db2094.codfw.wmnet']

and were ALL successful.

Marostegui added a comment.Apr 16 2021, 6:03 AM

db2094 has been migrated to Buster. I am now checking all the tables to look for corruption, as we saw some during while setting up the buster sanitarium hosts in eqiad.

gerritbot added a comment.Apr 16 2021, 6:04 AM

Change 680166 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2094: Disable notifications

https://gerrit.wikimedia.org/r/680166

gerritbot added a comment.Apr 16 2021, 6:05 AM

Change 680166 merged by Marostegui:

[operations/puppet@production] db2094: Disable notifications

https://gerrit.wikimedia.org/r/680166

ops-monitoring-bot added a comment.Apr 16 2021, 6:06 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2095.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104160606_marostegui_18089.log.

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2021, 6:10 AM
gerritbot added a comment.Apr 16 2021, 6:22 AM

Change 680172 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2095: Disable notifications

https://gerrit.wikimedia.org/r/680172

gerritbot added a project: Patch-For-Review.Apr 16 2021, 6:22 AM

Change 680172 merged by Marostegui:

[operations/puppet@production] db2095: Disable notifications

https://gerrit.wikimedia.org/r/680172

ops-monitoring-bot added a comment.Apr 16 2021, 6:29 AM

Completed auto-reimage of hosts:

['db2095.codfw.wmnet']

and were ALL successful.

Marostegui added a comment.Apr 16 2021, 6:33 AM

db2095 has been migrated to Buster. I am now checking all the tables to look for corruption, as we saw some during while setting up the buster sanitarium hosts in eqiad.

Marostegui claimed this task.Apr 16 2021, 6:34 AM
Marostegui added a parent task: T250666: Upgrade WMF database-and-backup-related hosts to buster.
Marostegui mentioned this in T280315: Migrate sanitarium masters to 10.4 and Buster.Apr 16 2021, 6:38 AM
Marostegui added a comment.Apr 16 2021, 6:45 AM

There's some corruption on both, db2094 and db2095, so once all the checks are finished, I will need to rebuild the affected tables.

Marostegui added a comment.Apr 18 2021, 7:20 AM

I have started to rebuild all tables that reported errors across all the sections on db2094 and db2095

Marostegui added a comment.Apr 19 2021, 4:58 AM

Only s4 and s8 are still being fixed. The rest of sections are done and replication is restarted

Marostegui added a comment.Apr 19 2021, 9:42 AM

db2094 is fully done

Marostegui closed this task as Resolved.Apr 20 2021, 5:44 AM

db2095 is fixed

Marostegui mentioned this in T280751: Upgrade s6 to Debian Buster and MariaDB 10.4.Apr 21 2021, 5:29 AM
Marostegui mentioned this in T283131: Upgrade s3 to Debian Buster and MariaDB 10.4.May 19 2021, 6:54 AM
Marostegui mentioned this in T283235: Upgrade s5 to Debian Buster and MariaDB 10.4.May 20 2021, 11:11 AM
Marostegui mentioned this in T287230: Upgrade s2 to Buster + MariaDB 10.4.Jul 23 2021, 7:26 AM
• Kormat mentioned this in T288244: Upgrade s7 to Debian Buster and MariaDB 10.4.Aug 5 2021, 2:32 PM
Marostegui mentioned this in T288803: Upgrade s4 to Buster + MariaDB 10.4.Aug 13 2021, 7:53 AM
Marostegui mentioned this in T290865: Upgrade s1 to Buster + MariaDB 10.4.Sep 13 2021, 12:51 PM
Marostegui mentioned this in T290868: Upgrade s8 to Buster + MariaDB 10.4.Sep 13 2021, 1:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits