Page MenuHomePhabricator
Create Task
Maniphest T280492

Upgrade all sanitarium masters to 10.4 and Buster
Closed, ResolvedPublic
Actions

Description

labsdb* hosts have been moved under 10.4 sanitarium hosts, so we can now migrate all sanitarium masters to Buster and 10.4

Hosts:

eqiad:

  • s1 db1106 - [x] tables checked - [x] tables checked after the upgrade
  • s2 db1074 (replace it with db1156 T258361 - [x] tables checked)
  • s3 db1112 - [x] tables checked - [x] tables checked after the upgrade
  • s4 db1121 - [x] tables checked (before the upgrade) - [x] tables checked after the upgrade
  • s5 db1082 (replace it with db1161 T258361 - [x] tables checked)
  • s6 db1085 (replace it with db1165 T258361 - [x] tables checked)
  • s7 db1079 (replace it with db1158 T258361 - [x] tables checked)
  • s8 db1087 (replace it with db1167 T258361 - [x] tables checked)

codfw:

  • s1 db2072 - [x] tables checked
  • s2 db2126 - [x] tables checked
  • s3 db2074 - [x] tables checked
  • s4 db2073 - [x] tables checked
  • s5 db2128 - [x] tables checked
  • s6 db2076 - [x] tables checked
  • s7 db2077 - [x] tables checked
  • s8 db2082 - [x] tables checked

Details

SubjectRepoBranchLines +/-
db1106: Disable notificationsoperations/puppetproduction+1 -0
mariadb: Remove s3 from db2098operations/puppetproduction+0 -0
db1112: Disable notificationsoperations/puppetproduction+1 -0
install_server: Reimage db1106 to Busteroperations/puppetproduction+0 -1
db1121: Disable notificationsoperations/puppetproduction+1 -0
install_server: Switch db1112 to busteroperations/puppetproduction+0 -1
install_server: Reimage db1121 to Busteroperations/puppetproduction+0 -1
dbbackups: remove db2098 s3 section for this codfw backup sourceoperations/puppetproduction+1 -2
db1106: Disable notificationsoperations/puppetproduction+1 -0
db1121: Enable notificationsoperations/puppetproduction+0 -1
db1121: Disable notificationsoperations/puppetproduction+1 -0
mariadb: Reenable notifications on db1156 after maintenanceoperations/puppetproduction+0 -1
mariadb: Reenable notifications on db1102 after maintenanceoperations/puppetproduction+0 -1
dbbackups: Add s2 to db1102, a buster backup sourceoperations/puppetproduction+3 -1
mariadb: Reenable notifications for db2139 after maintenanceoperations/puppetproduction+0 -1
dbbackups: Move backups of s3 on codfw from db2098 to db2139operations/puppetproduction+6 -6
dbbackups: Add s3 to db2139 (mariadb source backups)operations/puppetproduction+4 -2
install_server: Reimgae db2074 to Busteroperations/puppetproduction+0 -1
install_server: Reimage db2127 to Buster.operations/puppetproduction+0 -1
install_server: Reimage db1156 to Busteroperations/puppetproduction+0 -1
install_server: Reimage all codfw sanitarium mastersoperations/puppetproduction+0 -8
mariadb: Reimage db1161 to Buster and 10.4operations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• MarosteguiT233766 labsdb1011 mariadb crashed
Restricted Task
Restricted Task
 OpenNoneT215858 Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema
 OpenNoneT280152 Mitigate breaking changes from the new Wiki Replicas architecture
Unknown Object (Task)
 ResolvedRobHT260441 (Need By: ASAP) rack/setup/install clouddb10[13-20]
 Resolved• BstormT260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture
 Resolved• BstormT260843 Set up roles for new wiki replicas layout
 Resolved• MarosteguiT265135 wikireplicas: Define MW sections per host
 Resolved• MarosteguiT267090 Productionize clouddb10[13-20]
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 Resolved• MarosteguiT250666 Upgrade WMF database-and-backup-related hosts to buster
Restricted Task
 ResolvedLegoktmT281515 June 2021 Datacenter switchover
 Resolved• MarosteguiT285139 DB maintenance work to do while eqiad is passive (June 2021)
 Resolved• MarosteguiT266483 Enable report_host for mariadb
Unknown Object (Task)
 Resolved• MarosteguiT258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
 ResolvedRobHT267043 (Need By: 2020-11-29) rack/setup/install db11[51-76]
 Resolved• MarosteguiT268742 Test upgrading sanitarium hosts to Buster + 10.4
 Resolved• MarosteguiT272008 Move wikireplicas under the new sanitarium hosts (db1154, db1155)
 Resolved• MarosteguiT280492 Upgrade all sanitarium masters to 10.4 and Buster
 ResolvedRequestwiki_willyT281794 decommission db1082.eqiad.wmnet
 ResolvedRequest• CmjohnsonT281959 decommission db1074.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282079 decommission db1079.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282093 decommission db1087.eqiad.wmnet
 ResolvedRequest• CmjohnsonT282096 decommission db1085.eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui added a comment.Apr 26 2021, 3:22 PM

Excellent, thanks. It will take around a day I'd guess.

gerritbot added a comment.Apr 26 2021, 4:20 PM

Change 682682 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Reenable notifications on db1102 after maintenance

https://gerrit.wikimedia.org/r/682682

gerritbot added a comment.Apr 26 2021, 4:22 PM

Change 682682 merged by Jcrespo:

[operations/puppet@production] mariadb: Reenable notifications on db1102 after maintenance

https://gerrit.wikimedia.org/r/682682

jcrespo moved this task from In Progress to Done on the Data-Persistence-Backup board.Apr 27 2021, 7:24 AM
In T280492#7034626, @Marostegui wrote:

Excellent, thanks. It will take around a day I'd guess.

It finished at ~3am: all yours. Please note I loaded grants and events to the best of my ability, but please double check those, as I saw the grant files are very outdated (no tendril grants, prometheus and icinga don't use socket authentication, etc.) so I fixed to the best of my ability.

• Marostegui added a comment.Apr 27 2021, 7:25 AM

Thank you, I will take over it!

• Marostegui updated the task description. (Show Details)Apr 28 2021, 5:53 AM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Apr 28 2021, 7:19 AM

Change 682668 merged by Marostegui:

[operations/puppet@production] mariadb: Reenable notifications on db1156 after maintenance

https://gerrit.wikimedia.org/r/682668

• Marostegui added a parent task: T266483: Enable report_host for mariadb.Apr 28 2021, 1:00 PM
gerritbot added a comment.Apr 29 2021, 6:28 AM

Change 683483 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1121: Disable notifications

https://gerrit.wikimedia.org/r/683483

gerritbot added a comment.Apr 29 2021, 6:29 AM

Change 683483 merged by Marostegui:

[operations/puppet@production] db1121: Disable notifications

https://gerrit.wikimedia.org/r/683483

• Marostegui updated the task description. (Show Details)Apr 30 2021, 7:41 PM
gerritbot added a comment.May 4 2021, 4:39 AM

Change 684667 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1121: Enable notifications

https://gerrit.wikimedia.org/r/684667

gerritbot added a comment.May 4 2021, 4:40 AM

Change 684667 merged by Marostegui:

[operations/puppet@production] db1121: Enable notifications

https://gerrit.wikimedia.org/r/684667

Stashbot added a comment.May 4 2021, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T07:11:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1161 and db1082 to change s5 sanitarium master T280492', diff saved to https://phabricator.wikimedia.org/P15692 and previous config saved to /var/cache/conftool/dbconfig/20210504-071146-marostegui.json

• Marostegui updated the task description. (Show Details)May 4 2021, 7:15 AM

s5 sanitarium master switched: db1154 now replicates from db1161 (10.4)

• Marostegui added a subtask: T281794: decommission db1082.eqiad.wmnet.May 4 2021, 7:18 AM
Stashbot added a comment.May 4 2021, 8:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T08:02:07Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1106 from s1 vslow to get its tables checked and pool db1099:3311 instead T280492', diff saved to https://phabricator.wikimedia.org/P15699 and previous config saved to /var/cache/conftool/dbconfig/20210504-080206-marostegui.json

Stashbot added a comment.May 4 2021, 8:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-04T08:02:58Z] <marostegui> Check tables on db1106, lag will show up on s1 on wiki replicas (T280492)

• Marostegui added a comment.May 4 2021, 8:03 AM

Checking tables on db1106

gerritbot added a comment.May 4 2021, 8:04 AM

Change 684795 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1106: Disable notifications

https://gerrit.wikimedia.org/r/684795

gerritbot added a comment.May 4 2021, 8:05 AM

Change 684795 merged by Marostegui:

[operations/puppet@production] db1106: Disable notifications

https://gerrit.wikimedia.org/r/684795

Kormat updated the task description. (Show Details)May 4 2021, 12:52 PM
• Marostegui updated the task description. (Show Details)May 5 2021, 4:51 AM
• Marostegui mentioned this in T281937: heartbeat_p perhaps not showing correct replag on clouddb1013 and clouddb1017 s1.May 5 2021, 5:04 AM
Stashbot added a comment.May 5 2021, 6:41 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T06:40:59Z] <marostegui> Check tables on db1112 (lag might show up on s3 on wiki replicas) T280492

Stashbot added a comment.May 5 2021, 6:42 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-05T06:42:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1074 and db1156 to switch sanitarium hosts T280492', diff saved to https://phabricator.wikimedia.org/P15730 and previous config saved to /var/cache/conftool/dbconfig/20210505-064204-marostegui.json

• Marostegui added a comment.May 5 2021, 6:49 AM

s2 sanitarium master db1074 has been replaced by db1156

• Marostegui updated the task description. (Show Details)May 5 2021, 6:50 AM
• Marostegui added a subtask: T281959: decommission db1074.eqiad.wmnet.May 5 2021, 6:54 AM
• Marostegui updated the task description. (Show Details)May 6 2021, 4:49 AM
• Marostegui updated the task description. (Show Details)May 6 2021, 5:44 AM

s7 sanitarium master db1079 has been replaced by db1158

• Marostegui added a subtask: T282079: decommission db1079.eqiad.wmnet.May 6 2021, 5:46 AM
jcrespo added a comment.May 6 2021, 7:29 AM

I am going to remove the db2098 s3 10.1 instance, now that db2139 has been working fine for a while. A last backup of the old instance will be available on dbprov2002 until it is no longer recoverable (and we can always recover from logical backups).

gerritbot added a comment.May 6 2021, 7:36 AM

Change 685717 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: remove db2098 s3 section for this codfw backup source

https://gerrit.wikimedia.org/r/685717

gerritbot added a comment.May 6 2021, 7:50 AM

Change 685717 merged by Jcrespo:

[operations/puppet@production] dbbackups: remove db2098 s3 section for this codfw backup source

https://gerrit.wikimedia.org/r/685717

jcrespo added a comment.May 6 2021, 8:05 AM

db2098 s3 should be gone now, and will be soon gone from grafana/prometheus.

• Marostegui added a comment.May 6 2021, 8:48 AM

s8 sanitarium master db1087 has been replaced by db1167

• Marostegui updated the task description. (Show Details)May 6 2021, 8:48 AM
• Marostegui added a subtask: T282093: decommission db1087.eqiad.wmnet.
• Marostegui added a subtask: T282096: decommission db1085.eqiad.wmnet.May 6 2021, 8:52 AM
gerritbot added a comment.May 10 2021, 5:31 AM

Change 687965 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db1121 to Buster

https://gerrit.wikimedia.org/r/687965

gerritbot added a comment.May 10 2021, 5:32 AM

Change 687965 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db1121 to Buster

https://gerrit.wikimedia.org/r/687965

gerritbot added a comment.May 11 2021, 5:10 AM

Change 688742 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1121: Disable notifications

https://gerrit.wikimedia.org/r/688742

Stashbot added a comment.May 11 2021, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-11T05:11:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1121 - going to be reimaged to buster T280492', diff saved to https://phabricator.wikimedia.org/P15895 and previous config saved to /var/cache/conftool/dbconfig/20210511-051102-marostegui.json

Stashbot added a comment.May 11 2021, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-11T05:11:41Z] <marostegui> Reimage db1121 to buster, this will generate lag on s4 (commonswiki) on wikireplicas T280492

gerritbot added a comment.May 11 2021, 5:12 AM

Change 688742 merged by Marostegui:

[operations/puppet@production] db1121: Disable notifications

https://gerrit.wikimedia.org/r/688742

ops-monitoring-bot added a comment.May 11 2021, 5:19 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1121.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202105110519_marostegui_32061.log.

ops-monitoring-bot added a comment.May 11 2021, 5:40 AM

Completed auto-reimage of hosts:

['db1121.eqiad.wmnet']

and were ALL successful.

• Marostegui added a comment.May 11 2021, 5:42 AM

db1121 has been reimaged to Buster.
I am checking the tables now, this means commonswiki will show lag on wikireplicas.

• Marostegui updated the task description. (Show Details)May 11 2021, 8:30 AM
• Marostegui updated the task description. (Show Details)
• Cmjohnson closed subtask T281794: decommission db1082.eqiad.wmnet as Resolved.May 11 2021, 2:03 PM
• Marostegui added a comment.May 12 2021, 4:34 AM

db1121 is clean, replication restarted.

• Marostegui updated the task description. (Show Details)May 12 2021, 4:35 AM
• Marostegui added a comment.May 12 2021, 6:25 AM

db1121 is being automatically repooled

gerritbot added a comment.May 12 2021, 6:54 AM

Change 689523 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Switch db1112 to buster

https://gerrit.wikimedia.org/r/689523

gerritbot added a comment.May 12 2021, 6:55 AM

Change 689523 merged by Marostegui:

[operations/puppet@production] install_server: Switch db1112 to buster

https://gerrit.wikimedia.org/r/689523

Stashbot added a comment.May 17 2021, 4:35 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-17T04:35:52Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 T280492', diff saved to https://phabricator.wikimedia.org/P15975 and previous config saved to /var/cache/conftool/dbconfig/20210517-043551-marostegui.json

ops-monitoring-bot added a comment.May 17 2021, 4:40 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1112.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202105170439_marostegui_29261.log.

ops-monitoring-bot added a comment.May 17 2021, 5:00 AM

Completed auto-reimage of hosts:

['db1112.eqiad.wmnet']

and were ALL successful.

gerritbot added a comment.May 17 2021, 5:03 AM

Change 692111 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1112: Disable notifications

https://gerrit.wikimedia.org/r/692111

gerritbot added a comment.May 17 2021, 5:04 AM

Change 692111 merged by Marostegui:

[operations/puppet@production] db1112: Disable notifications

https://gerrit.wikimedia.org/r/692111

• Marostegui added a comment.May 17 2021, 5:05 AM

db1112 reimaged to Buster, - checking tables now.

• Marostegui added a comment.May 17 2021, 3:43 PM
In T280492#7091567, @Marostegui wrote:

db1112 reimaged to Buster, - checking tables now.

db1112 got all the tables checked, everything is clean. Replication started.

• Marostegui updated the task description. (Show Details)May 17 2021, 3:43 PM
gerritbot added a comment.May 17 2021, 3:44 PM

Change 692348 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db1106 to Buster

https://gerrit.wikimedia.org/r/692348

gerritbot added a comment.May 17 2021, 3:45 PM

Change 692348 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db1106 to Buster

https://gerrit.wikimedia.org/r/692348

gerritbot added a comment.May 17 2021, 5:07 PM

Change 681448 merged by Jcrespo:

[operations/puppet@production] mariadb: Remove s3 from db2098

https://gerrit.wikimedia.org/r/681448

ops-monitoring-bot added a comment.May 18 2021, 5:12 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1106.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202105180512_marostegui_20717.log.

gerritbot added a comment.May 18 2021, 5:18 AM

Change 692470 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1106: Disable notifications

https://gerrit.wikimedia.org/r/692470

gerritbot added a comment.May 18 2021, 5:19 AM

Change 692470 merged by Marostegui:

[operations/puppet@production] db1106: Disable notifications

https://gerrit.wikimedia.org/r/692470

ops-monitoring-bot added a comment.May 18 2021, 5:33 AM

Completed auto-reimage of hosts:

['db1106.eqiad.wmnet']

and were ALL successful.

• Marostegui added a comment.May 18 2021, 5:36 AM

db1106 reimaged. Checking its tables.

• Marostegui added a comment.May 19 2021, 4:41 AM

db1106 came back clean, replication restarted

• Marostegui updated the task description. (Show Details)May 19 2021, 5:57 AM
Stashbot added a comment.May 19 2021, 6:43 AM

Mentioned in SAL (#wikimedia-operations) [2021-05-19T06:43:44Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1106 T280492', diff saved to https://phabricator.wikimedia.org/P16082 and previous config saved to /var/cache/conftool/dbconfig/20210519-064343-marostegui.json

• Marostegui closed this task as Resolved.May 19 2021, 6:43 AM
• Marostegui updated the task description. (Show Details)

All sanitarium masters are now running Buster+10.4

• Marostegui mentioned this in T283131: Upgrade s3 to Debian Buster and MariaDB 10.4.May 19 2021, 6:54 AM
• Marostegui mentioned this in T283235: Upgrade s5 to Debian Buster and MariaDB 10.4.May 20 2021, 11:11 AM
• Cmjohnson closed subtask T281959: decommission db1074.eqiad.wmnet as Resolved.May 20 2021, 5:04 PM
• Cmjohnson closed subtask T282079: decommission db1079.eqiad.wmnet as Resolved.
• Cmjohnson closed subtask T282096: decommission db1085.eqiad.wmnet as Resolved.May 20 2021, 5:06 PM
• Cmjohnson closed subtask T282093: decommission db1087.eqiad.wmnet as Resolved.Jun 1 2021, 7:29 PM
• Marostegui mentioned this in T287230: Upgrade s2 to Buster + MariaDB 10.4.Jul 23 2021, 7:26 AM
Kormat mentioned this in T288244: Upgrade s7 to Debian Buster and MariaDB 10.4.Aug 5 2021, 2:32 PM
• Marostegui mentioned this in T288803: Upgrade s4 to Buster + MariaDB 10.4.Aug 13 2021, 7:53 AM
• Marostegui mentioned this in T290865: Upgrade s1 to Buster + MariaDB 10.4.Sep 13 2021, 12:51 PM
• Marostegui mentioned this in T290868: Upgrade s8 to Buster + MariaDB 10.4.Sep 13 2021, 1:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL