Page MenuHomePhabricator
Create Task
Maniphest T258361

Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
Closed, ResolvedPublic
Actions

Description

The following hosts are scheduled to be decommissioned in Q2 and need to be refreshed:
db1074-db1095 (22 servers)

We have started to see issues on them, especially BBU related: T258360 T258360

Replacement plan:

  • db1124 B1 (old sanitarium host, currently in use) to be placed in s7
  • db1125 D1 (old sanitarium host, currently in use) to replace db1077 in testing-s4
  • db1156 A1 to replace db1074 (sanitarium master)
  • db1157 A5 to replace db1075
  • db1158 A5 to replace db1079 (sanitarium master)
  • db1159 A6 to replace db1080 (m1 master)
  • db1160 A6 to replace db1081
  • db1161 A8 to replace db1082 (sanitarium master)
  • db1162 B1 to replace db1076 (candidate master)
  • db1163 B3 to replace db1083 (s1 master) CURRENTLY pooled on s1 as stretch to substitute db1134 T274472
  • db1164 B5 to replace db1084
  • db1165 B6 to replace db1085 (sanitarium master) T258361#6923913
  • db1166 C3 to replace db1078
  • db1167 C3 to replace db1087 (sanitarium master)
  • db1168 C5 to replace db1088
  • db1169 C5 to replace db1089
  • db1170 C6 to replace db1090 (multi-instance)
  • db1171 C6 to replace db1095
  • db1172 D1 to replace db1092
  • db1173 D3 to replace db1093 (candidate master)
  • db1174 D6 to replace db1094
  • db1175 D3 to be placed on s3

Decommissioning progress

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0instances.yaml: Add db1156 to dbctl
operations/puppetproduction+1 -0instances.yaml: Add db1167 to dbctl
operations/puppetproduction+0 -1db1167: Enable notifications
operations/puppetproduction+4 -2mariadb: Productionize db1167
operations/puppetproduction+1 -0instances.yaml: Add db1124 to dbctl
operations/puppetproduction+0 -1db1124: Enable notifications
operations/puppetproduction+2 -3check_private_data_report: Remove db1125
operations/puppetproduction+7 -7mariadb: Move db1125 from sanitarium to testing
operations/puppetproduction+1 -1install_server: Do not format db1124
operations/puppetproduction+1 -0instances.yaml: Add db1158
operations/puppetproduction+0 -1db1158: Enable notifications
operations/puppetproduction+3 -2mariadb: Productionize db1158
operations/puppetproduction+5 -11mariadb: Productionize db1124 into s7.
operations/puppetproduction+0 -1install_server: Reimage db1158 to buster
operations/puppetproduction+4 -2mariadb: Productionize db1156 on s2
operations/puppetproduction+2 -0db1124,db1125: Disable notifications
operations/puppetproduction+0 -1db1161: Enable notifications
operations/puppetproduction+1 -3install_server: Reimage db1124,db1125 to buster
operations/puppetproduction+1 -1install_server: Do not reimage db1165
operations/puppetproduction+1 -1install_server: Do not format db1164
operations/puppetproduction+1 -1install_server: Do not format db1161
operations/puppetproduction+1 -1install_server: Do not format db1159
operations/puppetproduction+0 -1db1165: Enable notifications
operations/puppetproduction+1 -0instances.yaml: Add db1165 to dbctl
operations/puppetproduction+1 -0db1165: Specify the future of this host
operations/puppetproduction+4 -2mariadb: Productionize db1161.
operations/puppetproduction+1 -0install_server: Reimage db1158 as Stretch
operations/puppetproduction+0 -1db1161: Enable notifications
operations/puppetproduction+1 -0install_server: Reimage db1156 as stretch
operations/puppetproduction+1 -0instances.yaml: Add db1161 to dbctl
operations/puppetproduction+4 -2mariadb: Productionize db1161
operations/puppetproduction+1 -1install_server: Do not reimage db1162
operations/puppetproduction+1 -0install_server: Reimage db1161 as stretch
operations/puppetproduction+0 -1db1162: Enable notifications
operations/puppetproduction+0 -1db1159: Enable notifications
operations/puppetproduction+8 -1mariadb: Productionize db1159
operations/puppetproduction+0 -1db1164: Enable notifications
operations/puppetproduction+1 -0instances.yaml: Add db1164 to dbctl
operations/puppetproduction+3 -2mariadb: Productionize db1164
operations/puppetproduction+1 -1install_server: Reimage db1162
operations/puppetproduction+1 -0instances.yaml: Add db1168 to dbctl
operations/puppetproduction+0 -1db1168: Enable notifications
operations/puppetproduction+3 -2mariadb: Productionize db1168
operations/puppetproduction+1 -0instances.yaml: Add db1172 to dbctl
operations/puppetproduction+0 -1db1172: Disable notifications
operations/puppetproduction+3 -2mariadb: Productionize db1172
operations/puppetproduction+1 -0instances.yaml: Add db1162 to dbctl
operations/puppetproduction+0 -1db1162: Enable notifications
operations/puppetproduction+2 -2mariadb: Reenable notifications for db1163 once it has been pooled
operations/puppetproduction+1 -0instances.yaml: Add db1163 to dbctl
operations/puppetproduction+5 -2db1163: Reimage to stretch to potentially become s1 candidate master
operations/puppetproduction+1 -1install_server: Do not reimage db1157
operations/puppetproduction+5 -2mariadb: Productionize db1162
operations/puppetproduction+2 -0instances.yaml: Add db1170 to dbctl
operations/puppetproduction+0 -1db1170: Enable notifications
operations/puppetproduction+1 -0install_server: Reimage db1162 to Stretch
operations/puppetproduction+0 -1db1157: Disable notifications
operations/puppetproduction+1 -0instances: Add db1157
operations/puppetproduction+8 -4mariadb: Productionize db1170
operations/puppetproduction+2 -2install_server: Disable disk format for db1171
operations/puppetproduction+5 -2mariadb: Productionize db1157
operations/puppetproduction+1 -0install_server: Reimage db1157 as stretch
operations/puppetproduction+1 -0instances.yaml: Add db1173 to dbctl
operations/puppetproduction+0 -1db1173: Enable notifications
operations/puppetproduction+5 -2mariadb: Productionize db1173
operations/puppetproduction+1 -0install_server: Reimage db1173 as stretch
operations/puppetproduction+1 -0instances.yaml: Add db1174.
operations/puppetproduction+3 -2mariadb: Productionize db1174
operations/puppetproduction+3 -2mariadb: Productionize db1174
operations/puppetproduction+9 -2backup-sources: Productionize db1171 to substitute db1095
operations/puppetproduction+1 -0install_server: Reimage db1171 into stretch so we still have 10.1 backups
operations/puppetproduction+1 -1mariadb: db1166 added to dbctl and notifications enabled
operations/puppetproduction+1 -0db1175: Add it to dbctl
operations/puppetproduction+0 -1db1175: Enable notifications
operations/puppetproduction+4 -2mariadb: Productionize db1166
operations/puppetproduction+4 -2mariadb: Productionize db1175
operations/puppetproduction+0 -1db1169: Enable notifications
operations/puppetproduction+1 -0instances.yaml: Add db1169
operations/puppetproduction+3 -2mariadb: Productionize db1169 in s1
operations/puppetproduction+1 -1install_server: Do not reimage db1160
operations/puppetproduction+0 -1db1160: Disable notifications
operations/puppetproduction+1 -0instances.yaml: Add db1160
operations/puppetproduction+5 -2mariadb: Productionize db1160
operations/puppetproduction+1 -0db1081: Disable notifications
operations/puppetproduction+1 -0db1160: Install stretch
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• MarosteguiT233766 labsdb1011 mariadb crashed
Restricted Task
Restricted Task
 OpenNoneT215858 Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema
 OpenNoneT280152 Mitigate breaking changes from the new Wiki Replicas architecture
Unknown Object (Task)
 ResolvedRobHT260441 (Need By: ASAP) rack/setup/install clouddb10[13-20]
 ResolvedBstormT260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture
 ResolvedBstormT260843 Set up roles for new wiki replicas layout
 Resolved• MarosteguiT265135 wikireplicas: Define MW sections per host
 Resolved• MarosteguiT267090 Productionize clouddb10[13-20]
 StalledNoneT302086 Set scap minimum python version to 3.7
 OpenNoneT247045 Migrate all of production metal and VMs to Buster or later
 Resolved• MarosteguiT250666 Upgrade WMF database-and-backup-related hosts to buster
Restricted Task
 ResolvedLegoktmT281515 June 2021 Datacenter switchover
 Resolved• MarosteguiT285139 DB maintenance work to do while eqiad is passive (June 2021)
 Resolved• MarosteguiT266483 Enable report_host for mariadb
Restricted Task
Restricted Task
Unknown Object (Task)
 Resolved• MarosteguiT258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
 DeclinedNoneT258386 db1080-95 batch possibly suffering BBU issues
 Resolved• MarosteguiT258336 db1082 crashed
 ResolvedJclark-ctrT258910 Remove db1082's BBU on-site
 Resolved• MarosteguiT258360 db1085 crashed
 Resolved• MarosteguiT264755 db1076 crashed - BBU failure
 ResolvedRequestwiki_willyT267088 decommission db1091.eqiad.wmnet
 Resolved• MarosteguiT271427 Switchover s4 (commonswiki) from db1081 to db1138
 ResolvedTrizek-WMFT271791 Database primary master failover on s4 (commonswiki)
 ResolvedKormatT272954 Fix db-switchover update zarcillo part
 Resolved• MarosteguiT274336 Switchover s7 from db1086 to db1136
 ResolvedsgrabarczukT276899 Read only time window needed for s7
 ResolvedCmjohnsonT277007 Upgrade firmware on db1136
 ResolvedCmjohnsonT262901 Physically move db1131 from B5 to C8
 ResolvedRobHT267043 (Need By: 2020-11-29) rack/setup/install db11[51-76]
 Resolved• MarosteguiT268742 Test upgrading sanitarium hosts to Buster + 10.4
 Resolved• MarosteguiT272008 Move wikireplicas under the new sanitarium hosts (db1154, db1155)
 Resolved• MarosteguiT280492 Upgrade all sanitarium masters to 10.4 and Buster
 ResolvedRequestwiki_willyT281794 decommission db1082.eqiad.wmnet
 ResolvedRequestCmjohnsonT281959 decommission db1074.eqiad.wmnet
 ResolvedRequestCmjohnsonT282079 decommission db1079.eqiad.wmnet
 ResolvedRequestCmjohnsonT282093 decommission db1087.eqiad.wmnet
 ResolvedRequestCmjohnsonT282096 decommission db1085.eqiad.wmnet
Restricted Task
 ResolvedRequestCmjohnsonT273040 decommission db1081.eqiad.wmnet
 ResolvedRequestCmjohnsonT273417 decommission db1089.eqiad.wmnet
 ResolvedRequestCmjohnsonT273597 decommission db1078.eqiad.wmnet
 ResolvedRequestCmjohnsonT273710 decommission db1094.eqiad.wmnet
 ResolvedRequestCmjohnsonT273732 decommission db1095
 ResolvedRequestCmjohnsonT273955 decommission db1093.eqiad.wmnet
 ResolvedRequestCmjohnsonT274333 decommission db1090.eqiad.wmnet
 ResolvedRequestCmjohnsonT274752 decommission db1076.eqiad.wmnet
 ResolvedCmjohnsonT275309 db1162 crashed
 ResolvedRequestCmjohnsonT276025 decommission db1088.eqiad.wmnet
 ResolvedRequestCmjohnsonT276302 decommission db1084.eqiad.wmnet
 Resolved• MarosteguiT276448 Failover m1 master: db1080 -> db1159 Wed 14th April at 10 AM UTC
 ResolvedRequestCmjohnsonT280121 decommission db1080.eqiad.mnet
 Resolved• MarosteguiT278214 Switchover s1 from db1083 to db1163
 Resolved• MarosteguiT276150 Schema change to make rc_id unsigned and rc_timestamp BINARY
 ResolvedsgrabarczukT279505 Read only time window needed for s1 (enwiki)
 ResolvedRequestCmjohnsonT281445 decommission db1083.eqiad.wmnet
 ResolvedRequestCmjohnsonT278229 decommission db1086.eqiad.wmnet
 ResolvedRequestCmjohnsonT281075 decommission db1077.eqiad.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Apr 26 2021, 5:40 AM

Change 682355 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1158

https://gerrit.wikimedia.org/r/682355

gerritbot added a comment.Apr 26 2021, 5:42 AM

Change 682355 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1158

https://gerrit.wikimedia.org/r/682355

ops-monitoring-bot added a comment.Apr 26 2021, 5:42 AM

Completed auto-reimage of hosts:

['db1124.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Apr 26 2021, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-26T05:47:01Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1158 to dbctl, depooled, T258361', diff saved to https://phabricator.wikimedia.org/P15521 and previous config saved to /var/cache/conftool/dbconfig/20210426-054700-marostegui.json

• Marostegui updated the task description. (Show Details)Apr 26 2021, 5:59 AM
• Marostegui added a subtask: T281075: decommission db1077.eqiad.wmnet.
• Marostegui added a comment.Apr 26 2021, 7:39 AM

checking tables on db1124 after the transfer from db1158

gerritbot added a comment.Apr 26 2021, 7:45 AM

Change 682495 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Do not format db1124

https://gerrit.wikimedia.org/r/682495

gerritbot added a comment.Apr 26 2021, 7:46 AM

Change 682495 merged by Marostegui:

[operations/puppet@production] install_server: Do not format db1124

https://gerrit.wikimedia.org/r/682495

gerritbot added a comment.Apr 26 2021, 10:14 AM

Change 682570 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move db1125 from sanitarium to testing

https://gerrit.wikimedia.org/r/682570

gerritbot added a comment.Apr 26 2021, 10:16 AM

Change 682570 merged by Marostegui:

[operations/puppet@production] mariadb: Move db1125 from sanitarium to testing

https://gerrit.wikimedia.org/r/682570

ops-monitoring-bot added a comment.Apr 26 2021, 10:17 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1125.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202104261017_marostegui_29781.log.

gerritbot added a comment.Apr 26 2021, 10:22 AM

Change 682571 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] check_private_data_report: Remove db1125

https://gerrit.wikimedia.org/r/682571

gerritbot added a comment.Apr 26 2021, 10:23 AM

Change 682571 merged by Marostegui:

[operations/puppet@production] check_private_data_report: Remove db1125

https://gerrit.wikimedia.org/r/682571

ops-monitoring-bot added a comment.Apr 26 2021, 10:39 AM

Completed auto-reimage of hosts:

['db1125.eqiad.wmnet']

and were ALL successful.

• Marostegui updated the task description. (Show Details)Apr 26 2021, 11:08 AM
gerritbot added a comment.Apr 27 2021, 4:38 AM

Change 682794 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1124: Enable notifications

https://gerrit.wikimedia.org/r/682794

• Marostegui updated the task description. (Show Details)Apr 27 2021, 4:38 AM
gerritbot added a comment.Apr 27 2021, 4:38 AM

Change 682794 merged by Marostegui:

[operations/puppet@production] db1124: Enable notifications

https://gerrit.wikimedia.org/r/682794

gerritbot added a comment.Apr 27 2021, 4:41 AM

Change 682795 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1124 to dbctl

https://gerrit.wikimedia.org/r/682795

gerritbot added a comment.Apr 27 2021, 4:43 AM

Change 682795 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1124 to dbctl

https://gerrit.wikimedia.org/r/682795

Stashbot added a comment.Apr 27 2021, 4:45 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-27T04:45:20Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1124 to dbctl, depooled, T258361', diff saved to https://phabricator.wikimedia.org/P15540 and previous config saved to /var/cache/conftool/dbconfig/20210427-044520-marostegui.json

Stashbot added a comment.Apr 27 2021, 4:46 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-27T04:46:10Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1124 with minimal weight for the first time in s7 T258361', diff saved to https://phabricator.wikimedia.org/P15541 and previous config saved to /var/cache/conftool/dbconfig/20210427-044609-marostegui.json

• Marostegui added a comment.Apr 27 2021, 4:46 AM

Pooled db1124 with minimal weight for the first time in s7

• Marostegui updated the task description. (Show Details)Apr 27 2021, 5:00 AM
Stashbot added a comment.Apr 27 2021, 5:08 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-27T05:08:27Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1124 with minimal weight for the first time in s7 T258361', diff saved to https://phabricator.wikimedia.org/P15545 and previous config saved to /var/cache/conftool/dbconfig/20210427-050826-marostegui.json

• Marostegui added a comment.Apr 27 2021, 5:18 AM

I am automatically pooling db1124 into s7.

• Marostegui updated the task description. (Show Details)Apr 27 2021, 5:18 AM
Stashbot added a comment.Apr 27 2021, 5:21 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-27T05:21:25Z] <marostegui> Stop mysql on db1087 to clone db1167 (lag will appear on wikidata on wikireplicas) T258361

gerritbot added a comment.Apr 27 2021, 7:01 AM

Change 682885 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1167

https://gerrit.wikimedia.org/r/682885

gerritbot added a comment.Apr 27 2021, 7:02 AM

Change 682885 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1167

https://gerrit.wikimedia.org/r/682885

• Marostegui added a comment.Apr 27 2021, 7:16 AM

Checking tables on db1167

gerritbot added a comment.Apr 28 2021, 4:32 AM

Change 683121 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1167: Enable notifications

https://gerrit.wikimedia.org/r/683121

gerritbot added a comment.Apr 28 2021, 4:33 AM

Change 683121 merged by Marostegui:

[operations/puppet@production] db1167: Enable notifications

https://gerrit.wikimedia.org/r/683121

gerritbot added a comment.Apr 28 2021, 5:49 AM

Change 683124 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1167 to dbctl

https://gerrit.wikimedia.org/r/683124

gerritbot added a comment.Apr 28 2021, 5:50 AM

Change 683124 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1167 to dbctl

https://gerrit.wikimedia.org/r/683124

Stashbot added a comment.Apr 28 2021, 5:51 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-28T05:51:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1167 in s8 T258361', diff saved to https://phabricator.wikimedia.org/P15605 and previous config saved to /var/cache/conftool/dbconfig/20210428-055144-marostegui.json

• Marostegui updated the task description. (Show Details)Apr 28 2021, 5:52 AM
gerritbot added a comment.Apr 29 2021, 4:36 AM

Change 683474 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1156 to dbctl

https://gerrit.wikimedia.org/r/683474

gerritbot added a comment.Apr 29 2021, 4:36 AM

Change 683474 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1156 to dbctl

https://gerrit.wikimedia.org/r/683474

Stashbot added a comment.Apr 29 2021, 4:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T04:38:13Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1156 to dbctl T258361', diff saved to https://phabricator.wikimedia.org/P15624 and previous config saved to /var/cache/conftool/dbconfig/20210429-043812-marostegui.json

Stashbot added a comment.Apr 29 2021, 4:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T04:38:57Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1156 into s2 for the first time with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P15625 and previous config saved to /var/cache/conftool/dbconfig/20210429-043857-marostegui.json

• Marostegui added a comment.Apr 29 2021, 4:39 AM

db1156 pooled in s2 with minimal weight

Stashbot added a comment.Apr 29 2021, 4:44 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T04:44:58Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1156 into s2 for the first time with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P15626 and previous config saved to /var/cache/conftool/dbconfig/20210429-044458-marostegui.json

Stashbot added a comment.Apr 29 2021, 4:50 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T04:50:15Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1156 into s2 for the first time with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P15627 and previous config saved to /var/cache/conftool/dbconfig/20210429-045015-marostegui.json

Stashbot added a comment.Apr 29 2021, 4:55 AM

Mentioned in SAL (#wikimedia-operations) [2021-04-29T04:55:57Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db1156 into s2 for the first time with minimal weight T258361', diff saved to https://phabricator.wikimedia.org/P15629 and previous config saved to /var/cache/conftool/dbconfig/20210429-045557-marostegui.json

• Marostegui added a comment.Apr 29 2021, 5:00 AM

Automatically pooling db1156 into s2.

• Marostegui updated the task description. (Show Details)Apr 29 2021, 5:01 AM

All the hosts in this task have been productionized. Pending: decommission the old ones.

• Marostegui closed subtask T278214: Switchover s1 from db1083 to db1163 as Resolved.Apr 29 2021, 6:55 AM
• Marostegui added a subtask: T281445: decommission db1083.eqiad.wmnet.Apr 29 2021, 6:57 AM
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)Apr 30 2021, 5:10 AM
• Marostegui added a subtask: T281794: decommission db1082.eqiad.wmnet.May 4 2021, 7:18 AM
• Marostegui updated the task description. (Show Details)
• Marostegui added a subtask: T281959: decommission db1074.eqiad.wmnet.May 5 2021, 6:54 AM
• Marostegui updated the task description. (Show Details)
• Marostegui added a subtask: T282079: decommission db1079.eqiad.wmnet.May 6 2021, 5:46 AM
• Marostegui updated the task description. (Show Details)
• Marostegui added a subtask: T282093: decommission db1087.eqiad.wmnet.May 6 2021, 8:49 AM
• Marostegui updated the task description. (Show Details)
• Marostegui closed this task as Resolved.May 6 2021, 8:52 AM
• Marostegui added a subtask: T282096: decommission db1085.eqiad.wmnet.
• Marostegui updated the task description. (Show Details)

All hosts that are scheduled for decommissioning are now ready (but waiting a few days to make sure their replacement work ok) and have their own decommissioning tasks.
Closing this as resolved.

• Marostegui updated the task description. (Show Details)May 10 2021, 5:09 AM
• Marostegui updated the task description. (Show Details)May 11 2021, 5:45 AM
Cmjohnson closed subtask T281794: decommission db1082.eqiad.wmnet as Resolved.May 11 2021, 2:03 PM
Cmjohnson closed subtask T281075: decommission db1077.eqiad.wmnet as Resolved.May 11 2021, 2:45 PM
Cmjohnson closed subtask T278229: decommission db1086.eqiad.wmnet as Resolved.
Cmjohnson closed subtask T274752: decommission db1076.eqiad.wmnet as Resolved.
• Marostegui updated the task description. (Show Details)May 12 2021, 9:21 AM
• Marostegui updated the task description. (Show Details)May 17 2021, 5:44 AM
• Marostegui changed the status of subtask T281445: decommission db1083.eqiad.wmnet from Stalled to Open.May 18 2021, 6:31 AM
• Marostegui updated the task description. (Show Details)May 18 2021, 6:42 AM
• Marostegui updated the task description. (Show Details)May 19 2021, 5:31 AM
• Marostegui closed subtask T280492: Upgrade all sanitarium masters to 10.4 and Buster as Resolved.May 19 2021, 6:44 AM
Cmjohnson closed subtask T281959: decommission db1074.eqiad.wmnet as Resolved.May 20 2021, 5:04 PM
Cmjohnson closed subtask T282079: decommission db1079.eqiad.wmnet as Resolved.
Cmjohnson closed subtask T282096: decommission db1085.eqiad.wmnet as Resolved.May 20 2021, 5:06 PM
Cmjohnson closed subtask T281445: decommission db1083.eqiad.wmnet as Resolved.
• Marostegui updated the task description. (Show Details)May 21 2021, 5:00 AM
Cmjohnson closed subtask T282093: decommission db1087.eqiad.wmnet as Resolved.Jun 1 2021, 7:29 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL