Page MenuHomePhabricator
Create Task
Maniphest T228258

Decommission db2043-db2070
Closed, ResolvedPublic
Actions

Description

The following hosts are scheduled to be decommissioned by January 2020.
This task will track its preparation to be sent to DCOps for final decommissioning

Some of them already have hardware problems and should be decommissioned

Details

SubjectRepoBranchLines +/-
db2048,db2061: Disable notificationsoperations/puppetproduction+2 -0
instances.yaml: Remove db2048,db2061operations/puppetproduction+0 -2
db2046: Disable notificationsoperations/puppetproduction+1 -0
db-eqiad,db-codfw.php: Remove db2049 from configoperations/mediawiki-configmaster+0 -3
mariadb: Promote db2096 to x1 codfw masteroperations/puppetproduction+3 -3
db-codfw.php: Promote db2096 to x1 codfw masteroperations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT208323 Predictive failures on disk S.M.A.R.T. status
 ResolvedMarosteguiT228258 Decommission db2043-db2070
 DeclinedNoneT227862 (OoW) db2045 failed battery
 ResolvedRequestPapaulT228281 decommission db2045.codfw.wmnet
 ResolvedPapaulT229543 Decommission db2058.codfw.wmnet
 ResolvedPapaulT230107 Decommission db2069.codfw.wmnet
 ResolvedPapaulT230311 Decommission db2043.codfw.wmnet
 ResolvedPapaulT230391 Decommission db2050.codfw.wmnet
 ResolvedPapaulT230394 Decommission db2057.codfw.wmnet
 ResolvedPapaulT230704 Decommission db2063.codfw.wmnet
 ResolvedPapaulT230721 Decommission db2049.codfw.wmnet
 ResolvedPapaulT230761 Decommission db2044.codfw.wmnet
 ResolvedPapaulT230777 Decommission db2056.codfw.wmnet
 ResolvedPapaulT230778 Decommission db2051.codfw.wmnet
 ResolvedPapaulT230883 Decommission db2052.codfw.wmnet
 ResolvedPapaulT230884 Decommission db2059.codfw.wmnet
 ResolvedPapaulT230885 Decommission db2066.codfw.wmnet
 ResolvedPapaulT231407 Decommission db2053.codfw.wmnet
 ResolvedPapaulT231625 Decommission db2060.codfw.wmnet
 ResolvedPapaulT231767 Decommission db2046.codfw.wmnet
 ResolvedPapaulT231852 Decommission db2047.codfw.wmnet
 ResolvedPapaulT232969 Decommission db2054.codfw.wmnet
 ResolvedPapaulT233185 Decommission db2067.codfw.wmnet
 ResolvedPapaulT233186 Decommission db2055.codfw.wmnet
 ResolvedPapaulT235399 Decommission db2068.codfw.wmnet
 ResolvedPapaulT237913 Decommission db2048.codfw.wmnet
 ResolvedPapaulT238526 Decommission db2061.codfw.wmnet
 ResolvedPapaulT238726 Decommission db2062.codfw.wmnet
 ResolvedRequestPapaulT239046 decommission db2065.codfw.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui updated the task description. (Show Details)Aug 21 2019, 7:55 AM
Marostegui updated the task description. (Show Details)Aug 21 2019, 8:13 AM
Marostegui updated the task description. (Show Details)Aug 22 2019, 5:30 AM
Marostegui updated the task description. (Show Details)Aug 23 2019, 5:17 AM
gerritbot added a comment.Aug 27 2019, 7:32 AM

Change 532545 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2046: Disable notifications

https://gerrit.wikimedia.org/r/532545

gerritbot added a project: Patch-For-Review.Aug 27 2019, 7:32 AM

Change 532545 merged by Marostegui:
[operations/puppet@production] db2046: Disable notifications

https://gerrit.wikimedia.org/r/532545

Maintenance_bot removed a project: Patch-For-Review.Aug 27 2019, 8:10 AM
Marostegui updated the task description. (Show Details)Aug 29 2019, 5:02 AM
Papaul closed subtask T228281: decommission db2045.codfw.wmnet as Resolved.Aug 29 2019, 4:09 PM
Marostegui updated the task description. (Show Details)Aug 30 2019, 5:12 AM
Marostegui updated the task description. (Show Details)Aug 30 2019, 5:26 AM
Marostegui updated the task description. (Show Details)Sep 3 2019, 6:31 AM
Marostegui updated the task description. (Show Details)Sep 6 2019, 8:35 AM
Marostegui updated the task description. (Show Details)Sep 10 2019, 5:37 AM
Marostegui updated the task description. (Show Details)Sep 16 2019, 6:05 AM
Marostegui changed the status of subtask T233185: Decommission db2067.codfw.wmnet from Open to Stalled.Sep 18 2019, 5:20 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Sep 18 2019, 5:49 AM
Marostegui updated the task description. (Show Details)Sep 19 2019, 5:13 AM
Papaul closed subtask T230311: Decommission db2043.codfw.wmnet as Resolved.Oct 1 2019, 10:39 PM
Papaul closed subtask T230761: Decommission db2044.codfw.wmnet as Resolved.Oct 1 2019, 10:46 PM
Papaul closed subtask T231767: Decommission db2046.codfw.wmnet as Resolved.Oct 1 2019, 10:51 PM
Papaul closed subtask T231852: Decommission db2047.codfw.wmnet as Resolved.
Papaul closed subtask T230721: Decommission db2049.codfw.wmnet as Resolved.Oct 1 2019, 10:57 PM
Papaul closed subtask T232969: Decommission db2054.codfw.wmnet as Resolved.Oct 4 2019, 12:10 AM
Papaul closed subtask T233186: Decommission db2055.codfw.wmnet as Resolved.Oct 9 2019, 8:19 PM
Papaul closed subtask T230391: Decommission db2050.codfw.wmnet as Resolved.
Papaul closed subtask T229543: Decommission db2058.codfw.wmnet as Resolved.Oct 11 2019, 1:32 AM
Papaul closed subtask T230107: Decommission db2069.codfw.wmnet as Resolved.
Papaul closed subtask T230394: Decommission db2057.codfw.wmnet as Resolved.Oct 11 2019, 8:27 PM
Papaul closed subtask T230704: Decommission db2063.codfw.wmnet as Resolved.
Marostegui updated the task description. (Show Details)Oct 14 2019, 5:27 AM
Marostegui updated the task description. (Show Details)Oct 14 2019, 7:14 AM
Papaul closed subtask T230778: Decommission db2051.codfw.wmnet as Resolved.Oct 16 2019, 11:17 PM
Papaul closed subtask T230777: Decommission db2056.codfw.wmnet as Resolved.
Papaul closed subtask T235399: Decommission db2068.codfw.wmnet as Resolved.Oct 16 2019, 11:19 PM
gerritbot added a comment.Oct 21 2019, 5:40 AM

Change 544615 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] instances.yaml: Remove db2048,db2061

https://gerrit.wikimedia.org/r/544615

gerritbot added a project: Patch-For-Review.Oct 21 2019, 5:40 AM

Change 544615 merged by Marostegui:
[operations/puppet@production] instances.yaml: Remove db2048,db2061

https://gerrit.wikimedia.org/r/544615

Stashbot added a comment.Oct 21 2019, 5:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T05:43:41Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db2048 and db2061, those hosts will be decommissioned T228258', diff saved to https://phabricator.wikimedia.org/P9400 and previous config saved to /var/cache/conftool/dbconfig/20191021-054340-marostegui.json

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2019, 6:10 AM
Marostegui updated the task description. (Show Details)Oct 24 2019, 5:02 AM
gerritbot added a comment.Oct 24 2019, 5:06 AM

Change 545700 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2048,db2061: Disable notifications

https://gerrit.wikimedia.org/r/545700

gerritbot added a project: Patch-For-Review.Oct 24 2019, 5:06 AM

Change 545700 merged by Marostegui:
[operations/puppet@production] db2048,db2061: Disable notifications

https://gerrit.wikimedia.org/r/545700

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 5:10 AM
Papaul closed subtask T230885: Decommission db2066.codfw.wmnet as Resolved.Oct 28 2019, 2:20 PM
jcrespo subscribed.EditedOct 29 2019, 10:17 AM

I noticed db2062 isn't set on m1, is that on purpose because it is going to be decommissioned? Or because we didn't want it to alert? Or something else? CC @Marostegui

Cannot be because db1135 also doesn't have the check 🤔

Papaul closed subtask T230883: Decommission db2052.codfw.wmnet as Resolved.Oct 31 2019, 9:50 PM
Papaul closed subtask T231407: Decommission db2053.codfw.wmnet as Resolved.
Papaul closed subtask T230884: Decommission db2059.codfw.wmnet as Resolved.
Papaul closed subtask T231625: Decommission db2060.codfw.wmnet as Resolved.Oct 31 2019, 9:52 PM
Marostegui added a comment.Nov 11 2019, 7:28 AM
In T228258#5614366, @jcrespo wrote:

I noticed db2062 isn't set on m1, is that on purpose because it is going to be decommissioned? Or because we didn't want it to alert? Or something else? CC @Marostegui

Cannot be because db1135 also doesn't have the check 🤔

db2062 will be decommissioned once the new hosts arrive and get setup

jcrespo added a comment.Nov 11 2019, 8:47 AM

What about db1135, then (probably others)? Is it on purpose or WIP, or something else? I didn't want to break anything, sorry.

Marostegui updated the task description. (Show Details)Nov 12 2019, 7:00 AM
Marostegui renamed this task from Decommission db2043-db2069 to Decommission db2043-db2070.Nov 18 2019, 10:23 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Nov 20 2019, 8:16 AM
Marostegui updated the task description. (Show Details)Nov 20 2019, 8:18 AM
Marostegui changed the status of subtask T233185: Decommission db2067.codfw.wmnet from Stalled to Open.Nov 20 2019, 9:25 AM
Papaul closed subtask T238526: Decommission db2061.codfw.wmnet as Resolved.Nov 22 2019, 11:18 PM
Papaul closed subtask T237913: Decommission db2048.codfw.wmnet as Resolved.
Marostegui added a subtask: T239046: decommission db2065.codfw.wmnet.Nov 25 2019, 7:28 AM
Marostegui updated the task description. (Show Details)
Marostegui mentioned this in T238183: Productionize db213[2-5}.Nov 27 2019, 6:22 AM
Marostegui updated the task description. (Show Details)Nov 29 2019, 6:19 AM
Volans reopened subtask T238526: Decommission db2061.codfw.wmnet as Open.Nov 30 2019, 6:29 PM
Papaul closed subtask T238526: Decommission db2061.codfw.wmnet as Resolved.Nov 30 2019, 6:40 PM
Marostegui updated the task description. (Show Details)Dec 3 2019, 5:56 AM
Marostegui updated the task description. (Show Details)Dec 3 2019, 6:22 AM
Marostegui updated the task description. (Show Details)Dec 5 2019, 7:46 AM
Papaul closed subtask T238726: Decommission db2062.codfw.wmnet as Resolved.Dec 6 2019, 12:21 AM
Papaul closed subtask T233185: Decommission db2067.codfw.wmnet as Resolved.
Marostegui closed this task as Resolved.Dec 12 2019, 6:01 AM
Marostegui updated the task description. (Show Details)

All these hosts have been decommissioned or pending the last on-site decommissioning steps.

jcrespo awarded a token.Dec 12 2019, 9:04 AM
Papaul closed subtask T239046: decommission db2065.codfw.wmnet as Resolved.Jan 3 2020, 10:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits