Page MenuHomePhabricator
Create Task
Maniphest T328490

restbase cluster: decommission end-of-life hosts
Closed, ResolvedPublic
Actions

Description

Decommission (4) end-of-life hosts:

  • restbase1016
  • restbase1017
  • restbase1018
  • restbase2012

See: https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Active_-%3E_Decommissioned

This Cassandra cluster has 6 hosts in all 3 rows of eqiad, 6 hosts in codfw row d, but only 5 in codfw rows b & c. Perhaps we failed to provision some additional nodes after the decommission of restbase2010 & restbase2011?

Until/unless this is remedied, the effective capacity of the cluster is limited to a multiple of 5 (making the capacity of 4 of these hosts unusable).

Details

SubjectRepoBranchLines +/-
site.pp: cleanup decommissioned restbase hostsoperations/puppetproduction+0 -10
Move restbase canaryoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans created this task.Jan 31 2023, 8:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2023, 8:08 PM
Eevans renamed this task from restbase cluster: missing hosts in rows b & c? to restbase cluster: missing hosts in codfw rows b & c?.Jan 31 2023, 8:09 PM
Eevans triaged this task as Medium priority.
Eevans added a subscriber: KOfori.Apr 14 2023, 3:07 PM
Eevans added a comment.Apr 14 2023, 3:10 PM

@hnowlan Do you have any insight on this? Is there hardware somewhere that didn't get provisioned, or hardware that was decommissioned without replacements?

hnowlan added a comment.Apr 14 2023, 4:00 PM

I believe it's not that B and C have too few, it's that D has too many - restbase2012 is also out of warranty and should be decommissioned. Perhaps this is the source of this discrepancy? restbase101[678] are also of the same purchase date.

Eevans added a comment.Apr 14 2023, 4:06 PM
In T328490#8782361, @hnowlan wrote:

I believe it's not that B and C have too few, it's that D has too many - restbase2012 is also out of warranty and should be decommissioned. Perhaps this is the source of this discrepancy? restbase101[678] are also of the same purchase date.

So we provisioned 6 new hosts as part of a refresh, but only decommissioned 2?

hnowlan added a comment.Apr 14 2023, 4:49 PM

It seems that way yeah... there was some inconsistency to the process I realise: T294377 / T291991#7657098

Eevans added a comment.EditedJul 20 2023, 2:22 PM
hostrowrackpurchase date
restbase2013codfw row BB5DellPowerEdge R4402018-11-14
restbase2014codfw row BB8DellPowerEdge R4402018-11-14
restbase2019codfw row BB5DellPowerEdge R4402019-02-28
restbase2021codfw row BB3DellPowerEdge R4402019-12-26
restbase2024codfw row BB6DellPowerEdge R4402021-11-18
restbase2015codfw row CC1DellPowerEdge R4402018-11-14
restbase2016codfw row CC5DellPowerEdge R4402018-11-14
restbase2020codfw row CC5DellPowerEdge R4402019-02-28
restbase2022codfw row CC1DellPowerEdge R4402019-12-26
restbase2025codfw row CC5DellPowerEdge R4402021-11-18
restbase2012codfw row DD1DellPowerEdge R4302016-11-04
restbase2017codfw row DD1DellPowerEdge R4402018-11-14
restbase2018codfw row DD5DellPowerEdge R4402018-11-14
restbase2023codfw row DD8DellPowerEdge R4402019-12-26
restbase2026codfw row DD5DellPowerEdge R4402021-11-18
restbase2027codfw row DD5DellPowerEdge R4402021-02-26
restbase1016eqiad row AA3DellPowerEdge R4302016-11-02
restbase1019eqiad row AA3DellPowerEdge R4402019-03-01
restbase1020eqiad row AA5DellPowerEdge R4402019-03-01
restbase1021eqiad row AA6DellPowerEdge R4402019-03-01
restbase1028eqiad row AA5DellPowerEdge R4402020-01-17
restbase1031eqiad row AA6DellPowerEdge R440 - Restbase Config 2021072021-11-25
restbase1017eqiad row BB5DellPowerEdge R4302016-11-02
restbase1022eqiad row BB3DellPowerEdge R4402019-03-01
restbase1023eqiad row BB5DellPowerEdge R4402019-03-01
restbase1024eqiad row BB8DellPowerEdge R4402019-03-01
restbase1029eqiad row BB5DellPowerEdge R4402020-01-17
restbase1032eqiad row BB3DellPowerEdge R440 - Restbase Config 2021072021-11-25
restbase1025eqiad row DD3DellPowerEdge R4402019-03-01
restbase1026eqiad row DD3DellPowerEdge R4402019-03-01
restbase1027eqiad row DD6DellPowerEdge R4402019-03-01
restbase1030eqiad row DD4DellPowerEdge R4402020-01-17
restbase1033eqiad row DD1DellPowerEdge R440 - Restbase Config 2021072021-11-25
restbase1018eqiad row DD3DellPowerEdige R4402016-11-02
Eevans moved this task from Backlog to In-Progress on the Cassandra board.Sep 29 2023, 3:27 PM
Eevans renamed this task from restbase cluster: missing hosts in codfw rows b & c? to restbase cluster: decommission end-of-life hosts.Sep 29 2023, 3:29 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Sep 29 2023, 3:33 PM
Eevans moved this task from In-Progress to Next on the Cassandra board.Sep 29 2023, 4:23 PM
Eevans moved this task from Next to In-Progress on the Cassandra board.Oct 3 2023, 4:26 PM
Eevans added a parent task: T331713: Migrate restbase servers to Bullseye.
Stashbot added a comment.Oct 11 2023, 1:24 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-11T13:24:57Z] <urandom> starting decommission of restbase2012-a — T328490

gerritbot added a comment.Oct 11 2023, 2:27 PM

Change 965174 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move restbase canary

https://gerrit.wikimedia.org/r/965174

gerritbot added a project: Patch-For-Review.Oct 11 2023, 2:27 PM
gerritbot added a comment.Oct 11 2023, 2:53 PM

Change 965174 merged by Muehlenhoff:

[operations/puppet@production] Move restbase canary

https://gerrit.wikimedia.org/r/965174

Maintenance_bot removed a project: Patch-For-Review.Oct 11 2023, 3:11 PM
Stashbot added a comment.Oct 13 2023, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2023-10-13T11:53:51Z] <urandom> starting decommission of restbase2012-c — T328490

Stashbot added a comment.Oct 14 2023, 6:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-14T18:30:58Z] <urandom> starting Cassandra decommission of restbase1016-a — T328490

Stashbot added a comment.Oct 15 2023, 7:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-15T19:10:42Z] <urandom> starting Cassandra decommission of restbase1016-b — T328490

Stashbot added a comment.Oct 19 2023, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-19T17:33:49Z] <urandom> Decommissioning Cassandra, restbase1018-{a,b,c} — T328490

Eevans added a subtask: T349526: decommission restbase2012.codfw.wmnet.Oct 23 2023, 3:14 PM
Eevans updated the task description. (Show Details)Oct 24 2023, 6:55 PM
Eevans added a subtask: T349711: decommission restbase1018.Oct 25 2023, 7:50 PM
Eevans added a subtask: T349710: decommission restbase1017.
Eevans added a subtask: T349709: decommission restbase1016.
Eevans closed this task as Resolved.Oct 25 2023, 7:58 PM
Eevans claimed this task.
Eevans updated the task description. (Show Details)

macro-deployed

gerritbot added a comment.Oct 25 2023, 8:07 PM

Change 968751 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] site.pp: cleanup decommissioned restbase hosts

https://gerrit.wikimedia.org/r/968751

gerritbot added a project: Patch-For-Review.Oct 25 2023, 8:07 PM
gerritbot added a comment.Oct 25 2023, 9:07 PM

Change 968751 merged by Eevans:

[operations/puppet@production] site.pp: cleanup decommissioned restbase hosts

https://gerrit.wikimedia.org/r/968751

Maintenance_bot removed a project: Patch-For-Review.Oct 25 2023, 9:10 PM
Jclark-ctr closed subtask T349711: decommission restbase1018 as Resolved.Oct 26 2023, 2:03 PM
Jclark-ctr closed subtask T349710: decommission restbase1017 as Resolved.
Jclark-ctr closed subtask T349709: decommission restbase1016 as Resolved.
Jhancock.wm closed subtask T349526: decommission restbase2012.codfw.wmnet as Resolved.Nov 1 2023, 8:06 PM
Eevans mentioned this in T353550: Cassandra (logstash) logging broken.Dec 15 2023, 8:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL