Page MenuHomePhabricator
Create Task
Maniphest T358882

Decommission elastic2037-2054
Closed, ResolvedPublic
Actions

Description

Hosts elastic2037-2054 have reached the end of their 5-year lifespan.

Creating this ticket to track their decommissioning.

Details

Other Assignee
RKemper
SubjectRepoBranchLines +/-
elastic: decom elastic20[37-54]operations/puppetproduction+0 -17
elasticsearch: remove soon-to-be-decommed codfw hostsoperations/puppetproduction+27 -79
elastic: replace some mastersoperations/puppetproduction+7 -8
elastic: move elastic2037 to insetupoperations/puppetproduction+5 -1
elastic-codfw: Add new master-eligiblesoperations/puppetproduction+3 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

bking created this task.Mar 1 2024, 5:28 PM
Gehel triaged this task as High priority.Mar 5 2024, 10:42 AM
bking mentioned this in T359742: Degraded RAID on elastic2037.Mar 19 2024, 8:56 PM
bking added a subtask: T313842: Decommission elastic2049.codfw.wmnet.Mar 20 2024, 6:37 PM
gerritbot added a comment.Mar 21 2024, 8:13 PM

Change #1013398 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic-codfw: Add new master-eligibles

https://gerrit.wikimedia.org/r/1013398

gerritbot added a project: Patch-For-Review.Mar 21 2024, 8:13 PM
gerritbot added a comment.Mar 21 2024, 8:16 PM

Change #1013398 merged by Bking:

[operations/puppet@production] elastic-codfw: Add new master-eligibles

https://gerrit.wikimedia.org/r/1013398

gerritbot added a comment.Mar 21 2024, 8:29 PM

Change #1013401 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: move elastic2037 to insetup

https://gerrit.wikimedia.org/r/1013401

gerritbot added a comment.Mar 21 2024, 8:37 PM

Change #1013401 merged by Bking:

[operations/puppet@production] elastic: move elastic2037 to insetup

https://gerrit.wikimedia.org/r/1013401

Stashbot added a comment.Mar 22 2024, 6:22 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-22T06:22:19Z] <ryankemper> T358882 Updating cross-cluster seeds to bring into concordance with newly added masters: ryankemper@mwmaint1002:~/elastic$ python push_cross_cluster_conf.py https://search.svc.codfw.wmnet:9643/_cluster/settings --ccc chi=chi_codfw_masters.lst psi=psi_codfw_masters.lst omega=omega_codfw_masters.lst

Stashbot added a comment.Mar 22 2024, 6:33 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-22T06:33:10Z] <ryankemper> T358882 Also updated cross-cluster seeds for ports 9243 and 9443. Everything should be as expected now.

Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.04 - 2024.03.24) board.Mar 22 2024, 8:46 AM
Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE (2024.03.04 - 2024.03.24).Mar 22 2024, 8:47 AM
bking closed this task as Invalid.Mar 26 2024, 4:49 PM
bking reopened this task as In Progress.
bking claimed this task.
bking lowered the priority of this task from High to Medium.
bking moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.
bking updated Other Assignee, added: RKemper.
gerritbot added a comment.Mar 26 2024, 7:46 PM

Change #1014600 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: replace some masters

https://gerrit.wikimedia.org/r/1014600

gerritbot added a comment.Mar 26 2024, 7:53 PM

Change #1014600 merged by Ryan Kemper:

[operations/puppet@production] elastic: replace some masters

https://gerrit.wikimedia.org/r/1014600

Stashbot added a comment.Mar 26 2024, 8:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-26T20:09:20Z] <ryankemper@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: cycle some masters - ryankemper@cumin2002 - T358882

Stashbot added a comment.Mar 26 2024, 9:45 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-26T21:45:32Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: cycle some masters - ryankemper@cumin2002 - T358882

Stashbot added a comment.Mar 27 2024, 1:06 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T01:06:46Z] <ryankemper> T358882 Updated remote cluster seeds for new master state

Stashbot added a comment.Mar 27 2024, 1:35 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T01:35:23Z] <ryankemper@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2037*,elastic2038*,elastic2041*,elastic2042*,elastic2045*,elastic2046*,elastic2047*,elastic2050*,elastic2051*,elastic2052*,elastic2039*,elastic2040*,elastic2043*,elastic2044*,elastic2048*,elastic2053*,elastic2054* for prepare for decom of hosts - ryankemper@cumin2002 - T358882

Stashbot added a comment.Mar 27 2024, 1:35 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T01:35:28Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic2037*,elastic2038*,elastic2041*,elastic2042*,elastic2045*,elastic2046*,elastic2047*,elastic2050*,elastic2051*,elastic2052*,elastic2039*,elastic2040*,elastic2043*,elastic2044*,elastic2048*,elastic2053*,elastic2054* for prepare for decom of hosts - ryankemper@cumin2002 - T358882

Volans subscribed.Mar 27 2024, 10:12 AM

elastic2037 is reported by Netbox for not being anymore in puppetdb, please either decommission it or shut it down. No host should be powered on without puppet running for extensive period of time.

Stashbot added a comment.Mar 27 2024, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:51:40Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 12 days, 0:00:00 on elastic2038.codfw.wmnet with reason: T358882

Stashbot added a comment.Mar 27 2024, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-27T15:51:44Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12 days, 0:00:00 on elastic2038.codfw.wmnet with reason: T358882

gerritbot added a comment.Mar 27 2024, 9:35 PM

Change #1015119 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: remove soon-to-be-decommed codfw hosts

https://gerrit.wikimedia.org/r/1015119

gerritbot added a comment.Mar 27 2024, 9:41 PM

Change #1015119 merged by Bking:

[operations/puppet@production] elasticsearch: remove soon-to-be-decommed codfw hosts

https://gerrit.wikimedia.org/r/1015119

gerritbot added a comment.Mar 27 2024, 9:48 PM

Change #1015123 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] elastic: decom elastic20[37-54]

https://gerrit.wikimedia.org/r/1015123

gerritbot added a comment.Mar 27 2024, 9:49 PM

Change #1015123 merged by Bking:

[operations/puppet@production] elastic: decom elastic20[37-54]

https://gerrit.wikimedia.org/r/1015123

ops-monitoring-bot added a comment.Mar 28 2024, 5:15 PM

cookbooks.sre.hosts.decommission executed by ryankemper@cumin2002 for hosts: elastic[2052-2054].codfw.wmnet

  • elastic2052.codfw.wmnet (FAIL)
    • Missing DNSName in Nebox for elastic2052, unable to verify it.
    • Unable to find/resolve the mgmt DNS record, using the IP instead: 10.193.3.58
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Host is already powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • elastic2053.codfw.wmnet (FAIL)
    • Missing DNSName in Nebox for elastic2053, unable to verify it.
    • Unable to find/resolve the mgmt DNS record, using the IP instead: 10.193.3.59
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Host is already powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • elastic2054.codfw.wmnet (FAIL)
    • Missing DNSName in Nebox for elastic2054, unable to verify it.
    • Unable to find/resolve the mgmt DNS record, using the IP instead: 10.193.3.60
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Host is already powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Mar 28 2024, 5:27 PM

cookbooks.sre.hosts.decommission executed by ryankemper@cumin2002 for hosts: elastic2037.codfw.wmnet

  • elastic2037.codfw.wmnet (FAIL)
    • Host not found on Icinga, unable to downtime it
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

RKemper added a subtask: T361305: decommission elastic20[37-54].codfw.wmnet.Mar 28 2024, 8:41 PM
bking mentioned this in T361306: Decommission cookbook: stop when user inputs "abort".Mar 28 2024, 9:33 PM
Jhancock.wm closed subtask T361305: decommission elastic20[37-54].codfw.wmnet as Resolved.Mon, Apr 1, 4:56 PM
bking mentioned this in T356887: Review and decom all Search Platform servers past the 5-year rotation date.Mon, Apr 1, 8:06 PM
RKemper closed this task as Resolved.Tue, Apr 2, 6:11 AM
RKemper moved this task from In Progress to Done on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.

With dc-ops having closed out the decom subtask, this should be all done.

dcausse reopened subtask T361305: decommission elastic20[37-54].codfw.wmnet as Open.Mon, Apr 8, 8:12 AM
bking closed subtask T361305: decommission elastic20[37-54].codfw.wmnet as Resolved.Tue, Apr 16, 4:50 PM
Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 6:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL