Page MenuHomePhabricator
Create Task
Maniphest T189727

decommission elastic1021
Closed, ResolvedPublic
Actions

Description

elastic1021 has a failed dimm, recorded on T188595. The discovery team reviewed options with @RobH & @faidon and came to the decision to simply decommission this particular server.

elastic1021:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) - asw-c-eqiad:ge-4/0/35
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
decom elastic1021 prod dnsoperations/dnsmaster+0 -0
Removing dns for elastic1021operations/dnsmaster+2 -6
decom elastic1021operations/puppetproduction+0 -11
elastic: decommission elastic1021operations/puppetproduction+11 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRobHT188595 Memory test failure on elastic1021
 Resolved• CmjohnsonT189727 decommission elastic1021

Event Timeline

RobH triaged this task as Medium priority.Mar 14 2018, 7:13 PM
RobH created this task.
RobH added a project: hardware-requests.Mar 14 2018, 7:15 PM

So this system was already offline for memory testing, but I want to confirm with @Gehel we're good to start decommission, which includes wipe of data.

@Gehel: Can you confirm you don't need any information off this host? Additionally, please ensure no other configuration files expect it. I can take over after ' - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.'

Thanks!

RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Mar 14 2018, 7:15 PM
RobH mentioned this in T188595: Memory test failure on elastic1021.
gerritbot subscribed.Mar 15 2018, 10:08 AM

Change 419702 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elastic: decommission elastic1021

https://gerrit.wikimedia.org/r/419702

gerritbot added a project: Patch-For-Review.Mar 15 2018, 10:08 AM
Gehel updated the task description. (Show Details)Mar 15 2018, 10:08 AM
Gehel added a comment.Mar 15 2018, 10:13 AM

Preliminary decommissioning steps are done (pending the merge of https://gerrit.wikimedia.org/r/#/c/419702/). A few notes:

Since elastic1021 is down and does not want to restart, the services normally running on it have not been stopped / masked. Also, modifying the icinga checks would require hacking into puppetdb. The checks are silences and will be purged during the puppet node cleanup phase.

From a service perspective, that host is banned from the cluster and marked as inactive in conftool. Even if it comes back from the deads for whatever reason, it should not cause any trouble.

Gehel moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Mar 15 2018, 1:05 PM
RobH claimed this task.Mar 15 2018, 3:26 PM

Thakns @Gehel, I'll steal and proceed from here!

gerritbot added a comment.Mar 15 2018, 5:47 PM

Change 419702 merged by Gehel:
[operations/puppet@production] elastic: decommission elastic1021

https://gerrit.wikimedia.org/r/419702

RobH updated the task description. (Show Details)Mar 15 2018, 5:58 PM

Since the host is down, I cannot power it on and disable puppet. I have disabled the switch port, so if it does power on it will be fine.

gerritbot added a comment.Mar 15 2018, 6:15 PM

Change 419811 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom elastic1021

https://gerrit.wikimedia.org/r/419811

gerritbot added a comment.Mar 15 2018, 6:16 PM

Change 419811 merged by RobH:
[operations/puppet@production] decom elastic1021

https://gerrit.wikimedia.org/r/419811

gerritbot added a comment.Mar 15 2018, 6:18 PM

Change 419813 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom elastic1021 prod dns

https://gerrit.wikimedia.org/r/419813

RobH reassigned this task from RobH to • Cmjohnson.Mar 15 2018, 6:19 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

This is now ready for disk wipe and unracking. Please note that since the system won't power on, you'll need to move the disks to a working system to wipe. (Perhaps another system you are already doing disk wipes on.)

Thanks!

• Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Mar 28 2018, 5:57 PM
Gehel removed a project: Discovery-Search (Current work).Apr 17 2018, 5:34 PM

Removing search backend team from this ticket, nothing left to do on our side.

gerritbot added a comment.Apr 18 2018, 4:19 PM

Change 427420 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing dns for elastic1021

https://gerrit.wikimedia.org/r/427420

gerritbot added a project: Patch-For-Review.Apr 18 2018, 4:19 PM

Change 427420 merged by Cmjohnson:
[operations/dns@master] Removing dns for elastic1021

https://gerrit.wikimedia.org/r/427420

• Cmjohnson closed this task as Resolved.Apr 18 2018, 4:23 PM
• Cmjohnson updated the task description. (Show Details)

Updated rackables and tracking sheet.

gerritbot added a comment.Jul 20 2018, 6:55 PM

Change 419813 abandoned by RobH:
decom elastic1021 prod dns

https://gerrit.wikimedia.org/r/419813

Gehel mentioned this in T239821: decommission elastic10[18-31].eqiad.wmnet.Dec 4 2019, 2:42 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 4 2019, 3:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL