Page MenuHomePhabricator
Create Task
Maniphest T200209

Decom graphite2001/WMF6160
Closed, ResolvedPublic
Actions

Description

This task will track the decommission-hardware of graphite2001/WMF6160

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place. (handled by wmf-decommission-host)
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - update status in netbox (inventory for decom, inventory for spare) - updated to planned since we aren't certain of host's future.
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) - asw-b-codfw:ge-5/0/1 - asset tag set to port description
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries & mgmt dns entries for the hostname, leave mgmt dns entries for asset tag in palce
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox status to offline and remove from rack in netbox.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: asset tag mgmt dns entries removed.
  • - IF RECLAIM: remove the hostname label and system is already back in 'spares pool' by having the asset tag name in netbox and state of 'inventory'

Details

SubjectRepoBranchLines +/-
decom graphite2001 dns entriesoperations/dnsmaster+0 -4
graphite2001 decomoperations/puppetproduction+0 -10
DNS: Remove mgmt and production DNS for graphite2001operations/dnsmaster+0 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Jul 23 2018, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 23 2018, 3:42 PM
fgiunchedi mentioned this in T196483: rack/setup/install graphite2003.Jul 23 2018, 3:43 PM
fgiunchedi moved this task from Inbox to Up next on the observability board.Oct 15 2018, 2:38 PM
colewhite subscribed.Nov 27 2018, 8:36 PM
colewhite added a parent task: T209357: Return graphite100[13] to spares pool (or decom).Dec 3 2018, 4:43 PM
fgiunchedi moved this task from Up next to Externally blocked on the observability board.Dec 10 2018, 4:24 PM
RobH triaged this task as Medium priority.Dec 12 2018, 4:35 PM
RobH updated the task description. (Show Details)
RobH added a project: decommission-hardware.Dec 12 2018, 4:39 PM
RobH moved this task from Backlog to Blocked on Service Owners on the decommission-hardware board.Dec 12 2018, 10:00 PM
fgiunchedi added a comment.Jan 17 2019, 2:05 PM

I'm taking graphite2001 now to do some tests for prometheus v2 upgrade in T187987: 100% of Prometheus traffic served by Prometheus v2

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:12 PM
Dzahn moved this task from Backlog to Decommission on the ops-codfw board.Apr 12 2019, 12:10 AM
MoritzMuehlenhoff subscribed.Jun 18 2019, 9:08 AM
In T200209#4888109, @fgiunchedi wrote:

I'm taking graphite2001 now to do some tests for prometheus v2 upgrade in T187987: 100% of Prometheus traffic served by Prometheus v2

@fgiunchedi Can you confirm this is no longer needed for that?

fgiunchedi added a comment.Jun 25 2019, 9:44 AM
In T200209#5264664, @MoritzMuehlenhoff wrote:
In T200209#4888109, @fgiunchedi wrote:

I'm taking graphite2001 now to do some tests for prometheus v2 upgrade in T187987: 100% of Prometheus traffic served by Prometheus v2

@fgiunchedi Can you confirm this is no longer needed for that?

Yes confirmed, the host is good to be decom'd

MoritzMuehlenhoff reassigned this task from fgiunchedi to RobH.Jun 25 2019, 10:15 AM
MoritzMuehlenhoff updated the task description. (Show Details)
RobH moved this task from Blocked on Service Owners to Ready for Decommission on the decommission-hardware board.Jul 15 2019, 6:08 PM
RobH added a subscriber: wiki_willy.Aug 23 2019, 6:59 PM

I wanted to check with @wiki_willy if we need to reclaim this to spares, or if we can decommission and pull it out of the rack.

The host was purchased on 2015-01-09, and will be 5 years old (and automatically decommissioned) on 2020-01-09. Since that is less than 5 months away, and spare pool systems tend to sit for a few months before reallocation, it may be easiest/best to just decommission and unrack.

RobH updated the task description. (Show Details)Aug 23 2019, 7:06 PM
RobH updated the task description. (Show Details)Aug 23 2019, 7:09 PM
ops-monitoring-bot added a comment.Aug 23 2019, 7:11 PM

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: graphite2001.codfw.wmnet

  • graphite2001.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
RobH updated the task description. (Show Details)Aug 23 2019, 7:13 PM
gerritbot added a comment.Aug 23 2019, 7:16 PM

Change 531965 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom graphite2001 dns entries

https://gerrit.wikimedia.org/r/531965

gerritbot added a project: Patch-For-Review.Aug 23 2019, 7:16 PM
RobH renamed this task from Decom graphite2001 to Decom graphite2001/WMF6160.Aug 23 2019, 7:19 PM
RobH updated the task description. (Show Details)
wiki_willy added a subscriber: Papaul.Aug 23 2019, 7:22 PM

@RobH - I'll leave it up to @Papaul, since he has a better idea on the chances of reusing the parts on this system. Thanks, Willy

RobH added a comment.Aug 23 2019, 7:34 PM

Ok, I synced with @wiki_willy about this and the comment above.

We're going to decommission this host, since it has 5 months of life left before it is 5+ years old. Willy was stating that this can go on the decom pile, and be used for spare parts or just stay on the decom pile as needed/determined by @Papaul as the codfw on-site.

Thanks!

RobH added a comment.Aug 23 2019, 7:35 PM

https://gerrit.wikimedia.org/r/c/operations/dns/+/531965

statsd.codfw.wmnet points to graphite2001.codfw.wmnet, so I'm not sure what to point this at.

I assume it is fine to just remove it entirely, since it was pointing at a server in the spare role.

I've assigned the patchset to @fgiunchedi for his approval/review/comment.

gerritbot added a comment.Aug 23 2019, 7:38 PM

Change 531968 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] graphite2001 decom

https://gerrit.wikimedia.org/r/531968

RobH reassigned this task from RobH to Papaul.Aug 23 2019, 7:40 PM
RobH updated the task description. (Show Details)
RobH moved this task from Ready for Decommission to pending onsite steps (codfw) on the decommission-hardware board.
RobH subscribed.
Papaul updated the task description. (Show Details)Sep 4 2019, 3:56 PM
Papaul updated the task description. (Show Details)Sep 4 2019, 6:02 PM
papaul@asw-b-codfw# run show interfaces ge-5/0/1 descriptions       
Interface       Admin Link Description
ge-5/0/1        down  down DISABLED
gerritbot added a comment.Sep 4 2019, 6:08 PM

Change 534500 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt and production DNS for graphite2001

https://gerrit.wikimedia.org/r/534500

gerritbot added a comment.Sep 4 2019, 6:09 PM

Change 534500 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt and production DNS for graphite2001

https://gerrit.wikimedia.org/r/534500

Papaul updated the task description. (Show Details)Sep 4 2019, 6:09 PM
Papaul renamed this task from Decom graphite2001/WMF6160 to Decom graphite2001/WMF6160 replaced with WMF6403.Sep 5 2019, 5:14 PM
Papaul renamed this task from Decom graphite2001/WMF6160 replaced with WMF6403 to Decom graphite2001/WMF6160 .
Papaul closed this task as Resolved.Sep 9 2019, 3:00 PM

complete

MoritzMuehlenhoff mentioned this in T199321: Return graphite2001 to spares pool.Oct 15 2019, 8:19 AM
gerritbot added a comment.Sep 17 2020, 3:33 PM

Change 531968 abandoned by RobH:
[operations/puppet@production] graphite2001 decom

Reason:

https://gerrit.wikimedia.org/r/531968

gerritbot added a comment.Sep 17 2020, 3:35 PM

Change 531965 abandoned by RobH:
[operations/dns@master] decom graphite2001 dns entries

Reason:
old neglected patchset, no longer needed.

https://gerrit.wikimedia.org/r/531965

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits