Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return)
Closed, ResolvedPublic

Description

pc2004 pc2005 and pc2006 leases expires the 31st Dec 2018 and they need to be returned (T204556)
The new hosts are online (T208383)

pc2004

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • -this is a lease, so this should be done ASAP for return in December!
  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.
  • - set aside for lease return for this month (December 2018)

pc2005

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • -this is a lease, so this should be done ASAP for return in December!
  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.
  • - set aside for lease return for this month (December 2018)

pc2006

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • -this is a lease, so this should be done ASAP for return in December!
  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.
  • - set aside for lease return for this month (December 2018)
Restricted Application added a project: Operations. · View Herald TranscriptMon, Nov 19, 4:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui moved this task from Triage to In progress on the DBA board.Mon, Nov 19, 4:22 PM
Marostegui triaged this task as Normal priority.
Marostegui added a parent task: Unknown Object (Task).
Marostegui renamed this task from Decommission parsercache hosts: pc2006 pc2007 pc2008 to Decommission parsercache hosts: pc2004 pc2005 pc2006.Mon, Nov 19, 4:28 PM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Mon, Nov 19, 4:33 PM

Change 474847 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Get ready to decom pc2004,pc2005,pc2006

https://gerrit.wikimedia.org/r/474847

Change 474847 merged by Marostegui:
[operations/puppet@production] mariadb: Get ready to decom pc2004,pc2005,pc2006

https://gerrit.wikimedia.org/r/474847

Marostegui updated the task description. (Show Details)Tue, Nov 20, 9:03 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-20T09:04:16Z] <marostegui> Remove pc2004, pc2005 and pc2006 from tendril and zarcillo - T209858

Mentioned in SAL (#wikimedia-operations) [2018-11-20T09:12:59Z] <marostegui> Stop MySQL on pc2004, pc2005 and pc2006 for decommission - T209858

Marostegui updated the task description. (Show Details)Tue, Nov 20, 9:14 AM
Marostegui reassigned this task from Marostegui to RobH.
Marostegui moved this task from In progress to Done on the DBA board.

These hosts are now ready for DCOps to take over.
MySQL has been stopped on them too.

RobH raised the priority of this task from Normal to High.Mon, Nov 26, 5:06 PM

This is high priority due to return to Farnam in December. I'll get these ready for onsite wipe ASAP.

RobH moved this task from Backlog to Decommission on the ops-codfw board.Mon, Nov 26, 5:23 PM
Marostegui mentioned this in Unknown Object (Task).Mon, Dec 3, 7:16 AM
RobH moved this task from pending onsite steps (eqiad) to Backlog on the decommission board.
RobH updated the task description. (Show Details)Mon, Dec 3, 8:50 PM

wmf-decommission-host was executed by robh for pc2004.codfw.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor

wmf-decommission-host was executed by robh for pc2005.codfw.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor

wmf-decommission-host was executed by robh for pc2006.codfw.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
RobH added a comment.Mon, Dec 3, 9:02 PM

Switch ports for later removal once unracked:

pc2004 asw-b-codfw:ge-5/0/35
pc2005 asw-c-codfw:ge-5/0/3
pc2006 asw-d-codfw:ge-5/0/6

RobH updated the task description. (Show Details)Mon, Dec 3, 9:02 PM

Change 477357 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] remove pc200[456] from site.pp

https://gerrit.wikimedia.org/r/477357

Change 477359 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom pc200[456] production dns entries

https://gerrit.wikimedia.org/r/477359

Change 477357 merged by RobH:
[operations/puppet@production] remove pc200[456] from site.pp

https://gerrit.wikimedia.org/r/477357

Change 477359 merged by RobH:
[operations/dns@master] decom pc200[456] production dns entries

https://gerrit.wikimedia.org/r/477359

RobH updated the task description. (Show Details)Mon, Dec 3, 9:11 PM
RobH updated the task description. (Show Details)
RobH reassigned this task from RobH to Papaul.
RobH moved this task from Backlog to pending onsite steps (codfw) on the decommission board.

@Papaul,

This is now ready for you to take over, wipe disks, and set aside for lease return this month (December 2018). This is high priority, which is unusual for a decom.

RobH renamed this task from Decommission parsercache hosts: pc2004 pc2005 pc2006 to Decommission parsercache hosts: pc2004 pc2005 pc2006 (Dec 2018 lease return).Mon, Dec 3, 9:18 PM
Marostegui updated the task description. (Show Details)Tue, Dec 4, 6:23 AM
Papaul updated the task description. (Show Details)Wed, Dec 5, 3:31 PM
Papaul added a comment.Wed, Dec 5, 3:35 PM

@RobH any reason why we have to add the servers that we are returning to the decommission tracking Google sheet since that sheet is to keep track of servers and other devices that are decommissioned and have on site.

Papaul updated the task description. (Show Details)Sat, Dec 8, 6:50 PM
papaul@asw-b-codfw> show interfaces ge-5/0/35 descriptions 
Interface       Admin Link Description
ge-5/0/35       down  down DISABLED

papaul@asw-c-codfw# run show interfaces ge-5/0/3 descriptions 
Interface       Admin Link Description
ge-5/0/3        down  down DISABLED

papaul@asw-d-codfw# run show interfaces ge-5/0/6 descriptions 
Interface       Admin Link Description
ge-5/0/6        down  down DISABLED
Papaul updated the task description. (Show Details)Tue, Dec 11, 8:41 PM

@Marostegui any reason why production DNS is still showing for pc2004?

170 1H IN PTR pc2004.codfw.wmnet.

Change 479042 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: removed mgmt DNS entries for pc200[4-6]

https://gerrit.wikimedia.org/r/479042

Papaul updated the task description. (Show Details)Tue, Dec 11, 8:55 PM

Change 479042 merged by Dzahn:
[operations/dns@master] DNS: remove mgmt DNS entries for pc200[4-6]

https://gerrit.wikimedia.org/r/479042

Change 479135 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom pc2004 dns entry

https://gerrit.wikimedia.org/r/479135

Change 479135 merged by RobH:
[operations/dns@master] decom pc2004 dns entry

https://gerrit.wikimedia.org/r/479135

After those last merges, is this good to be closed? @Papaul @RobH?
Thanks!

@Marostegui no need to close the task. It can be assign to @RobH so he can keep track

Marostegui reassigned this task from Papaul to RobH.Wed, Dec 12, 6:14 AM
RobH changed the task status from Open to Stalled.Thu, Dec 13, 6:22 PM
RobH reassigned this task from RobH to Papaul.

@Papaul,

We will be returning these to Farnam sometime this or next month. Go ahead and unrack/prepare these to be boxed up, we should have info on where the boxes will come from via Farnam sometime this month.

Once they are prepared for shipment, this task can resolve. (We have the parent task to track sending them back.)

Papaul closed this task as Resolved.Thu, Dec 13, 6:42 PM

This can be resolved then since i am done with it .