Page MenuHomePhabricator
Create Task
Maniphest T164341

Decommission old memcached hosts - mc1001->mc1018
Closed, ResolvedPublic
Actions

Description

The old mc1001->mc1018 hosts need to be decommissioned (new nodes are already serving traffic).

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - Set role::spare (system was not shut down)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove production dns entries https://gerrit.wikimedia.org/r/#/c/346823/
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS
[x ] - system disks wiped (by onsite)

  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove any reference of mc1001->mc1018 for decomoperations/puppetproduction+0 -95
Customize query in gerrit

Related Objects

Event Timeline

elukey created this task.May 3 2017, 7:47 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 3 2017, 7:47 AM
• Cmjohnson moved this task from Backlog to Lower Priority Items on the ops-eqiad board.May 3 2017, 1:13 PM
• Cmjohnson moved this task from Lower Priority Items to Up next on the ops-eqiad board.May 9 2017, 3:55 PM
elukey added a project: User-Elukey.May 17 2017, 8:30 AM
elukey updated the task description. (Show Details)May 17 2017, 8:55 AM
elukey moved this task from Backlog to Ops Backlog on the User-Elukey board.May 18 2017, 4:42 PM
gerritbot subscribed.May 19 2017, 11:16 AM

Change 354453 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove any reference of mc1001->mc1018 for decom

https://gerrit.wikimedia.org/r/354453

gerritbot added a project: Patch-For-Review.May 19 2017, 11:16 AM
elukey updated the task description. (Show Details)May 23 2017, 6:49 AM
elukey added a subscriber: • Cmjohnson.

@Cmjohnson: The hosts are ready for the non interruptible steps, including https://gerrit.wikimedia.org/r/354453, so I haven't merged it yet. Icinga alarms are off.

elukey assigned this task to • Cmjohnson.May 23 2017, 6:50 AM
elukey moved this task from Ops Backlog to Stalled on the User-Elukey board.May 26 2017, 12:42 PM
• Cmjohnson moved this task from Up next to Lower Priority Items on the ops-eqiad board.May 30 2017, 4:32 PM
elukey moved this task from Stalled to Done on the User-Elukey board.Jun 3 2017, 5:55 AM
MoritzMuehlenhoff mentioned this in T143695: Upgrade mc1* cluster to Linux 4.4.Jul 19 2017, 2:08 PM
• Cmjohnson moved this task from Lower Priority Items to Decommission on the ops-eqiad board.Jul 20 2017, 3:24 PM
elukey added a comment.Aug 4 2017, 9:39 AM

Any news on this? :)

Volans added subscribers: Joe, Volans.Aug 30 2017, 4:30 PM

@elukey, @Joe, @Cmjohnson: for testing purposes of the migration of the reimage script from salt to cumin, could I grab mc100[1-2] in the next few days as test hosts for the reimage script?

They are already in spare role in puppet, but let me know if you see any reason it's best not to take those ones.

Stashbot mentioned this in T166300: Remove Salt from wmf-auto-reimage / wmf-reimage.Sep 8 2017, 10:07 AM
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2017-09-08T10:07:44Z] <volans> testing wmf-auto-reimage on mc1001 T166300 T164341

Stashbot added a comment.Sep 8 2017, 2:58 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-08T14:58:08Z] <volans> testing wmf-auto-reimage also on mc1002 T166300 T164341

Volans added a comment.Oct 16 2017, 5:10 PM

@Cmjohnson FYI I'm not using anymore the above hosts for testing.

• Cmjohnson mentioned this in T177634: check mc1016 power supply redundancy.Oct 18 2017, 5:01 PM
gerritbot added a comment.Nov 21 2017, 11:28 AM

Change 354453 abandoned by Elukey:
Remove any reference of mc1001->mc1018 for decom

https://gerrit.wikimedia.org/r/354453

• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Dec 12 2017, 7:38 PM
• Cmjohnson moved this task from High Priority Task to Blocked on the ops-eqiad board.
• Cmjohnson moved this task from Blocked to High Priority Task on the ops-eqiad board.
gerritbot added a comment.Dec 12 2017, 8:16 PM

Change 397906 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Removing site.pp and dhcpd file entries for mc1001-18 T164341

https://gerrit.wikimedia.org/r/397906

gerritbot added a comment.Dec 13 2017, 3:26 PM

Change 397906 merged by Cmjohnson:
[operations/puppet@production] Removing site.pp and dhcpd file entries for mc1001-18 T164341

https://gerrit.wikimedia.org/r/397906

gerritbot added a comment.Dec 13 2017, 4:01 PM

Change 398066 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] removing dns entries for decom hosts mc1001-1018 T164341

https://gerrit.wikimedia.org/r/398066

gerritbot added a comment.Dec 13 2017, 4:41 PM

Change 398066 merged by Cmjohnson:
[operations/dns@master] removing dns entries for decom hosts mc1001-1018 T164341

https://gerrit.wikimedia.org/r/398066

• Cmjohnson updated the task description. (Show Details)Dec 13 2017, 5:34 PM
MoritzMuehlenhoff subscribed.Dec 15 2017, 12:07 PM

These are still showing up in https://servermon.wikimedia.org/hosts/, probably "puppet node deactivate" is missing

• Cmjohnson moved this task from High Priority Task to Up next on the ops-eqiad board.Feb 13 2018, 4:11 PM
elukey added a comment.Apr 19 2018, 2:49 PM

ping - status :)

elukey moved this task from Done to Keep an eye on it on the User-Elukey board.Apr 19 2018, 2:49 PM
• Cmjohnson moved this task from Up next to UnRacking Tasks on the ops-eqiad board.Aug 1 2018, 2:31 PM
• Cmjohnson closed this task as Resolved.Aug 7 2018, 8:44 PM
• Cmjohnson updated the task description. (Show Details)

These all had ssds and they have been removed. the ssds will not be included with the servers

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits