Page MenuHomePhabricator
Create Task
Maniphest T208734

Decommission asw-c-eqiad
Closed, ResolvedPublic
Actions

Description

Once T208584 and T208586 are done.

  • Move servers off asw-c-eqiad
  • Verify no more traffic on asw-c-eqiad <->asw2-c-eqiad link [Arzhel]
  • Disable asw-c-eqiad <->asw2-c-eqiad [Arzhel]
  • Remove from monitoring (LibreNMS/Rancid/Smokeping/Icinga) [Arzhel]
  • Unplug asw-c-eqiad <->asw2-c-eqiad links [Chris]
  • Cleanup interfaces config on asw2-c-eqiad [Chris/Arzhel]
  • Wipe and power down device [DCops/Arzhel]
  • Unplug mgmt/console port and remove from DNS [Chris]
  • Update console servers [Chris]
  • Unrack asw-c-eqiad members [Chris]
  • Update Netbox

Details

SubjectRepoBranchLines +/-
Remove asw-c-eqiad from monitoringoperations/puppetproduction+0 -10
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJclark-ctrT208734 Decommission asw-c-eqiad
 ResolvedJclark-ctrT208584 Decommission old eqiad caches
 ResolvedJclark-ctrT208586 Decommission lvs1007-1012
 DeclinedNoneT202966 Make cp1099 the new pinkunicorn

Event Timeline

ayounsi triaged this task as Medium priority.Nov 5 2018, 4:09 PM
ayounsi created this task.
Restricted Application added a project: SRE. · View Herald TranscriptNov 5 2018, 4:09 PM
ayounsi added subtasks: T208584: Decommission old eqiad caches, T208586: Decommission lvs1007-1012.Nov 5 2018, 4:11 PM
ayounsi updated the task description. (Show Details)
ayounsi mentioned this in T187962: Rack/cable/configure asw2-c-eqiad switch stack.
Volans subscribed.Nov 6 2018, 1:59 PM
ayounsi added a comment.Nov 6 2018, 1:59 PM

Note, we will probably run into the same issues as T208788, at least about wipe, and possibly about removing it from Icinga.

ayounsi mentioned this in T208630: Display remote port name in LLDP output.Nov 6 2018, 7:30 PM
RobH moved this task from Backlog to Decommission on the ops-eqiad board.Dec 12 2018, 11:34 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:37 PM
ayounsi added a subtask: T202966: Make cp1099 the new pinkunicorn.Mar 6 2019, 11:55 PM
ayounsi updated the task description. (Show Details)Mar 7 2019, 6:13 PM
Stashbot subscribed.Mar 7 2019, 6:15 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-07T18:15:49Z] <XioNoX> disable asw2-c-eqiad <-> asw-c-eqiad link - T208734

gerritbot subscribed.Mar 7 2019, 6:26 PM

Change 495007 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Remove asw-c-eqiad from monitoring

https://gerrit.wikimedia.org/r/495007

gerritbot added a project: Patch-For-Review.Mar 7 2019, 6:26 PM
gerritbot added a comment.Mar 7 2019, 6:34 PM

Change 495007 merged by Ayounsi:
[operations/puppet@production] Remove asw-c-eqiad from monitoring

https://gerrit.wikimedia.org/r/495007

ayounsi reassigned this task from ayounsi to • Cmjohnson.Mar 7 2019, 6:42 PM
ayounsi updated the task description. (Show Details)
ayounsi added a subscriber: decommission-hardware.Mar 12 2019, 6:26 PM
ayounsi added a project: decommission-hardware.Mar 12 2019, 6:29 PM
ayounsi moved this task from Backlog to pending onsite steps (eqiad) on the decommission-hardware board.
ayounsi removed a subscriber: decommission-hardware.
ayounsi mentioned this in T187960: Rack/cable/configure asw2-a-eqiad switch stack.Mar 19 2019, 8:16 PM
ayounsi added a subtask: T187960: Rack/cable/configure asw2-a-eqiad switch stack.
ayounsi removed a subtask: T187960: Rack/cable/configure asw2-a-eqiad switch stack.Mar 19 2019, 8:18 PM
Maintenance_bot removed a project: Patch-For-Review.May 22 2019, 3:48 PM
BBlack closed subtask T202966: Make cp1099 the new pinkunicorn as Declined.Aug 1 2019, 3:31 PM
faidon moved this task from Backlog to Watching on the netops board.Oct 31 2019, 1:01 AM
ayounsi added a comment.Feb 19 2020, 2:24 PM

Ping?

• Cmjohnson closed subtask T208586: Decommission lvs1007-1012 as Resolved.May 13 2020, 6:16 PM
• Cmjohnson closed subtask T208584: Decommission old eqiad caches as Resolved.May 13 2020, 6:21 PM
ayounsi added a comment.Sep 28 2020, 8:24 AM

I'm seeing interfaces down on asw2-c-eqiad, and I'm not able to ssh to asw-c-eqiad, so I guess some of those steps have been done?

As they are now alerting I'm deleting the config for:

Cleanup interfaces config on asw2-c-eqiad [Chris/Arzhel]

ayounsi updated the task description. (Show Details)Sep 28 2020, 8:26 AM
wiki_willy reassigned this task from • Cmjohnson to Jclark-ctr.Nov 5 2020, 7:55 PM
Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM
Jclark-ctr updated the task description. (Show Details)Aug 10 2021, 9:45 PM

Preformed factory reset removed from rack, updated netbox

Jclark-ctr closed this task as Resolved.Aug 10 2021, 9:46 PM
ayounsi reopened this task as Open.Aug 13 2021, 6:44 AM

One report is alerting with:

asw-c1-eqiad connected console ports attached to unracked device asw-c1-eqiad: console0

Please make sure to clean up the console server config as well.

I also deleted the matching virtual-chassis: https://netbox.wikimedia.org/extras/changelog/63607/
Feel free to do the same for the other old VCs, or leave it to Netops

• Cmjohnson closed this task as Resolved.Aug 17 2021, 4:02 PM

Removed the serial console port (port 9) for old asw-c1.resolving

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL