Page MenuHomePhabricator

(Need By: Sept 30) upgrade msw1-eqiad from EX4200 to EX4300
Open, HighPublic0 Story Points

Description

This task will track the scheduling and swap out of the central mgmt switch in eqiad, msw1-eqiad.

Current msw1/ex4200: https://netbox.wikimedia.org/dcim/devices/50/
Future msw1-EX4300: https://netbox.wikimedia.org/dcim/devices/2269/

The old/existing switch is an EX4200, and will be replaced with the new EX4300 ordered on T221883.

Items to confirm/update:

  • - netops confirms on task if they want new EX4300 racked for configuration in advance of migration
  • - migration window is scheduled
  • - racking and labeling of EX4300 (pending answer if this needs racking prior to migration, or if it can just go where existing EX4200 is presently.)
  • - configuration of EX4300
  • - migration of cables from EX4200 to EX4300
  • - wipe/decommission old EX4200
  • - test ALL mgmt uplinks by connecting to one server in every rack.

Event Timeline

RobH triaged this task as Normal priority.Jun 5 2019, 5:00 PM
RobH created this task.
Restricted Application added a project: Operations. · View Herald TranscriptJun 5 2019, 5:00 PM
RobH added a parent task: Unknown Object (Task).Jun 5 2019, 5:00 PM
RobH renamed this task from upgrade mr1-eqiad from EX4200 to EX4300 to upgrade msw1-eqiad from EX4200 to EX4300.Jun 5 2019, 5:02 PM
RobH changed the task status from Open to Stalled.
RobH updated the task description. (Show Details)
RobH added subscribers: Papaul, Cmjohnson.

Please note @Papaul is working with @ayongsi to upgrade the codfw msw1 on T224250. The current plan is to allow that to complete, and then replicate its work for eqiad.

At that time, @Papaul can work with @Cmjohnson directly to replicate the setup.

Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Jun 27 2019, 4:28 PM
wiki_willy renamed this task from upgrade msw1-eqiad from EX4200 to EX4300 to (Need By: Sept 30) upgrade msw1-eqiad from EX4200 to EX4300.Jul 2 2019, 10:37 PM
RobH mentioned this in Unknown Object (Task).Jul 15 2019, 8:01 PM
ayounsi reassigned this task from ayounsi to Papaul.Aug 2 2019, 3:36 PM
ayounsi added a subscriber: ayounsi.

codfw is done. @Papaul let me know if you need help to prepare the eqiad one.

Papaul added a comment.Aug 6 2019, 5:37 AM

@Cmjohnson I put together a "How to" at the link below on how to upgrade the switch. Please let me know if you have any questions.

https://wikitech.wikimedia.org/wiki/Juniper_switch_upgrade

wiki_willy reassigned this task from Papaul to Cmjohnson.Aug 30 2019, 6:19 PM
Cmjohnson updated the task description. (Show Details)Tue, Oct 1, 4:59 PM
ayounsi mentioned this in Unknown Object (Task).Wed, Oct 16, 7:53 PM
faidon changed the task status from Stalled to Open.Thu, Oct 17, 8:56 AM
faidon raised the priority of this task from Normal to High.
faidon added a subscriber: faidon.

What's the status of this? It seems like this migration is in some limbo state :)

As far as I understand it:

  • Old msw1-eqiad, EX4200, is still in production. It has been renamed to "msw1-eqiad-spare" in Netbox, but is not actually a spare.
  • New msw1-eqiad, EX4300, has been received (circa June 2019). It's not currently in Netbox at all, which has resulted into various issues; among others: we told Juniper that this S/N is not ours and we don't need support for it :)
  • The replacement is somewhat underway; the switch has been upgraded but has not been fully cabled(?)

This isn't a particularly urgent task, but it's been a few months now and it seems that we're at the point where this being kept in this state is causing more work to various people than it is to actually go through with the replacement, so perhaps we should prioritize it and complete it soon? Being bold and raising the priority but amenable to lower it again if DC-Ops folks disagree.

RobH updated the task description. (Show Details)Thu, Oct 17, 6:49 PM
RobH added a subscriber: Jclark-ctr.

Please note this states it was racked, but it was never added into netbox, so I'm not sure where it is racked.

I've gone ahead and put in the netbox entry, and need either @Cmjohnson or @Jclark-ctr to locate the new msw1-eqiad that is slated to replace the old msw1-eqiad and update the new msw1-eqiad netbox entry to show its asset tag and racked location.

RobH reassigned this task from Cmjohnson to Jclark-ctr.Thu, Oct 17, 6:52 PM

John,

Please locate the new msw1-eqiad that I describe below and update the netbox asset tag entry. This will clear up our reporting errors for this device. Then either yourself or @Cmjohnson need to coordinate with @ayounsi on when this can be replaced.

Please note this states it was racked, but it was never added into netbox, so I'm not sure where it is racked.
I've gone ahead and put in the netbox entry, and need either @Cmjohnson or @Jclark-ctr to locate the new msw1-eqiad that is slated to replace the old msw1-eqiad and update the new msw1-eqiad netbox entry to show its asset tag and racked location.