Page MenuHomePhabricator

cr1/cr2-eqiad: install new SCBs and linecards
Closed, ResolvedPublic

Description

In T128750 & T131784 we procured the following for each of cr1/cr2-eqiad:

  • (1) MPC4E-3D- 32XGE-SFPP
  • (2) SCBE2-MX-S

This task will track the installation of these.

Juniper's documentation on the subject is pretty accurate, as we followed that for the similar codfw upgrade. The only step that isn't explicitly mentioned is the need to set set chassis network-services enhanced-ip and reboot the REs.

Event Timeline

Restricted Application added subscribers: Zppix, Southparkfan, Aklapper. · View Herald Transcript

cr2's SCBs were upgraded today, which didn't go very smoothly for various reasons. T140770 has the full writeup.

cr2 still doesn't have the new linecard install, that will happen next week.

cr1 is still pending both SCB and linecard.

We installed the new FPC on cr2-eqiad today — it's now up and online, all of its 32 10G ports.

faidon claimed this task.
faidon added a subscriber: Cmjohnson.

This was finished on cr1-eqiad today as well. We had drained cr1-eqiad of traffic entirely for the JunOS upgrade (see T140770 for the whole account of that), so this time we opted to do a much quicker offline upgrade. Last time's online upgrade came with kernel crashes and FPC errors and it seems there is conflicting Juniper documentation about whether an SCB->SCBE2 online upgrade is even possible.

The process followed was:

  • Setting chassis network-services enhanced-ip and rebooting both REs (although the reboot was probably not needed)
  • Halting both REs at the same time(!)
  • @Cmjohnson powering off the box entirely, replacing both SCBs, while moving the REs to the new SCBE2s and keeping their (re0/re1) order.
  • @Cmjohnson installing the new linecard.
  • @Cmjohnson powering on the box.
  • Then waiting a while for the REs, fabric and FPCs to boot up and all interfaces to come up.

That's all — no master switchovers, fabric bandwidth upgrades etc. needed at all. The whole process lasted for approximately 15-20mins. (This of course does not include the drain/undrain work that was needed anyway for the JunOS upgrade.)