Page MenuHomePhabricator

Add linecard diversity to the router-to-router interconnect in codfw
Closed, ResolvedPublic

Description

cr1-codfw and cr2-codfw connect to each other via an aggregate of two 10G links, but each router has only one FPC linecard, and said linecard has both the cross-router links, as well as the peering/transit links, and the cross-cluster transport links.

This means that, if the FPC linecard fails on the current VRRP master, all codfw hosts will lose connectivity to the outside world indefinitely, until someone intervenes.

Possible solutions include adding a special router-to-router vlan to the row switches, or just buying more linecards and using them for the router-to-router link.

Incident: https://wikitech.wikimedia.org/wiki/Incidents/2020-03-25_codfw-network

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

As data point, FPC0 got purchased on 2014 and FPC5 in 2013 so it's also time to replace them.

ayounsi claimed this task.

Child task is completed enough so this is not an issue anymore.

ayounsi closed subtask Restricted Task as Resolved.Mar 25 2022, 3:45 PM