Page MenuHomePhabricator

asw-c1-eqiad uplinks are down
Closed, ResolvedPublic

Description

Links from asw-c1-eqiad to both cr1-eqiad and cr2-eqiad are reported as down:

xe-1/1/0        up    down << cr1-eqiad:xe-5/0/2
xe-1/1/2        up    down << cr2-eqiad:xe-5/0/2

They both got disconnected at the same time during the weekend:

Jan 17 21:51:59  asw-c-eqiad /kernel: : port status changed
Jan 17 21:51:59  asw-c-eqiad /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: new state is 0 cifd xe-1/1/0
Jan 17 21:51:59  asw-c-eqiad /kernel: : port status changed
Jan 17 21:51:59  asw-c-eqiad /kernel: KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: new state is 0 cifd xe-1/1/2

My guess would be that C1's uplink module is dead. Please investigate with priority.

Event Timeline

faidon assigned this task to Cmjohnson.
faidon raised the priority of this task from to High.
faidon updated the task description. (Show Details)
faidon added projects: ops-eqiad, SRE.
faidon subscribed.

The uplink module was indeed dead. Fortunately we had a spare module hanging out in c6.