lvs1016 interface down
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Jan 4 2021, 9:53 AM

Description

See https://librenms.wikimedia.org/device/device=160/tab=port/port=15312/

Usual optics/patch replacement dance :)

Please sync up with the traffic team so they depool the host first.

Related Objects

Mentioned In: T272258: lvs1015 interface errors

Event Timeline

ayounsi triaged this task as Medium priority.Jan 4 2021, 9:53 AM

ayounsi created this task.

Restricted Application added a project: SRE. · View Herald TranscriptJan 4 2021, 9:53 AM

@BBlack can we schedule this on Monday? 1530/1600UTC

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jan 7 2021, 7:37 PM

CDanis subscribed.Jan 7 2021, 7:40 PM

The interface seems having trouble at the moment, we have some icinga alerts about pybal not reaching row-a hosts:

elukey@asw2-a-eqiad> show interfaces descriptions | match lvs1016    
xe-4/0/7        up    down lvs1016:enp4s0f1 {#3917}

Also some updown event logged in the syslog of the switch..

ayounsi raised the priority of this task from Medium to High.Jan 13 2021, 2:47 PM

The lvs is a secondary so not taking traffic, added a day of downtime :)

ayounsi renamed this task from Interface errors on asw2-a-eqiad:xe-4/0/7 (lvs1016) to lvs1016 interface down.Jan 13 2021, 4:22 PM

@Cmjohnson - Please do it at your earliest convenience. It's not in the flow of live traffic and doesn't need any "depool" AFAIK (but it is problematic that we don't have it as a reliable backup option!).

I added a week of downtime, the alarm popped up again, remember it if the issue gets solved sooner!

@elukey @BBlack swapped both the optics at the switch on a4 and on the server. It appears that the server side optic was the issue, the link is backup

xe-4/0/7 up up lvs1016:enp4s0f1 {#3917}

ayounsi mentioned this in T272258: lvs1015 interface errors.Jan 18 2021, 8:17 AM

lvs1016 interface downClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

lvs1016 interface down
Closed, ResolvedPublic
Actions