I will reboot this tomorrow morning, Oct 6th at 08:00 and we can take it from there.

Change 839396 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Depool esams in gdns prior to reboot of line card

https://gerrit.wikimedia.org/r/839396

Change 839396 merged by Cathal Mooney:

[operations/dns@master] Depool esams in gdns prior to reboot of line card

https://gerrit.wikimedia.org/r/839396

Mentioned in SAL (#wikimedia-operations) [2022-10-06T08:54:45Z] <topranks> rebooting line card fpc 0 on cr2-esams (T318783)

Reboot completed sucessfully, currently router not showing any alarms:

root@re0.cr2-esams> show system alarms                                      
No alarms currently active

I'll leave it a week or so and if it remains clear close the task, or if we get more errors proceed to open a TAC case with Juniper.

I tried to create an JTAC ticket for an RMA but am getting:

Our records show that the Service Contract has expired for the serial number or Software Support Reference Number (SSRN) you entered or that the Serial Number is out of warranty. A valid product Serial Number or SSRN is required to open a Technical Case. If you believe this information is incorrect, please open an Admin Case or chat with Customer Care for assistance.

Most likely due to T315378

JTAC case 2022-1207-600204 opened asking for an RMA as it's the 2nd time the issue happens.

JTAC wants us to try to re-seat the linecard before doing any RMA. Work scheduled for Jan 12th. Opened procurement {T325048} for the remote hands work.

Mentioned in SAL (#wikimedia-operations) [2023-01-12T11:54:37Z] <XioNoX> re-seating cr2-esams fpc0 linecard - T318783

This got promoted to major.

cr2-esams> show system alarms 
2 alarms currently active
Alarm time               Class  Description
2023-07-28 23:46:09 UTC  Major  FPC 0 Major Errors
2023-01-30 12:36:42 UTC  Minor  FPC 0 Minor Errors

Let's hope it doesn't fully break before the onsite visit.

Looking more into the alert and status, both ports on FPC0 PIC2 are down, one of which is the link to asw2-esams, so we have a loss of redundancy (traffic now only goes through cr3-esams).

Opened high priority case 2023-0809-747283 asking for a RMA.

RMA in progress, Juniper happy with address for replacement and staff at destination are aware of delivery.

I will decom the existing faulty card on Sunday when on site and prep for return.

@cmooney I think this can be closed?

@ayounsi yeah I think so, the RMA is complete as far as Juniper is concerned and we are no longer using the old card.

It's unclear to me if the new card has been received in codfw, Interxion ticket suggests it was collected in Amserdam. That's slightly separate to this task of course.

@Jhancock.wm do you know if there was a delivery for us in codfw coming from Digital Realty / Interxion Amsterdam?

Was a box like this:

Has a juniper line card in it (and doesn't have the metal bracket @Papaul asked me to put in cos I found that in my suitcase when I got home - sorry!)

@cmooney I haven't received it yet. I checked with the dock to make sure it hasn't arrived and we weren't notified but no luck. Is there a tracking number for the package?

@Jhancock.wm not 100%, I will try to chase on that.