Ping?

Sorry about that, the SRX just shipped today! So I'll have Jin work on this when he goes out to install the SRX for mr1-eqsin replacement sometime next week?

Looks like we forgot about that during Jin' visits. What's the next step?

Jin,

When you were last onsite, I neglected to include the swap of a problematic optic we have.

Can you quote us for an on-site to swap the optic in cr3-eqsin:xe-0/1/1 located in 603, U40. It should have an SFP+ LR optic installed, and we need it swapped with one of the spare LR optics onsite in our racks. We should still have a couple spare there, but if not I'll order more. Do you recall if we had any? (My records show we shoudl have atleast 2 spares left, but my records could be wrong.) This optic has patch #1120 attached.

We'll have you swap the optic and put the possibly defective optic in a bag with the task # T300485 on it so if the errors disappear on the interface, we'll have you throw it away at a future onsite visit.

So an optic swap that we'll schedule to take place overlapping with our netops staff hours. They are located in the EU, so it'll be something like your afternoon, their morning.

Let me know if there are any questions.

Thanks in advance,

Entered https://wikimedia.coupahost.com/easy_form_responses/3234 into coupa for this work, Jin will coordinate with Arzhel via email and hangout for the actual work window.

Mentioned in SAL (#wikimedia-operations) [2022-06-15T06:52:58Z] <XioNoX> disable BGP to Telia in eqsin for optic replacement - T300485

Mentioned in SAL (#wikimedia-operations) [2022-06-15T08:03:27Z] <XioNoX> re-enable BGP to Telia in eqsin for optic replacement - T300485

We tried:

• New optic
• New patch cable
• New router port

And the errors are still present. Next step is to follow up with Telia so they check their side and then the DC for the X-connect.

Worked on the email draft with Arzhel and just emailed it in CC'd both Arzhel and Cathal. Once I have more info I'll update this ticket.

Arelion support case 01418061​ to investigate things. I'll followup with them as they progress the case.

Noted, that's the link we do the least traffic on so we can keep it down for some time. I'll take care of disabling BGP on Monday.

Ticket 1-218053856766 opened for the loopback test.

Support,

We need to test our cross-connection 20676697-A, which terminates into our panel @ PP:0603:1087235 - 15/16 and from that into our router cr3-eqsin (WMF7241)
xe-0/1/1.

We need the loopback placed on the Z side of this cross-connection, just before it terminates into Telia's patch panel. We're experiencing line errors, and are attempting to isoloate where the fault is occurring. This will allow us to test the cross-connection from our router.

We have already swapped the optics and patch cable in our rack, so this is our next step.

Please place the loopback starting on 2022-06-23 @ 09:00 Singapore time. Our engineers work in EU timezones, so they'll be able to perform any loopback testing required later on 2022-06-23.

Please plan to remove the loopback and return the line to normal service on 2022-06-24 @ 09:00 AM Singapore time.

Our Network Engineers will drain this link of use on 2022-06-22, so you can begin work on 2022-06-23 without confirming with us before start. To repeat, no need to confirm the start of this window, as we've already scheduled it with our internal teams.

Thanks!

Ok, they can place the loop back 1 hop away on sg1 side of things and asked if they could do so today while on the call. I advised not yet, as we haven't drained that of traffic.

@ayounsi or @cmooney: Can you drain this link of use immediately so they can start the work tomorrow rather than Thursday?

Change 807492 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] eqsin: disable Telia transit

https://gerrit.wikimedia.org/r/807492

Change 807492 merged by jenkins-bot:

[operations/homer/public@master] eqsin: disable Telia transit

https://gerrit.wikimedia.org/r/807492

Ok now SG3 staff are telling me my ticket isn't valid for this type of thing, despite telling me on a voice call yesterday they'd place it today, and require me to raise a trouble ticket, not a remote hands loopback test ticket.

Rather than argue with them, because they will win, I've opened 1-217601218912. They should install the loopback immediately and leave it in place until directed otherwise. The loopback should be placed back to our A size, but 1 hop from the Z side termination.

Updated SG3 to remove the loopback and return the circuit to service, sent email reply to Arelion support thread to request next steps since the cross connection tested fine.

Hello Rob,

We’ll open to our 2nd line to dispatch a technician to loop test at our panel.

In future, for the first step please let us know while bi-directional loops are placed by the cross-connect provider so we can check our side facing the loop.

Best Regards ,

Pravin Kularajah

Customer Support Engineer

Hello Team,

Kindly be informed that the Tech has been ordered for tomorrow at 0800 GMT.

Looking forward to keeping you updated.

Thank you

Best Regards,

George Mwakigali

Customer Support Engineer

Looks like there are no more errors. @RobH could you check it one last time before replying to Telia?

cr3-eqsin> show interfaces xe-0/1/1 extensive | match error
[...] # Everything should show 0, especially:
    Bit errors                             0
    Errored blocks                         0
[...]