Page MenuHomePhabricator
Create Task
Maniphest T196030

troubleshoot cr3/cr4 link
Closed, ResolvedPublic
Actions

Description

This task will track the troubleshooting of the cr3-cr4 link in ulsfo. Both of these mx204s have been TEMP racked, and cannot stay where they are when placed into service. Until then, they each have a mgmt link and are now remotely accessible.

The link between them is NOT working. This task will troubleshoot and document steps taken, pulled from an email for onsite work.

Related Objects
Search...

Event Timeline

RobH triaged this task as Medium priority.May 30 2018, 11:43 PM
RobH created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 30 2018, 11:43 PM
RobH added a comment.May 30 2018, 11:43 PM

Troubleshooting steps email:

Because of timezones I might not be around when you get to the DC.

Interfaces are configured and should come back up automatically as soon as the issue is fixed.

ssh to cr3-ulsfo.mgmt.ulsfo.wmnet and cr4-ulsfo.mgmt.ulsfo.wmnet
Interface is et-0/0/0 on each side.

show interfaces et-0/0/0 terse

Shows link: down, should go up when good.

show interfaces diagnostics optics et-0/0/0

currently shows on cr3-ulsfo:
Laser bias current : 0.000 mA
Laser output power : 0.000 mW / - Inf dBm
Laser receiver power : 1.045 mW
For all the lines.

And cr4-ulsfo:
Laser bias current : 32.762 mA
Laser output power : 1.330 mW
Laser receiver power : 0.000 mW

It looks like cr3 sends light, and cr4 receives it, but cr4 doesn't send light.

It can be due to many reasons but, my current guess would be cr3's optic being faulty.

Re-seating it or replacing it might do the trick.

Also on cr3:

show log messages | last

has a lot of:
cr3-ulsfo fpc0 qsfp-0/0/0 Chan# 3: Rx loss cleared
cr3-ulsfo fpc0 qsfp-0/0/0 Chan# 0: Rx loss set
Which should stop happening when solved.

RobH added a comment.May 30 2018, 11:45 PM

Ok, so no good news:

I went ahead and did the following things, in the following order. Each step was followed by checking the optics diagnostics to see the send/rcv power, as well as checking for a link light and status within the software. Each time failed to change our baseline result, which is cr3 receives, but does not send, light.

  • swapped out the fiber optic cable
  • changed out the optic in cr3
  • changed out the optic in cr4
  • switched back to the originally routed fiber optic cable, leaving new optics in place. all other optics have been installed into cr3/4. There is an odd number of them, so sr3 has 3 total and cr4 has 4.

I've also created a task for tracking this issue, as a sub-task of the ulsfo networking master task. https://phabricator.wikimedia.org/T196030

In codfw, our QSFP+ have to use a different/special light blue fiber optic cable. Is that what we need here?

• ayounsi added a comment.May 31 2018, 6:52 AM

Thanks,
codfw uses MMF with those optics QSFP+-40G-SR4
ulsfo is SMF with QSFP+-40G-LR4

Disabling/enabling the port once more and I now have light.

Laser output power                        :  1.228 mW / 0.89 dBm
Laser receiver power                      :  0.609 mW / -2.16 dBm

Still seeing the following in the logs:

fpc0 qsfp-0/0/0 Chan# 0: Rx loss cleared
fpc0 qsfp-0/0/0 Chan# 1: Rx loss cleared
fpc0 qsfp-0/0/0 Chan# 2: Rx loss cleared
fpc0 qsfp-0/0/0 Chan# 3: Rx loss cleared
fpc0 qsfp-0/0/0 Chan# 0: Rx loss set
fpc0 qsfp-0/0/0 Chan# 1: Rx loss set
fpc0 qsfp-0/0/0 Chan# 2: Rx loss set
fpc0 qsfp-0/0/0 Chan# 3: Rx loss set

Probably still need to swap RX/TX.

• ayounsi reassigned this task from • ayounsi to RobH.May 31 2018, 7:15 AM
RobH added a comment.EditedMay 31 2018, 3:16 PM
In T196030#4245080, @ayounsi wrote:

Probably still need to swap RX/TX.

I replaced both of the optics with wholly different optics and a wholly different fiber cable. So these are using a second set of optics/fiber and no fix.

I suppose the next step is to setup 0/0/1 and try those to ensure its not a bad port?

• ayounsi added a comment.Jun 4 2018, 12:48 PM
In T196030#4246171, @RobH wrote:

I replaced both of the optics with wholly different optics and a wholly different fiber cable. So these are using a second set of optics/fiber and no fix.

This plus the previous disable/enable solved the "interface stuck with no TX" issue.

Then annother issue where cr3's alarm was mentioning:

Minor FPC 0 PIC 0 Need bounce

> request chassis pic fpc-slot 0 pic-slot 0 offline
> request chassis pic fpc-slot 0 pic-slot 0 online

But we're seeing a different issue now:

Light is correct, but interfaces are flapping and error counters are incrementing:

Errored blocks                       274

and:

fpc0 MQSS(0): CHMAC1: Detected Ethernet MAC Local Fault Delta Event for Port 0 (et-0/0/0)
fpc0 MQSS(0): CHMAC1: Ethernet PCS Multilane Alignment Not Done Delta Event for Port 0 (et-0/0/0)
fpc0 ifp et-0/0/0 ifd_mdown
mib2d[5587]: SNMP_TRAP_LINK_DOWN: ifIndex 549, ifAdminStatus up(1), ifOperStatus down(2), ifName et-0/0/0

I suppose the next step is to setup 0/0/1 and try those to ensure its not a bad port?

I'd say re-try the same steps as before (as we're troubleshooting a different issue, more similar to a faulty interface/fiber than before), then indeed try a different router port.

• ema moved this task from Backlog to Network on the Traffic board.Jun 4 2018, 3:38 PM
RobH reassigned this task from RobH to • ayounsi.Jun 6 2018, 10:58 PM

Ok, I'm back onsite today, and I've taken the following steps:

  • verified both optics are working by connecting an lc-sc patch to a light meter, they put out the following:
    • cr3:et0/0/0 = 7.1dBm
    • cr4:et-0/0/0 = 5.5dBm - if I try another optic in that same slot, it raises up to 6.9, so the optic seems bad. swapped to the new optic.
      • @RobH tried the failed optic in other enabled slots, it put out the 5.5 (1.5 lower than other optics) in same slot as working optics outputting ~7dBm on the light meter.
  • plugged back in existing lc-lc 3m run, still shows no link when running the commands below. Also ran the optic measurment commands, outputted below:
robh@cr3-ulsfo> show interfaces et-0/0/0 terse                 
Interface               Admin Link Proto    Local                 Remote
et-0/0/0                up    down
et-0/0/0.2              up    down inet     198.35.26.196/31
                                   inet6    2620:0:863:fe00::1/64
                                            fe80::f24b:3a00:2ef:7a4b/64
                                   multiservice
et-0/0/0.32767          up    down multiservice

robh@cr3-ulsfo>
robh@cr4-ulsfo> show interfaces et-0/0/0 terse                 
Interface               Admin Link Proto    Local                 Remote
et-0/0/0                up    down
et-0/0/0.2              up    down inet     198.35.26.197/31
                                   inet6    2620:0:863:fe00::2/64
                                            fe80::ee38:7300:275:34cb/64
                                   multiservice
et-0/0/0.32767          up    down multiservice

robh@cr4-ulsfo>
robh@cr3-ulsfo> show interfaces diagnostics optics et-0/0/0    
Physical interface: et-0/0/0
    Module temperature                        :  32 degrees C / 89 degrees F
    Module voltage                            :  3.2400 V
    Module temperature high alarm             :  Off
    Module temperature low alarm              :  Off
    Module temperature high warning           :  Off
    Module temperature low warning            :  Off
    Module voltage high alarm                 :  Off
    Module voltage low alarm                  :  Off
    Module voltage high warning               :  Off
    Module voltage low warning                :  Off
    Module temperature high alarm threshold   :  80 degrees C / 176 degrees F
    Module temperature low alarm threshold    :  -10 degrees C / 14 degrees F
    Module temperature high warning threshold :  75 degrees C / 167 degrees F
    Module temperature low warning threshold  :  -5 degrees C / 23 degrees F
    Module voltage high alarm threshold       :  3.7000 V
    Module voltage low alarm threshold        :  2.9040 V
    Module voltage high warning threshold     :  3.5950 V
    Module voltage low warning threshold      :  3.0020 V
    Laser bias current high alarm threshold   :  74.999 mA
    Laser bias current low alarm threshold    :  9.999 mA
    Laser bias current high warning threshold :  69.999 mA
    Laser bias current low warning threshold  :  15.239 mA
    Laser output power high alarm threshold   :  2.8184 mW / 4.50 dBm
    Laser output power low alarm threshold    :  0.1349 mW / -8.70 dBm
    Laser output power high warning threshold :  2.5119 mW / 4.00 dBm
    Laser output power low warning threshold  :  0.1514 mW / -8.20 dBm
    Laser rx power high alarm threshold       :  2.8184 mW / 4.50 dBm
    Laser rx power low alarm threshold        :  0.0562 mW / -12.50 dBm
    Laser rx power high warning threshold     :  2.5119 mW / 4.00 dBm
    Laser rx power low warning threshold      :  0.0708 mW / -11.50 dBm
  Lane 0
    Laser bias current                        :  31.690 mA
    Laser output power                        :  1.241 mW / 0.94 dBm
    Laser receiver power                      :  1.061 mW / 0.26 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 1
    Laser bias current                        :  27.738 mA
    Laser output power                        :  1.308 mW / 1.17 dBm
    Laser receiver power                      :  0.793 mW / -1.01 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 2
    Laser bias current                        :  31.958 mA
    Laser output power                        :  1.230 mW / 0.90 dBm
    Laser receiver power                      :  1.107 mW / 0.44 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 3
    Laser bias current                        :  31.690 mA
    Laser output power                        :  1.339 mW / 1.27 dBm
    Laser receiver power                      :  0.891 mW / -0.50 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
robh@cr4-ulsfo> show interfaces diagnostics optics et-0/0/0    
Physical interface: et-0/0/0
    Module temperature                        :  37 degrees C / 99 degrees F
    Module voltage                            :  3.2530 V
    Module temperature high alarm             :  Off
    Module temperature low alarm              :  Off
    Module temperature high warning           :  Off
    Module temperature low warning            :  Off
    Module voltage high alarm                 :  Off
    Module voltage low alarm                  :  Off
    Module voltage high warning               :  Off
    Module voltage low warning                :  Off
    Module temperature high alarm threshold   :  80 degrees C / 176 degrees F
    Module temperature low alarm threshold    :  -10 degrees C / 14 degrees F
    Module temperature high warning threshold :  75 degrees C / 167 degrees F
    Module temperature low warning threshold  :  -5 degrees C / 23 degrees F
    Module voltage high alarm threshold       :  3.7000 V
    Module voltage low alarm threshold        :  2.9040 V
    Module voltage high warning threshold     :  3.5950 V
    Module voltage low warning threshold      :  3.0020 V
    Laser bias current high alarm threshold   :  74.999 mA
    Laser bias current low alarm threshold    :  9.999 mA
    Laser bias current high warning threshold :  69.999 mA
    Laser bias current low warning threshold  :  15.239 mA
    Laser output power high alarm threshold   :  2.8184 mW / 4.50 dBm
    Laser output power low alarm threshold    :  0.1349 mW / -8.70 dBm
    Laser output power high warning threshold :  2.5119 mW / 4.00 dBm
    Laser output power low warning threshold  :  0.1514 mW / -8.20 dBm
    Laser rx power high alarm threshold       :  2.8184 mW / 4.50 dBm
    Laser rx power low alarm threshold        :  0.0562 mW / -12.50 dBm
    Laser rx power high warning threshold     :  2.5119 mW / 4.00 dBm
    Laser rx power low warning threshold      :  0.0708 mW / -11.50 dBm
  Lane 0
    Laser bias current                        :  27.535 mA
    Laser output power                        :  1.393 mW / 1.44 dBm
    Laser receiver power                      :  0.471 mW / -3.27 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 1
    Laser bias current                        :  33.833 mA
    Laser output power                        :  1.204 mW / 0.81 dBm
    Laser receiver power                      :  0.536 mW / -2.71 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 2
    Laser bias current                        :  33.499 mA
    Laser output power                        :  1.343 mW / 1.28 dBm
    Laser receiver power                      :  0.530 mW / -2.76 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 3
    Laser bias current                        :  33.833 mA
    Laser output power                        :  1.043 mW / 0.18 dBm
    Laser receiver power                      :  0.538 mW / -2.69 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  • ran new 3m lc-lc singlemode fiber patch cable between cr3:et-0/0/0 and cr4:et-0/0/0, since the existing one didn't fix the issue when the optics are both working with a light meter. Same results, no link active, outputting the diagnostics below:
robh@cr3-ulsfo> show interfaces et-0/0/0 terse                 
Interface               Admin Link Proto    Local                 Remote
et-0/0/0                up    down
et-0/0/0.2              up    down inet     198.35.26.196/31
                                   inet6    2620:0:863:fe00::1/64
                                            fe80::f24b:3a00:2ef:7a4b/64
                                   multiservice
et-0/0/0.32767          up    down multiservice

robh@cr3-ulsfo> show interfaces diagnostics optics et-0/0/0    
Physical interface: et-0/0/0
    Module temperature                        :  31 degrees C / 89 degrees F
    Module voltage                            :  3.2400 V
    Module temperature high alarm             :  Off
    Module temperature low alarm              :  Off
    Module temperature high warning           :  Off
    Module temperature low warning            :  Off
    Module voltage high alarm                 :  Off
    Module voltage low alarm                  :  Off
    Module voltage high warning               :  Off
    Module voltage low warning                :  Off
    Module temperature high alarm threshold   :  80 degrees C / 176 degrees F
    Module temperature low alarm threshold    :  -10 degrees C / 14 degrees F
    Module temperature high warning threshold :  75 degrees C / 167 degrees F
    Module temperature low warning threshold  :  -5 degrees C / 23 degrees F
    Module voltage high alarm threshold       :  3.7000 V
    Module voltage low alarm threshold        :  2.9040 V
    Module voltage high warning threshold     :  3.5950 V
    Module voltage low warning threshold      :  3.0020 V
    Laser bias current high alarm threshold   :  74.999 mA
    Laser bias current low alarm threshold    :  9.999 mA
    Laser bias current high warning threshold :  69.999 mA
    Laser bias current low warning threshold  :  15.239 mA
    Laser output power high alarm threshold   :  2.8184 mW / 4.50 dBm
    Laser output power low alarm threshold    :  0.1349 mW / -8.70 dBm
    Laser output power high warning threshold :  2.5119 mW / 4.00 dBm
    Laser output power low warning threshold  :  0.1514 mW / -8.20 dBm
    Laser rx power high alarm threshold       :  2.8184 mW / 4.50 dBm
    Laser rx power low alarm threshold        :  0.0562 mW / -12.50 dBm
    Laser rx power high warning threshold     :  2.5119 mW / 4.00 dBm
    Laser rx power low warning threshold      :  0.0708 mW / -11.50 dBm
  Lane 0
    Laser bias current                        :  31.422 mA
    Laser output power                        :  1.245 mW / 0.95 dBm
    Laser receiver power                      :  0.572 mW / -2.43 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  Off
    Tx laser disabled alarm                   :  Off
  Lane 1
    Laser bias current                        :  27.738 mA
    Laser output power                        :  1.312 mW / 1.18 dBm
    Laser receiver power                      :  0.441 mW / -3.56 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  Off
    Tx laser disabled alarm                   :  Off
  Lane 2
    Laser bias current                        :  31.958 mA
    Laser output power                        :  1.232 mW / 0.91 dBm
    Laser receiver power                      :  0.504 mW / -2.98 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  Off
    Tx laser disabled alarm                   :  Off
  Lane 3
    Laser bias current                        :  31.690 mA
    Laser output power                        :  1.341 mW / 1.27 dBm
    Laser receiver power                      :  0.421 mW / -3.76 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  Off
    Tx laser disabled alarm                   :  Off

robh@cr3-ulsfo>
robh@cr4-ulsfo> show interfaces et-0/0/0 terse    
Interface               Admin Link Proto    Local                 Remote
et-0/0/0                up    down
et-0/0/0.2              up    down inet     198.35.26.197/31
                                   inet6    2620:0:863:fe00::2/64
                                            fe80::ee38:7300:275:34cb/64
                                   multiservice
et-0/0/0.32767          up    down multiservice

robh@cr4-ulsfo> show interfaces diagnostics optics et-0/0/0    
Physical interface: et-0/0/0
    Module temperature                        :  37 degrees C / 99 degrees F
    Module voltage                            :  3.2530 V
    Module temperature high alarm             :  Off
    Module temperature low alarm              :  Off
    Module temperature high warning           :  Off
    Module temperature low warning            :  Off
    Module voltage high alarm                 :  Off
    Module voltage low alarm                  :  Off
    Module voltage high warning               :  Off
    Module voltage low warning                :  Off
    Module temperature high alarm threshold   :  80 degrees C / 176 degrees F
    Module temperature low alarm threshold    :  -10 degrees C / 14 degrees F
    Module temperature high warning threshold :  75 degrees C / 167 degrees F
    Module temperature low warning threshold  :  -5 degrees C / 23 degrees F
    Module voltage high alarm threshold       :  3.7000 V
    Module voltage low alarm threshold        :  2.9040 V
    Module voltage high warning threshold     :  3.5950 V
    Module voltage low warning threshold      :  3.0020 V
    Laser bias current high alarm threshold   :  74.999 mA
    Laser bias current low alarm threshold    :  9.999 mA
    Laser bias current high warning threshold :  69.999 mA
    Laser bias current low warning threshold  :  15.239 mA
    Laser output power high alarm threshold   :  2.8184 mW / 4.50 dBm
    Laser output power low alarm threshold    :  0.1349 mW / -8.70 dBm
    Laser output power high warning threshold :  2.5119 mW / 4.00 dBm
    Laser output power low warning threshold  :  0.1514 mW / -8.20 dBm
    Laser rx power high alarm threshold       :  2.8184 mW / 4.50 dBm
    Laser rx power low alarm threshold        :  0.0562 mW / -12.50 dBm
    Laser rx power high warning threshold     :  2.5119 mW / 4.00 dBm
    Laser rx power low warning threshold      :  0.0708 mW / -11.50 dBm
  Lane 0
    Laser bias current                        :  27.535 mA
    Laser output power                        :  1.393 mW / 1.44 dBm
    Laser receiver power                      :  1.296 mW / 1.13 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 1
    Laser bias current                        :  33.833 mA
    Laser output power                        :  1.214 mW / 0.84 dBm
    Laser receiver power                      :  1.262 mW / 1.01 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 2
    Laser bias current                        :  33.499 mA
    Laser output power                        :  1.343 mW / 1.28 dBm
    Laser receiver power                      :  1.370 mW / 1.37 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off
  Lane 3
    Laser bias current                        :  33.833 mA
    Laser output power                        :  1.052 mW / 0.22 dBm
    Laser receiver power                      :  1.412 mW / 1.50 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser receiver power high alarm           :  Off
    Laser receiver power low alarm            :  Off
    Laser receiver power high warning         :  Off
    Laser receiver power low warning          :  Off
    Tx loss of signal functionality alarm     :  Off
    Rx loss of signal alarm                   :  On
    Tx laser disabled alarm                   :  Off

robh@cr4-ulsfo>
  • For the sake of testing, Ive plugged in a lc-lc 3m singlemode run between 0/0/1 on cr3/cr4. I've tested the optics in both of those ports and they output ~7dBm (cr3 seems to output just above 7 and cr4 just below, odd.) This is in place if a cross router link wants to be tested on those ports instead.

I'm not sure what to try next! I'll hold off on returning the possibly bad optic until we know whats going on.

• ayounsi added a comment.Jun 25 2018, 1:28 PM

I've been going back and forth with JTAC.
The next physical change we need to try is a "loop test", for example connect cr3:et-0/0/1 to cr3:et-0/0/2 and see if the links come up and stable.

If it fails to be on only one said, per the loop test, the probability of software issue is less. We need to try another QSFP/port.
If the loop works fine on both the sides, as we have already swapped the fiber/qsfp then we can share the details to engineering and debug for any software/hardware issues.
As I said earlier I have already approached engineering. These details they need anyways as part of troubleshooting.

• ayounsi mentioned this in T189552: Rack/cable/configure ulsfo MX204.Jun 25 2018, 1:31 PM
• Vvjjkkii renamed this task from troubleshoot cr3/cr4 link to 1xbaaaaaaa.Jul 1 2018, 1:06 AM
• Vvjjkkii removed • ayounsi as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from 1xbaaaaaaa to troubleshoot cr3/cr4 link.Jul 2 2018, 2:18 AM
CommunityTechBot assigned this task to • ayounsi.
CommunityTechBot lowered the priority of this task from High to Medium.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added a subscriber: Aklapper.
• ayounsi added a comment.Jul 9 2018, 3:26 PM

Did the loop test on cr3-ulsfo but still no stable link.

Update from JTAC:

I checked in my lab, I am able to bring up this link, however I couldn’t find Fiberstore optics in my lab to test with.
Let me check again our inventory if I can find one. Also will be following up with engineering on this.

The optics they used are FINISAR CORP. (part FTL4C1QE1C-J3) and JDSU (part JQPR04LWBAJN1).

• ayounsi mentioned this in Unknown Object (Task).Jul 12 2018, 10:00 PM
• ayounsi added a comment.Jul 12 2018, 10:19 PM

From JTAC:

I have to track with engineering is PR1372815. Currently it is not viewable publicly as still work in progress.

They are also still trying to source Fiberstore optics.

Opened T199483 to investigate trying different optic vendors.

• ayounsi changed the task status from Open to Stalled.Jul 13 2018, 1:40 PM

Latest news, Fiberstore optics are not qualified for the MX204, only Finisar are.
Waiting on T199483 to move forward here.

RobH added a comment.Aug 16 2018, 9:21 PM
Link-level type: Flexible-Ethernet, MTU: 9192, MRU: 9200, Speed: 40Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled

Link is now live between the two. The other 5 optics are in their packaging in the rack in case we want to return them or use them (I have no preference).

• ayounsi closed this task as Resolved.Aug 23 2018, 11:26 PM

We will use them for ulsfo.

If the new FS test optics (with new firmware) work, then we could consider using them in future deployments.

BBlack moved this task from Network to Done on the Traffic board.Oct 8 2021, 6:01 PM
Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptOct 8 2021, 6:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits