For some reason gnmic is unable to connect to cloudsw2-d5-eqiad. It appears it has never been able to do so.
gnmic if run in debug mode has these error logs repeating when it tries to connect:
2025/02/21 13:04:35.338810 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/grpclog/logger.go:53: [gnmic] [core] [Channel #23 SubChannel #42]Subchannel picks a new address "cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767" to connect
2025/02/21 13:04:35.343628 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/grpclog/logger.go:53: [gnmic] [core] Creating new client transport to "{Addr: \"cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767\", ServerName: \"cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767\", }": connection error: desc = "transport: authentication handshake failed: EOF"
2025/02/21 13:04:35.343693 /home/runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/grpclog/logger.go:65: [gnmic] [core] [Channel #23 SubChannel #42]grpc: addrConn.createTransport failed to connect to {Addr: "cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767", ServerName: "cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767", }. Err: connection error: desc = "transport: authentication handshake failed: EOF"
2025/02/21 13:04:35.343980 /home/runner/work/gnmic/gnmic/pkg/app/collector.go:123: [gnmic] target "cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767": subscription bgp rcv error: failed to create a subscribe client, target='cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767', retry in 10000000000. err=rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"
2025/02/21 13:04:35.344105 /home/runner/work/gnmic/gnmic/pkg/app/collector.go:123: [gnmic] target "cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767": subscription interfaces-states rcv error: failed to create a subscribe client, target='cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767', retry in 10000000000. err=rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"If I try to connect to it using openssl to validate the cert is ok the TCP connection opens but it fails after that:
cmooney@cumin1002:~$ openssl s_client -showcerts -connect cloudsw2-d5-eqiad.mgmt.eqiad.wmnet:32767 CONNECTED(00000003) write:errno=0 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 326 bytes Verification: OK --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok) ---
However manually taking the cert file off it and checking with openssl I can see it is valid (re-generated a few days back in testing):
cmooney@cumin1002:~$ openssl x509 -in cloudsw2.crt -noout -dates notBefore=Feb 13 16:49:00 2025 GMT notAfter=Feb 13 16:49:00 2026 GMT
I can't spot any config difference on it either. Looking at the JunOS versions on all the cloudsw it is the oldest, though only slightly behind E4/F4 which are working:
20.2R2-S3.5 cloudsw2-d5-eqiad 20.4R3.8 cloudsw1-e4-eqiad 20.4R3.8 cloudsw1-f4-eqiad 21.4R3.16 cloudsw1-c8-eqiad 21.4R3.16 cloudsw1-d5-eqiad 22.2R3.15 cloudsw1-b1-codfw
@ayounsi any ideas on what to try here? If nothing jumps out at us a JunOS upgrade is unfortunately what we need.