Page MenuHomePhabricator

rsyslogd tls issues on tools-nfs-2
Open, MediumPublic

Description

This is probably unrelated to the parent task, but I noticed it when investigating that one.

Nov 26 05:06:16 tools-nfs-2 rsyslogd: Warning: Certificate file is not set [v8.2102.0 try https://www.rsyslog.com/e/2330 ]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: Warning: Key file is not set [v8.2102.0 try https://www.rsyslog.com/e/2331 ]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: nsd_ossl:TLS Connection initiated with remote syslog server. [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: nsd_ossl:TLS Connection initiated with remote syslog server. [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: Certificate error at depth: 1   issuer  = /C=US/O=Internet Security Research Group/CN=ISRG Root X2  subject = /C=US/O=Let's Encrypt/CN=E5  err 20:unable to get local issuer certificate [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: SSL_ERROR_SSL Error in 'osslHandshakeCheck Client': 'error:00000001:lib(0):func(0):reason(1)(1)' with ret=-1  [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: OpenSSL Error Stack: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: Certificate error at depth: 1   issuer  = /C=US/O=Internet Security Research Group/CN=ISRG Root X2  subject = /C=US/O=Let's Encrypt/CN=E5  err 20:unable to get local issuer certificate [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: SSL_ERROR_SSL Error in 'osslHandshakeCheck Client': 'error:00000001:lib(0):func(0):reason(1)(1)' with ret=-1  [v8.2102.0]
Nov 26 05:06:16 tools-nfs-2 rsyslogd: OpenSSL Error Stack: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed [v8.2102.0]

Sounds like an expired root cert, although I would expect puppet to refresh those.

Event Timeline

This same error is present throughout tools: tools-cumin-1.tools.eqiad1.wikimedia.cloud,tools-elastic-[4-6].tools.eqiad1.wikimedia.cloud,tools-harbor-1.tools.eqiad1.wikimedia.cloud,tools-k8s-etcd-[22-24].tools.eqiad1.wikimedia.cloud,tools-nfs-2.tools.eqiad1.wikimedia.cloud,tools-package-builder-04.tools.eqiad1.wikimedia.cloud,tools-prometheus-[6-7].tools.eqiad1.wikimedia.cloud,tools-redis-[5-7].tools.eqiad1.wikimedia.cloud,tools-sgebastion-10.tools.eqiad1.wikimedia.cloud

sudo cumin --force O{*} "grep osslHandshakeCheck /var/log/syslog"

finds the string present on 245 VMs.

taavi renamed this task from letsencrypt issues on tools-nfs-2 to rsyslogd tls issues on tools-nfs-2.Nov 29 2024, 7:52 AM