@FCeratto-WMF noticed that attempting to use the etcd client from the python3-etcd package on debian trixie fails upon attempting to connect to the etcd-main cluster, raising errors consistent with cert validation failure (see, e.g., P96541):

urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', '', 'certificate verify failed')])",)

Note that this will fail in the client constructor when allow_reconnect is used and the client attempts to eagerly discover / cache cluster members. Otherwise, it fails on first use. This is curious, because we certainly have working python etcd clients on trixie-based hosts that have conftool installed.

What's going on?

So, the key here seems to be the presence or absence of ca_cert in the etcd client constructor args. When provided, as we always do in production for conftool by way of /etc/etcd/etcdrc, there is no issue - validation succeeds as usual, here using the WMF root CA certs (/etc/ssl/certs/wmf-ca-certificates.crt). It's only when ca_cert is not provided that we see a difference in behavior between debian bookworm (or earlier) and trixie.

etcd.Client repro and comparison between bookworm and trixie hosts: P96548

Initially, I suspected this was a change in behavior in urllib3 between bookworm (1.26.12) and trixie (2.3.0), that renders ineffective the specific pattern used by the etcd client to disable certificate verification in the absence of ca_cert (link).

However, I realized that doesn't quite match reality: The version of the code in the python3-etcd package does not have that logic for the no-ca_cert case. In both bookworm and trixie (link), it's just:

if ca_cert:
    kw["ca_certs"] = ca_cert
    kw["cert_reqs"] = ssl.CERT_REQUIRED

At which point, we note that client.py imports etcd, so we look over at __init__.py and find at the very end (link):

# Attempt to enable urllib3's SNI support, if possible
# Blatantly copied from requests.
try:
    from urllib3.contrib import pyopenssl
    pyopenssl.inject_into_urllib3()
except ImportError:
    pass

And that gives us our difference between urllib3 versions - this does noting (ImportError) on bookworm, but is effectful on trixie. Furthermore, if the debian package included the upstream logic to set cert_reqs to ssl.CERT_NONE in the no-ca_cert case, then it works perfectly fine. Minimal repro and fix shown here: P96549

Recap: On trixie, previously latent code in the etcd client became active, and is triggering certificate validation failures when the appropriate CA certs are not provided. If the debian package contained the same support as upstream for toggling cert_reqs in the absence of CA certs, it would be a non-issue.

What do we do about this?

One option is to open a bug with the package maintainers, requesting that (at least for trixie) they patch in the upstream the changes around cert_reqs handling in the absence of ca_cert.

Another (more tactical) change is to verify that existing production uses of python etcd clients that may come to exist on trixie consistently set ca_cert. We know that's already true for conftool and spicerack (since it reads the same configuration, and indeed works on trixie hosts). The only one that comes to mind as not doing so is etcd-mirror, which would require a bit of plumbing (i.e., command line flag and puppet changes).

While I won't be around to resolve this, hopefully this provides enough detail to pursue a path forward.