After depooling all etcd client traffic from eqiad today for T428495 following the etcd primary switchover yesterday (T433554), we discovered an undocumented client still reading from etcd in eqiad, polling dbconfig-instance entities every 2m from an aux-k8s-eqiad Pod IP.

This looks like zarcillo's data-sync worker. That statically follows the conftool.eqiad.wmnet SRV record domain, which switched over yesterday (with all of the other conftool SRV domains) to be backed by the codfw cluster. Unfortunately, zarcillo configures a network policy that only allows egress to eqiad cluster hosts. What that means is that I cannot simply restart zarcillo to force re-resolution, and indeed this has been subtly broken since then (i.e., a pod simply being rescheduled by k8s would break this).

While I could fix the network policy on my own, there's a second problem: There are pending helmfile diffs for the zarcillo service that appear unsafe to apply, as if there's an addition values file required that exists outside of version control.

In order to unblock the work in T428495, we need either:

1. To safely fix the network policy; or
2. An ACK from DBA that it is acceptable for zarcillo to degraded / broken until #1 is done.