I've verified that manually deleting an editor-analytics pod in staging will trigger crash looping, and then setting initialDelaySeconds on the liveness probe (in this case 40s) will resolve it.

Change #1270496 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] aqs2-common: Remove decommed aqs1012

https://gerrit.wikimedia.org/r/1270496

Change #1270496 merged by jenkins-bot:

[operations/deployment-charts@master] aqs2-common: Remove decommed aqs1012

https://gerrit.wikimedia.org/r/1270496

Plot twist:

Deploying https://gerrit.wikimedia.org/r/1270496 to editor-analytics staging failed, again with a (initial) liveness check timeout, but without the i/o timeout error. So, what gives?

It turns out there was also latent external-services network policy change, to allow egress to aqs1023 and aqs1024 added to the cluster last week. Once that was deployed to staging, I was able to cleanly apply https://gerrit.wikimedia.org/r/1270496.

So, we need to:

1. Apply the latent network policy change to other clusters.
2. Apply https://gerrit.wikimedia.org/r/1270496 to the remaining services x clusters.

Mentioned in SAL (#wikimedia-operations) [2026-04-13T17:40:19Z] <swfrench-wmf> applied latent external-services network policy changes for aqs{1023,1024} - T423168

So, once the external-services network policy changes were applied, the crash-looping pod in editor-analytics was able to start successfully.

That means the i/o timeout errors emitted during startup due to the continued presence of aqs1012 in the initial list were a red herring. In reality, it was the missing egress network policy entries for the discovered cluster hosts that were leading to the timeouts, and this was happening silently within the allowed liveness time limit (i.e., presumably a dial timeout that's longer than the one used to connect to the initial set), similar to what we saw in T366851.

Mentioned in SAL (#wikimedia-operations) [2026-04-13T19:14:52Z] <swfrench-wmf> applied aqs cassandra host list changes from https://gerrit.wikimedia.org/r/1270496 - T423168

In T423168#11816624, @Scott_French wrote:

@Eevans - Could I ask you to pick up the documentation change for Cassandra host turn-up?

Basically, once the new host reaches the point where it's returned by the PQL query linked in the task description, it will appear in the intended network policy, but that still needs applied manually. Reaching out to ServiceOps to do that step is totally fine.

Dropping this to medium now that the immediate issue is remediated.

These almost always occur in batches (i.e. hardware refreshes, expansions, etc), usually on the order of between 3-9 hosts at a time. The new hosts go up over a period of days, or even weeks, one-by-one. That's a lot of ServiceOps pings, no?

Also, I think the proper solution is for the driver to not block startup when one or more cluster nodes is down. That's not a stop the world condition, so it shouldn't block liveness IMO (and in fact, it doesn't for other language drivers, nor previous (and possibly subsequent) versions of this one). If this is the case, your list of action items above should probably include this (even if as a link to another issue), and it probably (hopefully) changes your 3rd and 4th action items to stopgap status, no?

In T423168#11816804, @Eevans wrote:

[...]
These almost always occur in batches (i.e. hardware refreshes, expansions, etc), usually on the order of between 3-9 hosts at a time. The new hosts go up over a period of days, or even weeks, one-by-one. That's a lot of ServiceOps pings, no?

The point here is that you need the external-services network policy changes deployed in order for the new hosts to be usable by dependent services. This is true regardless of whether the client is taught to behave properly at startup in the event of an unreachable host (see below).

ServiceOps can do this for you (largely because cluster-wide changes like this have a wider blast radius), but you're also welcome to do so yourself rather than ping us.

Also, I think the proper solution is for the driver to not block startup when one or more cluster nodes is down. That's not a stop the world condition, so it shouldn't block liveness IMO (and in fact, it doesn't for other language drivers, nor previous (and possibly subsequent) versions of this one). If this is the case, your list of action items above should probably include this (even if as a link to another issue), and it probably (hopefully) changes your 3rd and 4th action items to stopgap status, no?

Indeed, that's what I meant by the 2nd clause in the 4th item. However, I can see how the way I phrased it over-indexes on how the client behaves now rather than prescribing how it should behave. I'll update that now to make it clearer.

Also, to emphasize, item 3 is required regardless of client behavior (see above).

Updated the Cassandra docs (such as they are): https://wikitech.wikimedia.org/wiki/Cassandra#k8s_egress

Change #1270980 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/deployment-charts@master] Set initialDelaySeconds on aqs-http-gateway direct Cassandra clients

https://gerrit.wikimedia.org/r/1270980

Change #1270980 merged by jenkins-bot:

[operations/deployment-charts@master] Set initialDelaySeconds on aqs-http-gateway direct Cassandra clients

https://gerrit.wikimedia.org/r/1270980

Alright, for now we're in a holding pattern until we decide how to approach the remaining item in the task description - i.e., improving the interaction between unreachable Cassandra hosts, gocql client session initialization, and service initialization (i.e., liveness and readiness).

eevans opened https://gitlab.wikimedia.org/repos/sre/hoarde/-/merge_requests/5

Draft: kubernetes-friendly initialization

eevans merged https://gitlab.wikimedia.org/repos/sre/hoarde/-/merge_requests/5

kubernetes-friendly initialization

Change #1277172 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/deployment-charts@master] linked-artifacts: upgrade to hoarde v1.1.3

https://gerrit.wikimedia.org/r/1277172

Change #1277172 merged by jenkins-bot:

[operations/deployment-charts@master] linked-artifacts: upgrade to hoarde v1.1.3

https://gerrit.wikimedia.org/r/1277172

See gitlab.wikimedia.org/repos/sre/hoarde/-/commit/a78889e for an example of application initialization that separates liveness from readiness. A solution like this should be broadly applicable to other cassandra-enabled golang services.