Merged as mr-3, but still needs to be deployed (including a minor schema change).

Updated the Cassandra docs (such as they are): https://wikitech.wikimedia.org/wiki/Cassandra#k8s_egress

In T423168#11816624, @Scott_French wrote:

@Eevans - Could I ask you to pick up the documentation change for Cassandra host turn-up?

Basically, once the new host reaches the point where it's returned by the PQL query linked in the task description, it will appear in the intended network policy, but that still needs applied manually. Reaching out to ServiceOps to do that step is totally fine.

Dropping this to medium now that the immediate issue is remediated.

Done.

Done.

In T421439#11784276, @VRiley-WMF wrote:

This ticket seems like it relates to another ticket https://phabricator.wikimedia.org/T413559

@Eevans this server is out of warrenty, would you like us to replace this disk or leave it for the time being?

In T420008#11778630, @Snwachukwu wrote:

Hi @Eevans I'd like for the change to be applied to production tables

In T420468#11773435, @Eevans wrote:

[ ... ]

In this case 'One and only one...' is because there are zero listeners, this happened after the merging of r1262054. Allegedly we're not using RESTBase's parsoid endpoints for anything any more (/cc @daniel ?), but I don't know what it will do with a url that doesn't work at all (refuse to start? logspam? nothing at all?). Spoiler alert: I think we are about to find out.

Puppet on RESTBase is failing with:

There aren't any (normative) changes to config between 4.1.8 & 4.1.11:

component/cassandradev has had 4.1.11 imported to it, anything using target_version: dev will get a package upgrade (see: Cassandra#Cassandra version upgrades).

Apologies; My diagrams aren't nearly as pretty as yours:

In T419734#11753905, @BWojtowicz-WMF wrote:

Okay, I've done a few not too technical sketches trying to visualize the issue we're facing.

Applied https://gerrit.wikimedia.org/r/c/generated-data-platform/aqs/media-analytics/+/1260282 to staging as well.

In T419734#11748637, @BWojtowicz-WMF wrote:

I want to share a small update from our side on where we are.

Our immediate need is allowing integration of article topic data to the Mobile Apps team, where we agreed to start this integration end of Q3 / beginning of Q4. The estimated load would be ~1M requests/day (~12RPS) + backfilling to beginning of the year 2026. This is something our deployment with multiple replicas could handle, but having some sort of cache could help us, especially for latency.

There are a few issues that came up during integration work with hoarde:

First one is the difficulty of enabling gRPC networking to our cluster. This is what Luca described above and something Tobias is looking into regarding scope of this work.

Second and potentially the main one is not meeting hoarde assumptions about our service being entirely for the purpose of integrating with the cache, which opens a couple of practical integration issues:

1. We want to integrate with our existing article topic service. This service currently runs both request-response path, but also consumes page change events via changeprop and produces prediction events to Eventgate. We would need to make sure this event-driven path is still preserved.
2. We would still want to expose HTTP API for other users. We expose parameters beyond wiki_id/page_id/revision_id, which are being used by other users, so hoarde would mainly cover the access path for Mobile Apps team, where we need cache the most.
3. If we would expose a native gRPC endpoint to the current service, there is a gRPC contract mismatch. Hoarde expects LambdaService.GetArtifact with its own request/response types. Our inference stack is based on KServe, which has native gRPC support, however it exposes GRPCInferenceService.ModelInfer with different types (https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol#grpc-api). It means we would need adapter between hoarde and article topics anyway even if both use gRPC. I have prototyped and tested a gRPC/HTTP adapter that bridges the gap between our services, but maintaining those adapters per ML service would be additional infrastructure cost for us.

With the above in mind, the cost/benefit ratio of integrating with hoarde is starting to lean slightly more towards costs for us, especially if we could meet the load requirements purely by scaling our replicas. This is also because our other services we would potentially like to integrate with caching service would be in a similar position.

In T419734#11742584, @elukey wrote:

[ ... ]
My main point and doubt though is why the work is pushed to whoever exposes a lambda, versus allowing the artifact caching service to just use either grpc or http. I get the former is probably better for various reasons, but I haven't seen big issues with HTTP so far either. I am genuinely asking to understand what is the main benefit, I don't have a ton of familiarity with grpc so I am probably missing a big one.