Luca has raised a few questions that may reveal relevant information:

File T348144 for decomming.

After discussion on IRC, I have also shutdown 1001 and 2001.

The machines ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet have been shut down (1001 and 2001 are still running in case we need files from them).

Do you think it would be useful to also keep the checksums in a different place, with permissions independent of the backing store behind the published/ directory?

SLO dashboard now available at: https://grafana-rw.wikimedia.org/d/slo-Lift_Wing_Readability/lift-wing-readability-slo-s?orgId=1

I've done 1002-1008 today, and everything went smoothly. All done!

I am very in favor of this scheme.

The service has been moved from the experimental namespace to readability in staging-codfw, and newly deployed to the same namespace to serve-codfw and -eqiad.

One thing of note: after elevating the tier like Luca did yesterday, the token has to be re-issued using the webui to have the new limit baked into it.

2007 and 2008 are now also done, again without problems.

Machines ml-serve2001-2006 are now done. Zero errors or irregularities. Will do 7 and 8 later this week.

(copied from T343900, this ticket is more appropriate for this info)

The problem is only really relevant for LLMs (Large Language Models), since they need more local disk space. Or at least the specific ones we tried did. We have plenty of disk space on our workers so far, so having a bigger kubelet partition/fs is quite feasible.

I have done ml2002 and ml2003 today (two machines to force some pods back onto 2002, to see it works properly). So far, everything seems fine.

Should be all clean now:

Upsides of local-ish to LW caching (e.g. Cassandra):

While making/experimenting with the SLO dahsboard it became clear that the label cardinality of our input metrics is so high (>10k) that direct computation from the input metrics is not feasible --- the result set is so large that we get at best partial results.

I've also run the same test tool against RR-ML and got much worse latency overall (though no errors, which is great):

Luca and I have gotten to the bottom of where the 503s come from. It was ultimately caused by the autoscaling activators being underprovisioned (both replica-count and memory quota). After patches 940391 and 940391, the 503s are gone, even on somewhat stressful tests.

In the ingress gateway, these 503s look like this:

I am still seeing (rare) 503s, and in the queue proxy pod this is logged:

With 400(!) workers, for 5m:

With the fixed version deployed to all clusters, I ran a load test again. Note that the throughput would likely be higher with more workers (i.e. it's limited by the client, not the inference service).

I have done some testing today:

In an effort to solve the practical problem (getting good RR inference without too many errors and timeouts), I'll do some testing on the other RR model (agnostic) to see if it's suitable as an alternative (maybe temporarily).

I've now also managed to add some latency bucketing stuff. Not 100% yet if it is what we want, but in any case, it's progress.

I made some progress on the experimental dashboard (https://grafana.wikimedia.org/goto/VSolQfj4k?orgId=1). Request count (and 200 vs non-200) I now have a better mental model/grasp of. The current setup is not ML-specific, but rather Istio-specific. So it may not apply to all k8s setups, but only those where Istio is used.

I am working on Grafana/Thanos directly for now because it's a shorter change-try loop to find the right metrics than doing it with Grizzly directly. Even with templating, we still need specific metrics to put into slo_definitions.libsonnet. Arguably, that's the most difficult part due to the sheer number of metrics surfaced even just by our own Prom instance.

https://grafana.wikimedia.org/goto/x7S0HpjVk?orgId=1 I've started an SLO dahsboard here. It only has one metric (Latency) so far, but it's a start.

Digging into the logging of the kserve and istio containers a bit, I have found a few things:

Correction: the iDrac is still down.

The iDrac is reachable again, so it likely was a different issue.

shows URX error state, aka:

Luca and I had a longer discussion via mail and IRC, about whether the backend-induced latency of an Inference Service should count towards the SLO budget or not.

Change 930610 has been pushed to prod, so now we get the full feed from changeprop.

I discussed the above questions with Luca today, and I think for now we can proceed with telling WME to start exploring the documentation we have (and tell us where there are gaps), and start testing against LiftWing/APIGW. This should surface any issues that might still be there, even if in the future the actual implementation of access to LW and rate limiting changes.

After some experimenting, the state of how rate limits for API tokens, the API gateway and Lift Wing currently are applied seems to be this:

A few thoughts:

This was caused by me using the wrong host.

After examining the setup some more, I figure I can delete the images on S3 as well, they are easy enough to reproduce with the docs I have. So this has been completed.

Changes have been merged and deployed, Bot eqiad and codfw (and staging) sections of the API GW work fine (tested from within clusters), as well as remote (from my home machine).

This should probably until we have updated to kserve 0.11 (T337213), I will tackle this after that.

Instances have been shutdown and S3 has been cleared of all but the latest checkpoint etc.