Perfect :)

@akosiaris do you think that the idea of forming a dedicated working group for the next couple of quarters could be feasible? I can take care of kicking it off and finding volunteers (sounds like me and Scott are already in :D).

I see a lot of {"level":"error","timestamp":"2025-12-05T09:35:50.078Z","message":"error marshalling tile: ERROR: permission denied for table planet_osm_polygon_landuse_gen_z6 in the tegola's logs, we probably have a db password problem, lemme check.

In T411774#11433192, @JMonton-WMF wrote:

Hi @elukey!
We don't need this often to be honest, maybe it's more about being able to help the DP SRE team with small tasks rather than giving them more work, but I totally understand the concerns.

In this case I was working on a simple ticket to increase partitions on a topic we use in Kafka Jumbo.
Some time ago I wanted to help with topicmapr but the SRE team is on it already.

I usually work on our "Event Platform" tickets and I'm not sure if we'll need similar operations often. I can imagine small tasks about changing configs, like retention, compression, partitions... I already have a way of connecting to the Kafka Jumbo to explore topics, consumer groups, messages and so on, but altering a topic is denied.
At some point we'd like to work on a way of configuring topics too.

Maybe adding some ACLs to the cluster could be another approach? It won't require ssh access to the brokers. I could explore that if it helps.

My understanding from the last updates is that we are not actively pushing anything to the new registry with apus as backend, we have just quickly tested some months ago. Is it the right understanding?

In T406392#11247933, @Scott_French wrote:

It's a long road to migrating the registry from Swift to apus Ceph as the long-term solution for T390251, and even that is currently only focused on the /restricted prefix of the namespace (though that could change).

@JMonton-WMF Hi! I have used the kafka tools like topic mapper in the past and if not handled correctly (like throttling etc..) they can have nasty side effects, like causing bandwidth usage problems. If there is any maintenance in progress from SRE and those tools are used, we can have other unwanted side effects too.

ml-serve1013 has been added as k8s worker with the necessary taints to avoid regular pods to run on it by accident.

@Pikne wow good to know thanks! I am planning to create an endpoint for testing (likely maps-staging.wikimedia.org), that the community will be able to use to test changes like this one before hitting production. So I hope that in the future we'll be able to use the staging stack to carefully review a change before hitting real traffic (fingers crossed). I'll keep you in the loop when/if my patch will be scheduled on staging.

@Mvolz ahhh ok thanks for the explanation! I rechecked the graph and it shows a neat recovery towards 0%, it is still in the negative but I am confident that it will steadily progress towards a healthy/green error budget as the time passes. Let's keep it open and re-eval before the holiday break!

In the Pyrra dashboard I see a big hole during November, including the 23rd when the alert fired:

mmm but is allocatable something that varies dynamically? Probably not, if so everything seems working fine. Or am I missing anything?

I checked the Allocated resources for ml-serve1009, where we run the revise-tone-task pod on a GPU, and I see the following:

@Eevans I am reasoning out loud, so no need to be sorry, thanks for the follow ups :) The external service should be basically a sort of software LB for hosts outside k8s, so you can contact them from k8s simply starting from a common name. I've read a bit of code and discovered stuff like https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/953283, my worry was that we had to duplicate the cassandra instance hostnames in the egress rules and in the initial hostnames list (for discovery), but it seems that we have a workaround in place.

My bad, the GPU is there:

yeah I think amd.com/gpu: 1 wasn't added when deploying aya, only tolerations, that would explain the result..

@MoritzMuehlenhoff what we may need to do is to move all disk/partition/raid/etc.. commands from datacenter-ops to ops-limited, what do you think?

To keep archives happy - the ml-serve1012 and 1013 hosts have been removed from the analytics vlan.

High level summary: while reviewing the Pyrra's availability graphs with the Abstract Wikipedia team we noticed several things that didn't make sense, like short severe drops affecting the downstream error budget calculations as well. After an investigation with Observability, it seems that Thanos, the system that Pyrra uses to create efficient and more compact time series / recording rules from the SLI metrics, has some consistency issues with its internal caching and sometimes it ends up storing the wrong datapoints/values in its long term storage.
The Wikifunction's SLI metrics seem to be the most affected ones, we are going to investigate the issue and report back a more permanent solution. Sadly the old time series, already in the Thanos long term storage, cannot be easily refreshed/overwritten so history may need to remain like it is.

This is a great milestone! Thanks a lot for the work Kevin :)

After a chat with Yiannis, the following commit was highlighted https://github.com/wikimedia/osm-bright.tm2/commit/e5b7a05b692199238ce54eb8e2e879331256d7ae. We added it while transitioning to the new kartotherian/mapnik versions, but we lost the context on what it was meant to resolve. The fonts are brought in by the osm-bright.tm2 nodejs package, so https://gerrit.wikimedia.org/r/c/mediawiki/services/kartotherian/+/1201020/ is not worth pursuing anymore. It may be worth to test a new kartotherian version with e5b7a05b692199238ce54eb8e2e879331256d7ae reverted, we now have a fully working staging environment that could be used for it.

+1 on all, seems a good plan, not sure if a higher level approach for blocking registries compared to iptables is available, but that is something that can be investigated later on.

In T410075#11409819, @Eevans wrote:

Well, like I mentioned above, (among other things) it avoids a significant number of network hops by not relying on randomly selected nodes to forward to one in the replica set. It's definitely doing things that would be unrealistic to expect of a load balancer.

Again, it's probably possible to configure each client (or perhaps in some cases, to implement a custom routing/load-balancing policy) to eschew this behavior, but a) we'd be giving up those optimizations, b) we'd have to touch/test a bunch of different code in different projects using different library implementations, and c) we'd have to support a solution that isn't idiomatic, in perpetuity. I can be skeptical about the benefits of (a), but I'm quite confident that (b) and (c) would be really painful. :)

Next steps:

@DPogorzelski-WMF I think the plan is good, I have only a few further questions:

At this point another alternative for the k8s world could be to have an externalservice configured, so that clients will use it to connect to random host and discover the full list. We'll still need to have egress policies for all nodes, but at least clients won't need to specify cassandra instance hostnames (but only the externalservice endpoint).

@Eevans thanks for the explanation, I kinda assumed that a query to any of the cassandra nodes would have worked as-is, routing the request to the right node (if needed) behind the scenes. IIUC you are saying that whatever endpoint is provided to connect to a cassandra, it is used to retrieve the list of nodes and then the client picks up the right IP address based on $routing policy. If this is true I am a little puzzled, it seems a really big over-complication to avoid a load balancer, but it defeats the purpose of an LVS endpoint yes.

Very weird. In theory http://outlink-topic-model-predictor.articletopic-outlink.svc.cluster.local/v1/models/outlink-topic-model:predict should set the Host header to outlink-topic-model-predictor.articletopic-outlink.svc.cluster.local behind the scenes (at least any HTTP client should do it). Maybe something specific in the Python code? We can try to inspect istio-proxy logs and see what is the reason for the 502 (usually there is a note about it), but I don't want to slow down the task if it is already working. It would be nice to figure out what's happening :D

In T408538#11408923, @BWojtowicz-WMF wrote:

@elukey @klausman @akosiaris

Thank you for all of your help investigating and finding the solution to enable the pod-to-pod communication!
I'm very happy to confirm that the solution Luca suggested works and is already integrated in our production service. We use a combination of http://outlink-topic-model.articletopic-outlink/v1/models/outlink-topic-model:predict as URL and outlink-topic-model-predictor.articletopic-outlink.svc.cluster.local as Host header to communicate with the service.

This is amazing as it's the first instance of cross-service communication on LiftWing cluster and it enables us to efficiently query other models within the cluster now! 🎉

@herron this task should be good in my opinion for the pilot's goals, we'll may need to tune it a little further if we decide to use Sloth but I wouldn't spend a ton of time on it in Q2. Lemme know!

Next steps:

The host is depooled:

@Mvolz all merged, the new dashboard is available here, but it seems that we are seeing a lot of errors anyway :(

In T411082#11409426, @Jclark-ctr wrote:

Additionally, this will make four Radeon PRO WX 9100 GPUs in storage. Should we consider selling them if they’re no longer well supported?

Hi folks!

Updated deletion list:

Today I tried to take a look to spikes like the following, shown in Grafana by Pyrra metrics:

@jcrespo sadly the upstream website changed and the way that we used to get the latest firmware doesn't work anymore. The only supported/working way is to stage the firmwares manually on the cumin nodes and use those :(

Looping in also @BTullis and @brouberol for a quick high level discussion, since AQS will be probably the first cluster to target :)

In T407503#11390492, @cmassaro wrote:

We've now deployed with the slightly reduced timeout. I hope to see the SLO number at 100% in the coming days, but let's see.

In T407503#11371534, @elukey wrote:

@Jdforrester-WMF @cmassaro one thing that I am wondering - what is the HTTP response code that Wikifunctions returns when a request hits the 10s timeout?

I believe it's a 504.

@Eevans my proposal would be to evaluate the possibility of having an LVS endpoint in front of all clusters. This would allow the following:

I always forget that istioctl is always a good friend :)

In T408538#11399806, @elukey wrote:

IIUC Knative just sits behind the kube svcs for the inference services to provide autoscaling-like services/buffering etc.. It shouldn't influence the routing, in theory :D

IIUC Knative just sits behind the kube svcs for the inference services to provide autoscaling-like services/buffering etc.. It shouldn't influence the routing, in theory :D

All good thanks!

@brouberol I think that the image is still running in a few places:

Really nice! I'll be afk next week for holidays, but @RLazarus may be able to follow up in the meantime :) Otherwise I'll pick it up when I am back!

Let's keep it open until the alerts are up :)

@Jdforrester-WMF @cmassaro one thing that I am wondering - what is the HTTP response code that Wikifunctions returns when a request hits the 10s timeout?

After a chat with Moritz we realized that the better path is probably to create another account for staging, and create the new container in there. In this way we fully disentangle prod from staging, and we don't risk to mess up prod tiles when working in staging.

Created a new bucket with swift post and the Tegola AUTH credentials on thanos-fe1004:

In T409414#11350459, @Eevans wrote:

In T409414#11350324, @elukey wrote:

[ ... ]

@Eevans Hi! Is there a load balancing endpoint in front of the cassandra nodes, or should we randomly pick one to connect to?

There is not, and if you choose just one random node, you risk connection failures on a restart (when there is a failure, or when the node gets refreshed, etc). What we do elsewhere (in more than once place), is use the full list of nodes, try really hard to keep them up to date, fail, but (somehow) narrowly avoid problems by virtue of it being a long list. Obviously this is terrible. 😢

In T407503#11365652, @DSantamaria wrote:

@elukey No, we do not have a number in mind. Our approach is going to be to iterate on those metrics (not just that one), but @Jdforrester-WMF can correct me if I am mistaken.

In T407503#11366980, @cmassaro wrote:

In T407503#11365529, @elukey wrote:

Exactly yes, it is here. We also have an extra envoy timeout set to 15s IIUC as extra fence, but the one that counts is the orchestrator's one.

So, this is still very unclear to me. If I understand correctly, there are two options:

• we reduce the timeout you've linked here to 9700ms, OR
• somehow, mediawiki_WikiLambda_mw_to_orchestrator_api_call_seconds_bucket gets changed and the 10s bucket becomes 10.2s or something.

The former is very easy for us to handle on our side. Is that the ideal path forward, or is it possible (and desirable) to do the latter?

The client seems to be a single one from the Istio Ingress logs, but the URI seemed to be /v1/models/revertrisk-multilingual:predict so it may make sense that they tried to query the service with a JSON payload that triggered the problem.

@gkyziridis I am still puzzled by the main exception:

@DSantamaria while we decide the best approach, it would be great to also discuss the path that we'll take after this first round of configurations. IIUC from our last discussion the 10s bucket is only the first step, to then find a more suitable/realistic target for the SLO (that wouldn't be always report an error budget of 100%). Does AW have a number in mind? What steps are we going to take to find the right value?

Exactly yes, it is here. We also have an extra envoy timeout set to 15s IIUC as extra fence, but the one that counts is the orchestrator's one.

@Nikki great investigation and datapoints, thanks a lot!

@herron Hi! Could you please backfill slo:period_error_budget_remaining:ratio too? I see that the time series start from Oct 27th, this is the rolling window metric and I'd like to see how it looks over a quarter.

@cmassaro I think it is probably something in the docker image / WF service itself, I haven't found a k8s configuration that triggers the 10s timeout yet.

In T408884#11334014, @Pppery wrote:

I tried to look at the code to understand how one would go about fixing this and got hopelessly lost in a maze of abandoned repositories without finding anything.

In T409411#11362422, @BTullis wrote:

Thanks both for your help with this.

In T409411#11348901, @elukey wrote:

It should be https://github.com/apache/druid/blob/druid-27.0.0/distribution/pom.xml#L119

<profiles>
    <profile>
        <id>dist-hadoop2</id>

But it seems gone from 29.0 onwards. So we may need to re-add it manually and see if it works with more recent versions as well.

I think that mybe it might be good to stick to version 0.28 for this upgrade, while we still need Hadoop 2 support.

Had a chat with David and other folks from the AW team, and the rock-solid-always-100% target is what the aim to verify that everything works as they expect, to then iterate on more precise values.

After a chat with the AW team last week I tried to follow up again on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1192609, and I may have got what James is trying to do. I'll write down my understanding:

@gkyziridis I am not 100% sure if the rev-id in the task's description is the problematic one, I thought it was when checking the logs but you may need to review /home/elukey/T409657 on deploy2002 to get other testing samples :(

Looks like we are back in acceptable ranges again! Please let me know if anything is missing.

Judging from the metrics it seems to me that the queues stopped growing, and they are slowly getting processed. Let's wait a bit more to see if the mitigation worked as expected.

I am really really ignorant about postfix so please bear with me :)

Option A would require some talk with SRE but given the size of the topic and the current /srv usage in main-eqiad / codfw I don't see any big opposition in having the stream hosted there (especially if we advertise that ML will not need to query the mediawiki API as direct consequence for the use case). It would probably be the most clean and reliable option in my opinion.

Me and @tappof spent quite a bit of time today trying to debug the above problem, namely that the graph showed only some days in September and nothing more. The issue seemed the sum_over_time applied to:

I had to use sum without(recorder) since the backfill process for edit-check caused another label to be added, ending up in errors while evaluating the group_left()` (many-to-many relationship).

editcheck's metrics seem to lead to:

One thing that I cannot solve is that vector(${__to:date:seconds}) returns a unix ts for Mon Dec 1 12:59:59 AM CET 2025 and 12 when selecting the month, while the time picker in grafana is set from Sep 1st to Nov 30th (happens the same in Grafana explore). I have no idea is I am missing something stupid or not..

New version of the two queries for the quarterly sloth panel:

To keep archives happy: I added the uid to the ops ldap group as well!

On the Lift Wing side, we need to configure two things:

Today I spent some time going through the Druid changelogs, more info in T409411. The major problem seems to be that they removed the support for Hadoop 2, but there may be a hack/workaround that we could do to make it work again (but it needs to be tested).

The Java 11 version, that we currently have on Bullseye, seems deprecated but supported on recent Druid versions.

It should be https://github.com/apache/druid/blob/druid-27.0.0/distribution/pom.xml#L119

The worst update in my opinion is related to dropping the Hadoop 2 support, so we'll need to investigate what this means:

Version 0.20 is interesting since it seems that a Query cache issue gets solved: https://github.com/apache/druid/releases/tag/druid-0.20.0#20-result-caching

New spicerack release deployed, cumin1002 is not needed anymore from Data Platform folks.

Spicerack deployed, thanks all for the work!

This is the current query for a month: