The patch, for posterity: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305065

It seems to be happening much, much more often since 11:30 UTC today.

In T431112#12091106, @elukey wrote:

The only reliability concerns have been related to Druid failing to index for problems that were known (super old Druid version not upgraded in years, etc..). I'll check the doc that you are proposing, but let's put things in the right framing please: the current setup is good and reliable. Can it be improved? Yes for sure :)

00:42:36 <Amir1> okay, the user impact should be gone now
00:43:11 <Amir1> it's removed from writes

Don't have much context here, but I'm very happy to help figure out how LAC/hoarde can best integrate with tracing as deployed in WMF prod.

Today (2026-06-17) was a recurrence of this, with many puppet-compiler jobs stuck in "queued" state for 3+ hours because of issues in other Zuul pipelines.

There's a hidden Option 4 here, which is to declare that urldownloader would be the first Sophroid-only service, only accessible via the service mesh and Envoy, but apparently the little remaining Sophroid work isn't scheduled until Q2.

In T428573#12024436, @JMeybohm wrote:

Not completely. But I can imagine this might come up again in X time because another thing came along that does not have the dummy headless service. Because of this I'm also not opposed to testing kubepods which would solve this in a more generic way (I'm thinking of maintenance and other k8s jobs rn). Since we build CoreDNS from source anyways it should not be too hard to give this a try.

It might be possible to work around this by creating headless services for these jobs that access the databases -- definitely worth trying.

In T427457#12010832, @Gehel wrote:

• Why use urldownloader and not the HTTP proxies ? - https://wikitech.wikimedia.org/wiki/HTTP_proxy

Because we have no idea :) And we need you to tell us what needs to be used.

The MR looks good to me! I'm happy to help with the haproxy patch portion of possibility 1, which is my preference of the two.

In T428573#12001514, @cmooney wrote:

It doesn't seem to have a service endpoint registered though, which I think is needed before CoreDNS will publish any records for it:

cmooney@dse-k8s-ctrl1001:~$ sudo kubectl get endpointslices -A | grep 10.67.28.73
cmooney@dse-k8s-ctrl1001:~$

This does indeed look promising! Let's watch for another week or two, and then we can probably call it good enough.

In T427491#11965123, @daniel wrote:

Oh, I think I know what happened. There are log entries for category_redirect (commons:commons; User:RussBot) Pywikibot/10.2.0 (g19732) requests/2.32.3 Python/3.11.2.final.0 with no x-ua-contact set, but they are classified as known-network. I guess we don't extract the contact info for requests from known networks then?

+1 from me on supporting the format pywikibot already emits for on-wiki usernames.

Thank you Luca!

I think, as compared to aggressive, middle-ground mostly adds extra work without actually buying much safety or flexibility.

Can confirm this is also affecting https://gitlab.wikimedia.org/repos/sre/CIDERGRINDER

In T425545#11900651, @cooltey wrote:

The issue is that the lzh.wikipedia.org does not support API requests (e.g. page/summary or feed/featured), but the zh-classical.wikipedia.org works.

If you visit https://lzh.wikipedia.org, you will be redirected to https://zh-classical.wikipedia.org, which means we need to take it to the other team about handling the redirect correctly.

FWIW I kept the names unchanged on the past-Benthos side

Apologies, SRE put a rule in place as part of responding to an especially-impactful scraping incident, and it was too broadly scoped. It's been reverted since about 09:05 UTC today (thanks @Fabfur!)

BTW -- here are two canned queries for distributed traces of uploads: https://w.wiki/LSLK and https://w.wiki/LSLn
What Jaeger sees only includes the requests from MediaWiki towards ms-fe hosts, of course -- but you can use the x-request-id from the tags to pivot into logs.
I've no idea if Swift can omit OTel, or if it could use a proxy that could.

FYI: as of my Puppet patches above, you can now use an x-request-id value to find all the intra-Swift requests associated with that request.

thanks for the report, sorry for missing this as part of T414486

Live-patched in production, service is restored.

Sounds a lot like T248872: puppet-merge lockout/tagout ?

This should be working now:
deployment.eqiad.wmnet
user fundraising-data-uploader
/var/lib/fundraising-data-uploader (the user's homedir)

Apologies, I'll write a host firewall patch tomorrow morning.

Thanks Joseph <3

https://wikitech.wikimedia.org/wiki/PKI/CA_Operations#Renewing_a_new_intermediate

+1 from me on making UUIDv4 the only codepath

In T306550#11651360, @taavi wrote:

(FWIW, a major reason these connections are so long-lived is the rate limiting that's there to try to handle all of the traffic from so little infrastructure. Offloading at least the most frequently used files to the CDN might possibly help with that?)

I haven't played around with https://connectrpc.com/ at all, but it looks like it was designed with these exact concerns in mind, and could allow us to write and ship a fully-compliant gRPC server that also, with zero additional work required, would accept an easy-to-speak dialect of plain HTTP.

In T418377#11678861, @HSwan-WMF wrote:

Hey Chris,

Perhaps we can talk live about this. I'm concerned about you mentioning that there will be no version of the limits that can facilitate an acceptable UX. I think we are hoping for 10 images/second if that's possible, but I don't want to make life difficult for all of you. I know you're dealing with a lot on the incident load front. We may end up just going live with the internal test as things stand, but would love to set up a time with you and my PM if that's possible.

Most user JS scripts should be working again.

Apologies for the conflicting information, that's partially my fault.

The app involves fetching a large number of apps to present to the user.

tagging High because it is so low-effort

We have used llama-server on our relforge instances (dual socket xeon silver w/ 12 cores each) to provide reranking in our prototype via a custom Qwen3-Reranker-0.6B-Q8_0.gguf but the performance is borderline unacceptable (2-3s for n=10). Q8 was used as it provided a significant improvement to inference latency over the unquantized model.

Sounds good to me.

In particular it'd be good to know if you only care about encryption going over the wire, or if encryption at rest is also necessary.

The latency trend for this past ~week is even worse:

Thanks for filing that @Tgr !

This should be fixed now -- I did some quick checks but further confirmation appreciated :)

Parsoid looks to be consuming basically all the CPU that mw-jobrunners have to offer:

Looks like it lines up with the train deployment of MediaWiki 1.45/wmf.6 ?

• https://sal.toolforge.org/production?p=7&q=&d=2025-06-19

It isn't just s2. It's all of them. Since the switchover.

I found the request:
https://logstash.wikimedia.org/goto/78f43ba4e27abb0fa03ad86fcae79dfc

In T416452#11586518, @elukey wrote:

@CDanis Hi! I saw your name for otelcol and this is why I am reaching out :) IIUC it is a golang binary so it should be ok to create a component for trixie-wikimedia and copy over the otelcol-contrib package right? Or should we follow another process? I know there is a bookworm variant but I'd love to target stable if possible. Lemme know!

In T414719#11565262, @Jelto wrote:

@CDanis do you think it makes sense to offer a /etc/hosts-solution as well in the task description? So something like "use tunnnelencabulator or change you etc hosts to the nearest address of gerrit-lb.<dc>.wikimedia.org" with a few examples. That might make it easier for volunteers to try it without installing tunnelencabulator. I'm happy to update the description.

In T278056#11549526, @elukey wrote:

The heap size didn't seem to be under pressure, so I am wondering if it is just something related to queries (maybe with regexes) that allocate a ton of object with small life, hitting some constraints like small young gen etc.. Since we use G1 now, we could try to increase something like -XX:G1MaxNewSizePercent and see how it goes.

Totally fine from SRE's side, just check in with the current oncallers before you begin the disruptive part of the maintenance.

The appservers RED k8s dashboard makes even heavier queries, and was the trigger of a Thanos outage this week.

In T414719#11526630, @bd808 wrote:

I think this may just be a quirk of tunnelencabulator, but when I run it and use ssh gerrit -- gerrit show-connections --wide I see myself entering via IPv6 and not an edge proxy. For me ssh -4 is necessary to route traffic over the tunnel.

In T414719#11526396, @thcipriani wrote:

When I do ssh gerrit.wikimedia.org -p 29418 gerrit show-connections --wide I'm seeing 4 "connections" per PoP without any associated user (confirmed with ss on the host). They seem to be short-lived connections. I guess this is some kind of tcp probe? Is this some kind of health-check behavior?

Q: How long should I leave this running?
A: Up to one whole workday at a time. Don't set it and totally forget it -- it's directing you towards one of our edge sites, and that will need to change if we need to depool that edge site.