thanks for the report, sorry for missing this as part of T414486

Live-patched in production, service is restored.

Sounds a lot like T248872: puppet-merge lockout/tagout ?

This should be working now:
deployment.eqiad.wmnet
user fundraising-data-uploader
/var/lib/fundraising-data-uploader (the user's homedir)

Apologies, I'll write a host firewall patch tomorrow morning.

Thanks Joseph <3

https://wikitech.wikimedia.org/wiki/PKI/CA_Operations#Renewing_a_new_intermediate

+1 from me on making UUIDv4 the only codepath

In T306550#11651360, @taavi wrote:

(FWIW, a major reason these connections are so long-lived is the rate limiting that's there to try to handle all of the traffic from so little infrastructure. Offloading at least the most frequently used files to the CDN might possibly help with that?)

I haven't played around with https://connectrpc.com/ at all, but it looks like it was designed with these exact concerns in mind, and could allow us to write and ship a fully-compliant gRPC server that also, with zero work additional required, would accept an easy-to-speak dialect of plain HTTP.

In T418377#11678861, @HSwan-WMF wrote:

Hey Chris,

Perhaps we can talk live about this. I'm concerned about you mentioning that there will be no version of the limits that can facilitate an acceptable UX. I think we are hoping for 10 images/second if that's possible, but I don't want to make life difficult for all of you. I know you're dealing with a lot on the incident load front. We may end up just going live with the internal test as things stand, but would love to set up a time with you and my PM if that's possible.

Most user JS scripts should be working again.

Apologies for the conflicting information, that's partially my fault.

The app involves fetching a large number of apps to present to the user.

tagging High because it is so low-effort

We have used llama-server on our relforge instances (dual socket xeon silver w/ 12 cores each) to provide reranking in our prototype via a custom Qwen3-Reranker-0.6B-Q8_0.gguf but the performance is borderline unacceptable (2-3s for n=10). Q8 was used as it provided a significant improvement to inference latency over the unquantized model.

Sounds good to me.

In particular it'd be good to know if you only care about encryption going over the wire, or if encryption at rest is also necessary.

The latency trend for this past ~week is even worse:

Thanks for filing that @Tgr !

This should be fixed now -- I did some quick checks but further confirmation appreciated :)

Parsoid looks to be consuming basically all the CPU that mw-jobrunners have to offer:

Looks like it lines up with the train deployment of MediaWiki 1.45/wmf.6 ?

• https://sal.toolforge.org/production?p=7&q=&d=2025-06-19

It isn't just s2. It's all of them. Since the switchover.

I found the request:
https://logstash.wikimedia.org/goto/78f43ba4e27abb0fa03ad86fcae79dfc

In T416452#11586518, @elukey wrote:

@CDanis Hi! I saw your name for otelcol and this is why I am reaching out :) IIUC it is a golang binary so it should be ok to create a component for trixie-wikimedia and copy over the otelcol-contrib package right? Or should we follow another process? I know there is a bookworm variant but I'd love to target stable if possible. Lemme know!

In T414719#11565262, @Jelto wrote:

@CDanis do you think it makes sense to offer a /etc/hosts-solution as well in the task description? So something like "use tunnnelencabulator or change you etc hosts to the nearest address of gerrit-lb.<dc>.wikimedia.org" with a few examples. That might make it easier for volunteers to try it without installing tunnelencabulator. I'm happy to update the description.

In T278056#11549526, @elukey wrote:

The heap size didn't seem to be under pressure, so I am wondering if it is just something related to queries (maybe with regexes) that allocate a ton of object with small life, hitting some constraints like small young gen etc.. Since we use G1 now, we could try to increase something like -XX:G1MaxNewSizePercent and see how it goes.

Totally fine from SRE's side, just check in with the current oncallers before you begin the disruptive part of the maintenance.

The appservers RED k8s dashboard makes even heavier queries, and was the trigger of a Thanos outage this week.

In T414719#11526630, @bd808 wrote:

I think this may just be a quirk of tunnelencabulator, but when I run it and use ssh gerrit -- gerrit show-connections --wide I see myself entering via IPv6 and not an edge proxy. For me ssh -4 is necessary to route traffic over the tunnel.

In T414719#11526396, @thcipriani wrote:

When I do ssh gerrit.wikimedia.org -p 29418 gerrit show-connections --wide I'm seeing 4 "connections" per PoP without any associated user (confirmed with ss on the host). They seem to be short-lived connections. I guess this is some kind of tcp probe? Is this some kind of health-check behavior?

Q: How long should I leave this running?
A: Up to one whole workday at a time. Don't set it and totally forget it -- it's directing you towards one of our edge sites, and that will need to change if we need to depool that edge site.

💙cdanis@wmftop ~ 🕙☕ DC=magru ; curl -I -X GET https://gerrit.wikimedia.org --connect-to ::gerrit-lb.${DC}.wikimedia.org ; nc -vW1 gerrit-lb.${DC}.wikimedia.org 29418
HTTP/2 302
date: Thu, 15 Jan 2026 15:10:29 GMT
server: Apache
location: https://gerrit.wikimedia.org/r/
content-length: 215
content-type: text/html; charset=iso-8859-1
age: 1
vary: X-Forwarded-Proto
x-cache: cp7008 miss, cp7008 pass
x-cache-status: pass
server-timing: cache;desc="pass", host;desc="cp7008"
strict-transport-security: max-age=106384710; includeSubDomains; preload
report-to: { "group": "wm_nel", "max_age": 604800, "endpoints": [{ "url": "https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_uri=/w3c/reportingapi/network_error/1.0.0" }] }
nel: { "report_to": "wm_nel", "max_age": 604800, "failure_fraction": 0.05, "success_fraction": 0.0}
set-cookie: WMF-Uniq=lduuB793lSeIP6Ao8KGhOQLpAAAAAFvdmXX8iIiZ92DS7Rg0aVcUoUhHLVdOEKDe;Domain=gerrit.wikimedia.org;Path=/;HttpOnly;secure;SameSite=None;Expires=Fri, 15 Jan 2027 00:00:00 GMT
x-request-id: 2f56b390-8ac2-49e6-9794-6de1c5215d95

In T414460#11524635, @BTullis wrote:

My assumption is that this is more likely related to the cephfs interface, than to the rbd interface.
That's because we mainly use the block devices for postgresql, which doesn't have a high pod churn rate like the airflow tasks.

🎉 the ssh port isn't yet working, but https is!

Sure thing. This changeid & these patchsets both spent several minutes minutes showing "queued" on https://integration.wikimedia.org/zuul/ before the Jenkins job seemed to begin

In T414460#11521085, @cmooney wrote:

The k8s host sent a FIN to the remote side but due to the packet-loss issue the remote side didn't get it, or it did and the ACK for it wasn't received. Which explains it not moving to FIN-WAIT-2, however surely it should try to resend the FIN, and if this state persists eventually just delete the connection?

I took a quick look at the state of sockets on dse-k8s-worker1010, since FIN_WAIT_1 is not supposed to stick around for longer than a minute or two. Increasingly-conplicated ss flags showed that there's a busy: field available, reporting a number of milliseconds. This lets us do some dating on the sockets that are still around:

This is at least High and possibly UBN!

In T411503#11425771, @daniel wrote:

In T411503#11424246, @taavi wrote:

I don't think we currently have any places outside of https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_IP_space that publish our IP space. Would it be helpful if we published the same information in some machine-readable format?

Probbly... maybe this could use the same mechanism we use for "know clients" like googlebot? Curious what @CDanis thinks.

Question: are there any restrictions about recording this information in logs for some time (e.g. 90 days)? The same log would also include the client's IP address. It may also contain the user name or user ID of authenticated users in certain cases.

In T400119#11395040, @Guycn2 wrote:

This seems to be blocking legitimate web captures by the Internet Archive (see, for example, here). Is this actually the intended outcome?

The Internet Archive is a reliable source for viewing old Wikipedia articles exactly as they appeared at the time of capture, in a way that the Revision History feature does not allow (for example, it does not show old revisions of transcluded templates). It is also used to record the history of some special pages that do not normally have their histories saved on Wikipedia — such as Special:Tags and Special:GadgetUsage.

I'm afraid that the complete blockage of Internet Archive captures of all wiki pages is unfortunate, and may cause other legitimate features and services that rely on such captures to cease functioning.

@RobH please proceed at your convenience -- these two hosts are not in active service.

merged and fast-deployed to A:bastion OR A:cumin

Hi @Chandra-WMDE , seems like you posted the private key in the task instead of the public. Please stop using that key for anything, and generate a new one, and then we can get you sorted with access.

On trixie, the attempt to read the v6 address from the qemu variables in late_command.sh isn't working, and so the host gets configured for SLAAC instead (shown in P85100 from /var/log/installer/syslog). Which doesn't work at all.

In T409183#11340639, @BTullis wrote:

What I have in mind is to deploy oauth2-proxy behind envoyproxy.

Tracing updates section LGTM, no config changes needed. Thanks!