Just to make clear what I did yesterday:

I would suppose the poolcounter limit that is reached is the one for expensive files:

It would be great if envoy fixed the TLS 1.3 to work well when two envoys talk to each other - we should check if that's been solved in the latest versions.

I think the current solution works well. Basically:

Regarding the last point - I think the best way to do this is to actually leave those hostpaths empty until someone needs to do something with the code - then allow people to run a command that will:

• Copy the code out of the latest mediawiki container to a path on the mw-debug kubernetes node
• Re-deploy the individual developer session, mounting the code from the hostPath
• tell the developer where they can find their code.

So the steps to do what Alex proposed would be:

AFAICT this is now resolved.

An alternative idea from @akosiaris which is also very interesting:

@jnuche is this task still valid? I think it's not really an issue given how scap works now, including the pre-download of images on the k8s nodes.

@jnuche I'm closing this task as resolved as AIUI we do this since months.

I can see the following two paths to solve this:

After discussion with @MoritzMuehlenhoff - it makes sense that snapshots might be broken right now that things change hectically for a testing distro before the freeze - we should pick up this task after June 3rd when the freeze will be enacted.

It's much simpler than that - debuerreotype would work correctly - the problem is that snapshots are full of broken links for bookworm - again:

While I've added bookworm to the build process, I think I'll revert that part of my change.

In T324200#8850266, @Ottomata wrote:

Basically add a configuration that allows to figure out article_url => derived urls as a mapping, and eliminate all the need for additional configuration everywhere.

I like this idea.

Q. If this was done, would there be a need for the resource_purge event at all? Wouldn't the resource_change event (or, more ideally, the actual state change event ;) ) be enough to indicate that a purge should happen?

The videoscaling has slowed down, so the current situation is that we have:

My idea for implementing this is as follows:

• Create a benthos container
• Add a release containing a Deployment with N replicas (is this the best solution?) to the namespace of the service, running benthos with the adequate configuration, one per datacenter, listening to the local queue
• Limit what we need to configure to just the prefix for the URL we get from resource-change

The number of servers pooled in jobrunning was just 7, and some servers (mw1458,mw1461,nw1466,mw1495) were depooled from jobrunning but receiving no traffic; This is yet another case where we should probably reduce the number of connections reused by envoy as, especially for videoscaling, it means basically no load-balancing from LVS, because the connections between changeprop and the backends live "forever".

Our previous assumption that this was only happening (as far as our ability to reproduce the bug) in codfw just proved wrong, I randomly got a page on V22 while having set V10 in my preferences for one of the reported pages.

I would aslo add - purging pages shouldn't help, unless we broke something fundamental in how parsercache works.

In T209892#8696187, @jrbs wrote:

In T209892#8670708, @tstarling wrote:

In T209892#8667435, @Joe wrote:

Also: who is going to take on this work?

@jrbs and @drochford may have thoughts on that, since they've been organising maintenance work on SecurePoll recently.

I can prioritise this with the contractors we have at the moment particularly if this is blocking other work.

In T336025#8841663, @Krinkle wrote:

@Joe Thanks!

If it's any concellation, these are not user-generated SVGs. They're only ~1KB SVGs that are code-reviewed, checked-in, on-disk as part of the deployed software.

This is solved thanks to @Legoktm's patch.

So, in order to get this working, we would need to install rsvg-convert in the base mediawiki image in production-images, and then use that in building our mw-on-k8s images.

For reference, the code in question is in Resourceloader\Image::rasterize:

Indeed. The code for resourceloader wants to use rsvg-convert and do it without going through shellbox - I guess for performance reasons.

In order to do this properly, we need to do as follows, IMHO:

1. Pick a k8s node, or even better, reimage one appserver to act as an additional k8s node with specific node taints so that no "normal" pod can be executed
2. Add a deployment of mw-on-k8s targeting those taints, add as many replicas as we can fit in that node; remember to also allow http connections as ab doesn't work well with TLS
3. Use benchmw against this node (with the port you chose for http) and a depooled appserver with the same generation of hardware

The parsoid servers (reachable via localhost:6002 on any appserver, or any server running the service mesh proxy) can reply to requests for any url that mediawiki would respond to on an api or appserver. So your code could just call the api URL connecting to port 6002 on localhost (inside mediawiki-config we might need to add a configuration for that).

I'm frankly not sure how checking appserver.svc.eaqiad.wmnet:9090 from an appserver would work - that IP resolves locally to the loopback interface on any appserver. We'd need to pick another internal IP.

we also need to add wikifunctions to our internal certs

Another thing to consider is: given there are known XSS vectors in vega 2, this might be the first time we get a report, but not the first time this has been found out. We should probably get someone to check all the past revisions of all pages containing graphs for suspiciously-looking patterns.

A starting point for this investigation can be which services currently call restbase:

All stale nodes have been updated.

Adding @hnowlan as FYI so that we are careful about this in the future.

Very simply, those 3 servers are not in the targets file for deployment.

In T330507#8735183, @Ottomata wrote:

@Joe we discussed the use of page_content_change in Kafka main today, and decided that we don't need it for now after all. We may one day, but it looks like all the immediate use cases of it are more data lakey (incremental dumps, mediawiki history, etc.), so Kafka jumbo will be fine.

We'd still like to deploy to wikikube, as DSE is multi DC and also would prefer not to run long lived apps as this time. This means that while the app would consume page_change from local DC Kafka main, it would always produce page_content_change to Kafka jumbo-eqiad. This means that when MW is pooled in codfw, most page_content_change events would be produced cross DC to Kafka jumbo. FWIW, this data would be produced cross DC links if we were using Kafka main too (via MirrorMaker). However, in this case, the Kafka producer in the app would have to deal with any cross DC latencies, so we'll have to be careful with producer buffering.

(Given that webrequest Kafka producers produce cross DC, I'm hoping we'll be okay :) )

I see a few ways to be able to enable this job on all wikis, but fundamentally the procedure I think would make sense is something as follows:

Sorry, I'm getting confused; to my understanding, WDQS/search will use mediawiki.page_change which AIUI are generated from mediawiki, not mediawiki.page_content_change.

On further thoughts:

I wouldn't consider this task done, but we took all the actions that are reasonable on the SRE side of the issue. Retagging as necessary.