Waiting for the last comments in the doc @Jopparn. Then I will upload the report for the wiki. @MariaMolander did you want to add any feedback? I have not gotten any comments yet

Change #1325416 had a related patch set uploaded (by Hashar; author: Jakob):

[mediawiki/extensions/WikibaseLexeme@wmf/1.47.0-wmf.15] Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default"

https://gerrit.wikimedia.org/r/1325416

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1151.eqiad.wmnet with OS bookworm

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:22:12Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055' (T431682)

Completed depooling of db1161 by marostegui@cumin1003: Depool db1161.eqiad.wmnet to then clone it to db1275.eqiad.wmnet - marostegui@cumin1003

Started cloning db1161.eqiad.wmnet to db1275.eqiad.wmnet - marostegui@cumin1003

Don't be discouraged. I kind of get how it feels to be unsure if things are actually better or worse, and the limited test suite in the WikiWho repo probably doesn't help with that uneasiness.

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:19:48Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054' (T431682)

The second version of the dataset is now created and available here:

The patch at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1313958 will export compute service status as prometheus metrics. That will enable us to (for example):

@Rscout we ask kindly to use the process pointed to you by @Milimetric, otherwise this cannot be properly tracked and fulfilled, as we lack the necessary context.

anyhow, looking into the failed probes.

Change #1325408 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/WikimediaAntiAbuse@wmf/1.47.0-wmf.14] Improve performance of DB query in BackfillAbuseReview.php

https://gerrit.wikimedia.org/r/1325408

Change #1325407 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/WikimediaAntiAbuse@wmf/1.47.0-wmf.15] Improve performance of DB query in BackfillAbuseReview.php

https://gerrit.wikimedia.org/r/1325407

Change #1325293 merged by Marostegui:

[operations/puppet@production] mariadb: Productionize db1275

https://gerrit.wikimedia.org/r/1325293

Reply from memory-prime: "My benchmark batch went through without stopping or crashing. As far as I'm concerned, the problem was solved".

Change #1325328 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/WikibaseLexeme@master] Revert "REST: Enable `GET /lexemes/{lexeme_id}` by default"

https://gerrit.wikimedia.org/r/1325328

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:02:00Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054' (T431682)

Oh, it seems control 13 was removed:

!log andrew@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host toolsbeta-test-k8s-control-13

https://sal.toolforge.org/log/v3LR-J8B1kByGTxAlB6H

I realized github pull requests won't show up here. I got these out:

Change #1325294 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[operations/deployment-charts@master] Deploy GrowthBook 5.0.0 to staging

https://gerrit.wikimedia.org/r/1325294

Change #1325293 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionize db1275

https://gerrit.wikimedia.org/r/1325293

there's no control-13 in the cluster that I can see:

root@toolsbeta-test-k8s-control-12:~# kubectl get nodes
NAME                               STATUS   ROLES           AGE    VERSION
toolsbeta-test-k8s-control-10      Ready    control-plane   708d   v1.33.13
toolsbeta-test-k8s-control-11      Ready    control-plane   708d   v1.33.13
toolsbeta-test-k8s-control-12      Ready    control-plane   706d   v1.33.13
toolsbeta-test-k8s-gateway-1       Ready    gateway         156d   v1.33.13
toolsbeta-test-k8s-gateway-2       Ready    gateway         156d   v1.33.13
toolsbeta-test-k8s-gateway-3       Ready    gateway         156d   v1.33.13
toolsbeta-test-k8s-worker-12       Ready    <none>          700d   v1.33.13
toolsbeta-test-k8s-worker-13       Ready    <none>          568d   v1.33.13
toolsbeta-test-k8s-worker-nfs-10   Ready    <none>          640d   v1.33.13
toolsbeta-test-k8s-worker-nfs-11   Ready    <none>          330d   v1.33.13
toolsbeta-test-k8s-worker-nfs-7    Ready    <none>          701d   v1.33.13
toolsbeta-test-k8s-worker-nfs-8    Ready    <none>          701d   v1.33.13
toolsbeta-test-k8s-worker-nfs-9    Ready    <none>          701d   v1.33.13

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:56:27Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051' (T431682)

Mentioned in SAL (#wikimedia-operations) [2026-08-13T07:56:12Z] <marostegui@cumin1003> dbctl commit (dc=all): 'Add db1273 to dbctl T434288', diff saved to https://phabricator.wikimedia.org/P96035 and previous config saved to /var/cache/conftool/dbconfig/20260813-075611-marostegui.json

Update tests with /mnt/nfs/dumps directory

Change #1325289 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[machinelearning/liftwing/inference-services@main] revise-tone-task-generator: Remove topic filtering.

https://gerrit.wikimedia.org/r/1325289

Change #1325286 merged by Marostegui:

[operations/puppet@production] instances.yaml: Add db1273

https://gerrit.wikimedia.org/r/1325286

Change #1324770 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] instrument(EarlyOnboarding): enable reader retention metrics

https://gerrit.wikimedia.org/r/1324770

Change #1324745 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Remove outdated TODO

https://gerrit.wikimedia.org/r/1324745

In T431682#12209331, @VRiley-WMF wrote:

Hey @fgiunchedi Yes, that would work for me! Would monday next week work for you?

Change #1325286 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Add db1273

https://gerrit.wikimedia.org/r/1325286

Let's wait a few days to make sure db1273 is stable.

Approved as Randall’s manager

forgot to update with findings:

Change #1321976 merged by jenkins-bot:

[operations/deployment-charts@master] topic: webrequest-page-view

https://gerrit.wikimedia.org/r/1321976

Change #1325284 had a related patch set uploaded (by Itamar Givon; author: Itamar Givon):

[mediawiki/extensions/WikibaseLexeme@master] REST: Reject lexeme creation with missing fields

https://gerrit.wikimedia.org/r/1325284

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:39:50Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051' (T431682)

Thanks for reporting this. See https://commons.wikimedia.org/wiki/Help:Gadget-ZoomViewer and https://phabricator.wikimedia.org/T428524#12003358

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:38:04Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049' (T431682)

I believe the only missing piece is to merge this patch that removes old stuff:

• https://gerrit.wikimedia.org/r/c/operations/puppet/+/1321974

@RobH : this seems to be completed on our side. I'll keep this task open and let you close it when it makes sense for you.

@RobH : this seems to be completed on our side. I'll keep this task open and let you close it when it makes sense for you.

Data-Platform-SRE does not need to be involved (but thanks for the visibility!). Moving to "watching".

@ARamirez_WMF I've added you to the wmf ldap group. Could you try again logging in into superset.wikimedia.org ? It should work now.

I thought that we used this ticket only to collect issues that are related to MathJax Client-Rendering. I got the impression that over the last few days, tickets have been added here that only concern client-side MathML rendering without MathJax, such as T434504. This is probably also a duplicate of T352698.

I could reproduce it upstream by tweaking the HTML of the the CSS-only demo, but not with the Vue version. Can this be fixed in Codex rather than downstream?

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:29:57Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049' (T431682)

In T432212#12210704, @Gehel wrote:

Tagging Wikidata Platform Team for visibility. We can probably completely remove that mount for w[cd]qs* hosts, at this point we're in transition to Qlever anyway. To be confirmed by WDT.

Section s3: Wikis testwiki redacted - marostegui@cumin1003

It was quite tedious to test manually, but I think I can conclude rolling back to c3fa64d569 did not fix the issue for the [[Yeah Yeah Yeahs (EP)]] article, specifically. Sorry if I was overreacting :(