We have removed the rate limits on OPTIONS requests and I don't see anymore 429 responses to them in logs. Resolving, please feel free to reopen if the issue presents itself again.

In T416390#11702311, @Jclark-ctr wrote:

@Clement_Goubert Could you update site.pp it is missing. Also add these to preseed for efi booting also Thanks!

I'll leave the MediaWiki internals for you to decide, but as far as what we want to achieve, yes, that sounds right. Given pageviews and unique-devices are now two services, the PageViewInfo extension needs to be able to address both individually through two different service mesh listeners.

Ack, thanks for following up.

In T390773#11685687, @Tgr wrote:

[...]

1. Update deployment-charts, not really sure what needs to be done here (here it seems to claim it's not used anymore)

In T419212#11681013, @MLechvien-WMF wrote:

@JMeybohm as discussed today:

• Redis hosts upgrade should be done at the same time as HW refreshes tasks T418918, T418261 and T418924 - @Clement_Goubert to confirm

Host back in the pool, thanks <3

Depending on exactly what you want this rewrite to do, it may be that an apache rule isn't the right choice here.

It's cordoned and depooled, fire away.

Updated task description with racking details and OS. Waiting for review on the puppet patch.

Updated racking details, changed OS to Trixie, puppet patches merged. All yours.

In T418922#11672695, @MoritzMuehlenhoff wrote:

@Clement_Goubert The current rdb* hosts are on Bullseye, and you listed Bookworm as the designated OS, if we move to a new OS, let's directly move to Trixie?

All yours.

In T418919#11670190, @RobH wrote:

@Clement_Goubert,

I've assumed the racking details, please double check them for accuracy and provide racking preferences in addition to the boilerplate:

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

This should now be unblocked.

Assigning to Scott for reshaping.

Moving to serviceops Radar as we have provided the required information.

Unstall once T418492: Redirect API Portal wiki URLs to www.mediawiki.org/wiki/Wikimedia_APIs is ready to do.

Setting stalled for @apaskulin to unstall when ready for us to implement.

In T418188#11653771, @gerritbot wrote:

Change #1242576 merged by jenkins-bot:

[operations/deployment-charts@master] Simplify spec-json-wikimedia route and use meta.wikimedia.org

https://gerrit.wikimedia.org/r/1242576

I was misled by the namespace changes caused by the puppet patch, the namespace definition itself wasn't merged. I just merged and deployed it this morning.

In T418212#11651683, @RobH wrote:

In T418212#11650297, @Clement_Goubert wrote:

In T418212#11650095, @MLechvien-WMF wrote:

Thanks Blake. All those hosts were cross checked by Clement (thanks!) and now have corresponding tasks in our spreadsheet tracker.

DC-Ops could you double check if the following conditions would work for a Herald Rule:

When a task:

• has rack/setup/install {hosts} in its task name
• has the tags #DC-Ops and #serviceops
• has its status becoming Resolved

Then:

• create a task Implementation {hosts} with tags #serviceops and #serviceops-upgrades-hardware under the same parent

Once confirmed I'll request one of our Phab admin to create it

s/#serviceops/#serviceops-new/g

Please note I didn't know about #serviceops-new but I can adjust my workflow accordingly and when I create a racking task use serviceops-new rather than serviceops.

In T414112#11651045, @gerritbot wrote:

Change #1243850 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/deployment-charts@master] admin_ng: add namespace for linked-artifacts

https://gerrit.wikimedia.org/r/1243850

Since this behaviour was apparent on multiple mw-on-k8s deployments, and disappeared with more replicas, it's very likely that it's load related and not an actual memory leak. This should be confirmed if we get either longer time to oomkill or the behaviour disappears with more replicas. If the behaviour disappears, no more changes should be needed. If the time to oomkill is longer, we should raise the memory limits until the asymptote reached by the RAM usage curves doesn't hit the limit.

When rate limits on "internal" traffic (WME/WMCS/etc) start being enforced on the rest-gateway, PageViewInfo will get rate limited as well unless we add an exception just for this extension.

I agree that the second option makes the most sense with the overall strategy, as well as waiting until after the switchover (while the implementation is very late, it's only one host that is not being decommissioned).

In T418212#11650095, @MLechvien-WMF wrote:

Thanks Blake. All those hosts were cross checked by Clement (thanks!) and now have corresponding tasks in our spreadsheet tracker.

DC-Ops could you double check if the following conditions would work for a Herald Rule:

When a task:

• has rack/setup/install {hosts} in its task name
• has the tags #DC-Ops and #serviceops
• has its status becoming Resolved

Then:

• create a task Implementation {hosts} with tags #serviceops and #serviceops-upgrades-hardware under the same parent

Once confirmed I'll request one of our Phab admin to create it

In T364245#11647997, @Novem_Linguae wrote:

In T364245#11644601, @Clement_Goubert wrote:

In T364245#11637524, @tstarling wrote:

It would be nice if we had a diagram or explanation of the whole request flow, including every TCP connection. I see Effie's diagrams at c:Category:Wikimedia servers diagrams with their "it's complicated" boxes, and Timo's backend-focused diagrams, but none of them are really what I need. I've looked into the Envoy service inside the k8s pod in the past, so I should probably have been able to answer my own question, but I do get rusty when I don't have to touch this stuff for a while.

I've made a quick one, omitting LVS, the rest-gateway for API calls, and non-request-flow sidecars. Hope it helps.

This diagram is great. Would encourage you to upload it and insert it into a Wikitech page somewhere :)