It's possible although I'm not sure exactly how. Maybe someone with a better understanding of the internal error capture routing would know.
In any case we can call it resolved as it isn't happening anymore.

Calling to attention T375382: Post pc1013 crash, failover may need to be done by DBA before the switchover.

Logistics... Thanks for the update!

In T374231#10140143, @JMeybohm wrote:

We got paged again and we see lots of /view/ requests being cut off at 60s, an example can be found here https://trace.wikimedia.org/trace/00baa42e12812e63ec9ebd5d2d96a3c2, the request ID in question has been logged as well:

[3eb4cf2d-d4f0-9df4-b4f4-32ade54f52c8] /view/kho/Z14554   PHP Fatal Error from line 486 of /srv/mediawiki/php-1.43.0-wmf.22/includes/exception/MWExceptionHandler.php: Allowed memory size of 1468006400 bytes exhausted (tried to allocate 2097160 bytes)

link T374241

Reset the RAID config and the disk is still in Foreign state, so I can't use it for a Virtual Disk. I think a replacement is in order.

It's not showing up in system, and still shows foreign on the RAID controler interface, but that host is part of T358489: mw2420-mw2451 do have unnecessary raid controllers (configured) and should not actually have hardware RAID. I can try and run it through the procedure in that task to see what shakes, what do you think?

Host depooled and downtimed for a week, all yours.

Yep that's ours. I'll depool the node so you can reseat when you want.

Sure, what URLs and expected HTTP codes/text would you like httpbb to test for?
Do you want serviceops to disable httpbb for wikifunctions in the meantime?

Sorry I didn't see the updates to the discussion before merging the previous iteration. Patch up to disable puppet-agent-timer.timer

In T373591#10131008, @Jhancock.wm wrote:

mw2379 is causing an alert in netbox https://netbox.wikimedia.org/extras/scripts/results/89809/

In T374249#10131011, @Jhancock.wm wrote:

mw2431 is causing an alert in netbox https://netbox.wikimedia.org/extras/scripts/results/89809/

Correction, it worked for puppetdb, but they got added back to debmonitor. Will investigate further.

Tested via test-cookbook on mw2428 and mw2429 and they seem to have been correctly removed from both puppetdb and debmonitor.

Sounds good.
From what I can see, poolcounter2004.codfw.wmnet and poolcounter1005.eqiad.wmnet are the least used, depending on whether you plan on doing the update before or after T370962: Southward Datacenter Switchover (September 2024)
Thanks for taking care of this <3

@Papaul We'll be decommissioning this host, sorry :)

Pulling restricted images now works from wikikube-worker2080, resolving.

There was a point release yesterday which could explain why it changed cc @elukey

• The 5 nodes with an incorrect RAID config from T358489: mw2420-mw2451 do have unnecessary raid controllers (configured) that haven't yet been reimaged
• The codfw nodes to be decommissioned
• The 3 nodes waiting for T367949: Spin down api_appserver and appserver clusters (the fourth is part of the nodes to be decommissioned)

Our ATS configuration was wrong and as such, traffic was being sent to the mw-web cluster instead of mw-api-ext. This had functionally no impact as the clusters are identical except for the amount of resources they have, but it is now fixed.

Only hosts left are:

• The 5 nodes with an incorrect RAID config from T358489: mw2420-mw2451 do have unnecessary raid controllers (configured) that haven't yet been reimaged
• The codfw nodes to be decommissioned
• The 3 nodes waiting for T367949: Spin down api_appserver and appserver clusters (the fourth is part of the nodes to be decommissioned)

In T364417#10089240, @jwang wrote:

Hi, I failed to ssh deployment.eqiad.wmnet. The message I got is deployment.eqiad.wmnet: Permission denied (publickey). I was able to ssh a couple of months ago. Is this related to the deployment? Is there any update on my end that could fix it?

In T373149#10088261, @Dwisehaupt wrote:

@Clement_Goubert Oh. Thanks for that. I must have forgot it last night. Sorry about that.

Just a heads up that the removal of the management DNS entries for these three servers popped up in a sre.dns.netbox run this morning, since they're in decommissioning state in Netbox I've proceeded with it.

All good now

cgoubert@cumin1002:~$ httpbb /srv/deployment/httpbb-tests/appserver/*.yaml --host mw-api-ext.discovery.wmnet --https_port 4447
Sending to mw-api-ext.discovery.wmnet...
PASS: 131 requests sent to mw-api-ext.discovery.wmnet. All assertions passed.
cgoubert@cumin1002:~$ curl --connect-to en.wikipedia.org:443:mw-api-ext.discovery.wmnet:4447 https://en.wikipedia.org/api/
<!DOCTYPE html>
<html lang="en" dir="ltr">
<head>
  <meta charset="utf-8">
  <title>APIs</title>
  <meta name=viewport content="width=device-width, initial-scale=1">
  <meta name="robots" content="index, follow">
  <style>
body { background: #fff; margin: 7% auto 0; padding: 2em 1em 1em; font: 15px/1.6 sans-serif; color: #333; max-width: 640px; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645AD; text-decoration: underline; }
</style>
</head>
<body>
        <h2>APIs</h2>
        <ul>
            <li><a href="/w/api.php">Action API</a>, providing rich queries, editing and content access.</li>
            <li><a href="/api/rest_v1/?doc">REST API v1</a>, mainly focused on high-volume content access.</li>
        </ul>
    <h2>Legal</h2>
    <ul>
        <li><a href="https://foundation.wikimedia.org/wiki/Developer_app_guidelines">App Guidelines</a>, for developers on how to properly reuse Wikimedia data, API, trademarks, and other content.</li>
    </ul>
</body>
</html>

As expected:

cgoubert@cumin1002:~$ httpbb /srv/deployment/httpbb-tests/appserver/test_main.yaml --host mwdebug1002.eqiad.wmnet
Sending to mwdebug1002.eqiad.wmnet...
PASS: 54 requests sent to mwdebug1002.eqiad.wmnet. All assertions passed.

cgoubert@cumin1002:~$ httpbb /srv/deployment/httpbb-tests/appserver/test_main.yaml --host mw-api-ext.discovery.wmnet --https_port 4447
Sending to mw-api-ext.discovery.wmnet...
https://en.wikipedia.org/api/ (/srv/deployment/httpbb-tests/appserver/test_main.yaml:48)
    Status code: expected 200, got 404.
    Body: expected to contain 'providing rich queries, editing and content access', got 'File not found.\n'.
===
FAIL: 54 requests sent to mw-api-ext.discovery.wmnet. 1 request with failed assertions.

In T373048#10084087, @Ladsgroup wrote:

Maybe it's an urban myth but I always though if index.php/index.html exists in a directory, apache automatically sends requests to the directory there (maybe it's lighttp only but I have seen this work in many places)

At first I thought it could be due to this change routing /api/ to /w/rest.php for T364400

Thank you!

Apparently the removal from the puppetserver wasn't properly done by the cookbook, I've done it manually and it should resolve. Sorry about that.

This is most likely caused by envoy terminating before mediawiki.

From what I can gather the automation is there with the --move-vlan option to the reimage cookbook, I think the cabling is already correct and only the port's vlan and server ip need to change. We can probably take advantage of these reimages to rename the former appservers at the same time.

Changes to sidecar images are generally fine to deploy, if in doubt you can ask on IRC either in #wikimedia-operations or #wikimedia-serviceops and someone should be able to answer. Thanks for deploying all of them :)

I don't remember to be honest, I created the task after digging around for a little bit, finding the timing coincidental, and it fixed itself with the next run.

Resolved with the 10:10UTC run

In T369011#10014356, @cmooney wrote:

In T369011#10014019, @Clement_Goubert wrote:

Host BGP re-enabled, back in Active status and uncordoned, all looks good. Thanks @VRiley-WMF

Small note about the confusing names of the BGP state machine. "Active" means "actively trying to establish the session". "Established" means its managed to connect, and usually things progress from though all the states to "Esatablished" in a second or two unless there is a problem. So if you see a state of "active" when checking it usually means there is a problem!

In this case all is good:

cmooney@lsw1-e3-eqiad> show bgp summary | match "10.64.132.28|2620:0:861:10b:10:64:132:28" 
10.64.132.28          64601        917        883       0       1     2:17:09 Establ
2620:0:861:10b:10:64:132:28       64601        920        883       0       1     2:17:09 Establ