Thanks @cmooney, looks good! One small update to the above since we will most likely transpose these to hieradata/common/lvs/interfaces.yaml: 10.140.1.3/24 is private1-b4-magru like 10.140.0.2/24 is private1-b3-magru (and not just private-b4-magru).

@Lina_Farid_WMDE: to speed up things, you can also send an email to @KFrancis ( https://meta.wikimedia.org/wiki/User:KFrancis_(WMF) ; kfrancis@wikimedia.org) from your email, requesting an NDA. Thanks!

Hi @KFrancis: @Lina_Farid_WMDE will require an NDA as well as I don't see their name on the spreadsheet. Thank you as always!

Update is that we will need to add a DKIM record for MailChimp so a patch will follow. Rest everything seems to be in order.

We fixed it but forgot to close this task so resolving. Thanks @Dzahn!

Discussed a bit with @EdErhart-WMF on what the goal is here on Slack and will update this task later when there is more clarity.

This request is now merged; please re-open if there are any issues, thanks!

@Kgraessle: The request has been merged; please let us know if there is any issue. Thanks!

Thanks @jcrespo! I should have silenced the alert or restarted the service; both of those are in progress now so we should see this resolve soon.

Thanks for the task! At least for now, I restarted haproxy so that we don't get this alert and we also don't leave it silenced in case the initial restart (below) was nothing more than a transient one.

@DMburugu: this requires your approval, thanks!

@NBaca-WMF: This needs your approval, thanks!

@Aitolkyn I am marking this as resolved but if that's not the case, please re-open it again thanks!

@Papaul deserves a lot of love for fixing this persistent issue. The 21.x firmware (specifically, Network_Firmware_YK81Y_WN64_21.60.22.11_03) worked in the first attempt when reimaging cp1114. I think we can consider this closed given we have observed the fix on two hosts now.

@Papaul: Thanks for the update! Looks promising indeed and to actually close this, we should downgrade another host in eqiad and then try it out. Because what happens sometimes is that if a given host reimaged successfully once then it continues to reimage successfully for some more period of time (we don't know what that is but at least the same day :).

Thanks for the task @RobH! As in the previous runs, please feel free to leave these for Traffic:

Thanks @Muehlenhoff! And good to know @Michael that this is resolved; closing this task.

Added to wmf LDAP group (as well as Phabricator). Please try to access Logstash and let us know if there are any issues.

Marking this as resolved; if kinit doesn't work for you or if there are any issues, please re-open this. Thanks!

In T362602#9717940, @BTullis wrote:

I have created the principal for Surbhi.

btullis@krb1001:~$ sudo sudo manage_principals.py get sg912
get_principal: Principal does not exist while retrieving "sg912@WIKIMEDIA".
btullis@krb1001:~$ sudo manage_principals.py create sg912 --email_address=sgupta@wikimedia.org
Principal successfully created. Make sure to update data.yaml in Puppet.
Successfully sent email to sgupta@wikimedia.org

In T328457#9714365, @ssingh wrote:

@BTullis: Sorry if you are not the right person for this but it seems like we are having a related permissions issue in T362533 where @Aitolkyn is unable to use SQL Lab and I suspect that this is related. If not, please let me know. Thanks!

Thanks indeed @Urbanecm_WMF! Nice catch. @Aitolkyn: the contract expiry and date have been updated. If this has been resolved for you, please feel free to the task.

Can we open this task up now or we should still keep it private? The subtasks will still be private if this is public -- can someone confirm? If yes, we should open this up so that we can use it for tracking and if not, we should create a new public task.

@BTullis: Sorry if you are not the right person for this but it seems like we are having a related permissions issue inT362533 where @Aitolkyn is unable to use SQL Lab and I suspect that this is related. If not, please let me know. Thanks!

In T362533#9713602, @Aitolkyn wrote:

@ssingh Thank you for checking! I get the following error when trying to access my tables:

mysql error: SELECT command denied to user 'research'@'10.67.30.187' for table `aitolkyn`.`domain_reverted_added_2019_2023`

Maybe I need to set anything before I can access those?

Hi @FNavas-foundation: @Aitolkyn already should already have access to Superset as they are part of the analytics_privatedata_users group. @Aitolkyn, can you please confirm?

In T350179#9711211, @Papaul wrote:

@ssingh one thing that I found between the server NiC and the switch interface is the vendor . In Eqiad, I checked 3 nodes cp1115, 1113 and 1100 all have for vendor under Transceiver inventory W2W and in Esams the vendor is FS. Since @ayounsi mentioned this morning that the request was not reaching the switch I focused on the media type used in esams and in eqiad so it looks like both connections are Direct Attach Copper but different vendor.

What i will like to test next

• use a DAC from FS
• just use a transceiver and connect a fiber

@Jclark-ctr @VRiley-WMF if next on site can you please find a DAC cable from FS.com and replace the cable on cp1115 if you have no DAC cable from FS.com can you please use a 10G transceiver with a fiber.

@Papaul suggested to try a host in codfw and cp2042 PXE booted successfully. In one of the above messages, @cmooney suggested looking at if the new QFX5120 in esams/drmrs can be why it works in esams.

Traffic reimaged 8 text nodes in esams and all of them PXE-booted the first time, without any issues. I think looking at why things worked flawlessly in esams but not in other sites such as eqiad and ulsfo is probably how we should try to get to the bottom of this ticket!

For cp1115 that we tried today, I downgraded the BIOS, NIC and iDRAC firmwares, to match what we have in esams, where 6/6 hosts have been reimaged without any issue (PXE-booting the first time).

One more thing I will try to do is to successively try all NIC firmwares in 22.x instead of picking the highest supported version but we can't do multiple reimages in a day as it clears the caches and while one host being down is not a big deal, it's not ideal. So I will report back tomorrow.

Some updates with the TL;DR that it is still failing for hosts in eqiad and ulsfo:

cp4052 BIOS version 1.9.2 also didn't work; no PXE boot. I am going to focus on the install server now and see if we can pick up something there.

cp3069 also did PXE boot successfully, in the first attempt, so it makes it the fourth host in esams to not have any issue. I think maybe focusing on why it works in esams but not in eqiad/ulsfo might be the way forward.

Continuing to trying to isolate the possible causes of this, I noticed when dumping the facter output between the difference hosts (ones that work vs the ones that don't), that the BIOS versions also seems to vary:

In T350179#9692338, @MoritzMuehlenhoff wrote:

We could also consider to pass this over to Dell support?

In T350179#9692080, @cmooney wrote:

In T350179#9690121, @ssingh wrote:

Any other opinions/thoughts on how we can try and fix this and where? I am very happy to do the legwork but kind of lost here on what to check next.

Yeah it's very odd alright. That pattern of firmware versions looked so promising - nice sleuthing all the same!

esams is working fine for us without any issues but ulsfo continues to be an issue and that uncertainty extends to magru as well.

100% clutching at straws here but I wonder if the type of switch is having any effect? esams and drmrs have newer QFX5120 switches (I guess small good news here is magru will have same setup as these locations). ulsfo and (most of) eqiad have older QFX5100 devices. I fail to see why that would make such a difference, but we're at the clutching at straws level so maybe?

Eqiad rows E and F, and codfw rows A and B, have the newer model switches too. So if we have the same problem in those places we can rule out it being anything to do with the switch.

Any other opinions/thoughts on how we can try and fix this and where? I am very happy to do the legwork but kind of lost here on what to check next. The worry continues to be that magru is coming close and we should not be running the cookbook multiple times to get a reimage done. esams is working fine for us without any issues but ulsfo continues to be an issue and that uncertainty extends to magru as well.

Update: I ran the firmware-upgrade cookbook on cp4052 and updated it's firmware to 6.10.30.20, did a racreset to be absolutely sure and it still failed for me in the first attempt. It seems my job that this issue has been resolved was short-lived so we move on and try to debug it more :)

Traffic has been reimaging hosts in esams (we have done three so far for T360430) and we observed that we didn't have this issue on any of those hosts. Relatedly, we reimaged cp4052 last week where we did hit the issue again.

This happened today as well, at 00:35 UTC, when we were paged for this:

For posterity, an annotated Grafana dashboard that shows incoming traffic to esams after and during the depool and power-off events: https://grafana.wikimedia.org/goto/DbIJc7bSk?orgId=1. Note that the current TTL for dyna.wikimedia.org is five minutes, reduced in T140365. This is only for incoming traffic to Varnish, not the number of connections.

Hi, thanks for reporting this. The timing does seem to match an increase in 503s for this period but it was transient due to an increase in the number of requests and seems to have resolved at around ~06:10, with a start time of around ~06:07. As such, there isn't any actionable on our end but the error above is expected in the sense that it was an issue for that period.

@RobH: Verified the hosts, serial numbers, racking and the cadence. Looks good!

Hi Rob: Checking if the date/time above has been confirmed by remote hands?

For another data point: this can also be handy when you are running -b1 -s<something> and want to cancel the execution for any reason; if you know which host was currently being affected, it makes doing so a lot easier.

Rob, once the time/data is confirmed, please let me know here or on IRC and I will send an email to sre@. Thanks!

In T347054#9584958, @Volans wrote:

Even from itself? As in what happens if an operator runs authdns-update on a depooled host?

In T360430#9642721, @RobH wrote:

Remote hands won't have any ability to power down a host other than by pressing the front power button. It would reduce potential complexity if we power down all the hosts for them to work on to prevent confusion. That way they know if it is powered off and matches the list, they can work on it.

Would that adjustment work, and traffic send a power off to those hosts in advance of the work?

Thanks for creating the task. In some further discussion with @BBlack today, we decided that we will do the following:

@cmooney reimaged lvs2011 today and it seems to have gone fine. Sharing this here in case we want to check `lvs' in the list above.

We have finished rolling the changes today, so all state management -- authdns-update, recdns, NTP (Debian installer), authdns-ns[0-2] -- is now managed via confd/conftcl, for all DNS hosts.

Update: We have merged the service depooling change on dns6001. This means that service depooling -- recdns, ntp, authdns-ns2 -- on dns6001, is now managed via confd.

Thanks for sharing @ayounsi!

Hi folks: Just wondering if there is a path forward on this task as we hit the same issue last week while reimaging cp4052. No PXE boot during the first reimage attempt but it worked the second time.

In T347054#9585007, @Volans wrote:

In T347054#9584989, @ssingh wrote:

In T347054#9584958, @Volans wrote:

Even from itself? As in what happens if an operator runs authdns-update on a depooled host?

The NAMESERVERS list that is populated by confd affects only the hosts to which we SSH and run authdns-update, not the host itself. So if you depool dns1004 and run authdns-update from there, nothing changes. If you run authdns-update from dns1005 (or anywhere else), it won't touch dns1004. On my end, I think this behaviour makes sense. But is it fine from your perspective of automation and cookbooks?

I'm more worried for the human side of the problem. In my experience people rely a lot on muscle memory and shell history, so they tend to go always to the same host to run a given command. So I think it would be very easy that someone will run the authdns-update on dns1001 even if depooled just because they're used to run it there.

In T347054#9584973, @Volans wrote:

Do we need to change anything on the sre.dns.netbox cookbook?
It currently runs:

cd {git} && utils/deploy-check.py -g {netbox} --deploy

In T347054#9584958, @Volans wrote:

Even from itself? As in what happens if an operator runs authdns-update on a depooled host?

Status as of today: we are now managing authdns-update state, that is, the list of hosts in /etc/wikimedia-authdns.conf. To depool a host so that it doesn't receive authdns updates (state, not service) you can do something like:

When I looked at this last night as the alerts were coming in, I noticed that some hosts were not reporting the connection failure but simply the content error; example https://puppetboard.wikimedia.org/report/parse1004.eqiad.wmnet/ef81872fca86746fbbd87800da1da74b64d3839b.

Thanks to @MoritzMuehlenhoff, we have imported the forward port of OpenSSL 1.1.1 and have built haproxy 2.6 against it. We will be reimaging a cp host to bookworm.

For further context: we have a request from @dr0ptp4kt for running a Blazegraph experiment and we are trying to free up a cp node for him. So we were wondering if this hardware has still not been hardware decomissioned, we can just bring up a host here.

Hi dc-ops team: quick question: have these hosts already been hardware decommissioned?

We have rolled this out today. For a complete list of domains affected, see the commit above.

In T357449#9539221, @Volans wrote:

The cookbook doesn't reboot the host once in the Debian Installer, it's the Debian Installer that reboots the hosts once the base installation is completed.

Have you checked what was the status of the host during the installation?

One more data point: note that gnt-instance console FQDN is broken because of T309724 so we don't know the exact failure.