Reimaged pybal-test2003 to bullseye, added component/pybal and everything appears to be fine with the installation.

This has been resolved with the https://gerrit.wikimedia.org/r/898957 and all R:Class = dnsrecursor hosts running bullseye:

We reimaged two hosts to bullseye and didn't notice any auditd failure, so confirming what @MoritzMuehlenhoff said above and marking this as resolved.

Closing this in favour of T321309 where it is being tracked and also given that the Ganeti reimaging cookbook exists which was the primary motivation behind this task.

@ayounsi, @cmooney: Quick question about Junos OS: so we are planning to spread ns0 over dns100[123] and ns1 over dns200[123], similar to how we are doing with ns2:

Thanks to @Vgutierrez for taking care of the rollout of this. For posterity, the final result for now before we do more enhancements:

cr2-codfw (replicated to cr1-codfw as well):

cr2-eqiad (replicated to cr1-eqiad as well):

In T331478#8676676, @SLyngshede-WMF wrote:

@BCornwall / @ssingh we've removed the clear dhcp cache part of the cookbook. It's technically not need at this point, as no interfaces or IPs move during reimaging.

That should allow the cookbook to run to completion, also in DRMRS.

In T331478#8676530, @Volans wrote:

In T331478#8674505, @Volans wrote:

From a quick look the current data is correct and doesn't error out:

>>> node.primary_ip.assigned_object.connected_endpoint.device
asw-0603-eqsin

Discard this, I tested it against ncredir5001 as reported in the task description, but that's a typo, the actual error is for ncredir6001 and only drmrs hosts failed.

First of all, thanks so much for the Ganeti cookbook -- it's a lifesaver. I can't imagine reimaging these hosts without the cookbook and all the manual hours that would have gone into that, so much thanks to @SLyngshede-WMF and @Volans for working on it!

In T330670#8652102, @jbond wrote:

lgtm just some curiosity :)

After the above change, we will have three DNS boxes in the core DCs, with ns0 pointing to dns1001 in eqiad and ns1 pointing to dns2001

Curious why you dont point
ns0 -> dns1001 & dns1002
ns1 -> dns2001 & dns2002

In T330670#8649554, @Volans wrote:

@ssing

1. for the cookbooks all that I see is that they use the A:dns-auth cumin alias, so they will follow along.

Confirming that I did a new reimage and it completed successfully. Thanks everyone who worked on this to resolve it so quickly.

In T330318#8640963, @Volans wrote:

@ssingh just run the Netbox script https://netbox.wikimedia.org/extras/scripts/interface_automation.ImportPuppetDB/ against the reimaged host (checking the commit changes checkbox) and you should be good to go.

FYI the fix is about to be released to prod.

Thanks all for the quick response to this task, everyone!

Hi @BCornwall: Thanks for checking! I am pretty convinced that this is not related to the NIC firmware, for the following reasons; but I may be missing something and so just writing it here:

In T321309#8582443, @MoritzMuehlenhoff wrote:

In T321309#8581111, @ssingh wrote:

Steps to follow for manual upgrade of the iDRAC firmwares for the cp hosts in eqiad for us and in case someone else stumbles on this issue.

Can you please mention/link or integrate this into https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation ? We'll probably run into this with other servers as well.

Steps to follow for manual upgrade of the iDRAC firmwares for the cp hosts in eqiad for us and in case someone else stumbles on this issue.

Using a slight modification of @jbond's script in T328593, the list of cp nodes in eqiad with the oudated firmware (3.15.17.15) is basically all the cp nodes in eqiad:

In T321309#8579719, @jbond wrote:

@ssingh i have finished with cp1075, i have upgraded it to the most recent network, bios and idrac version. in relation to other servers that you may have issues with i have noticed that any machine with an idrac version < 3.30.30.30 first needs to have a manual upgrade to that version. the script cant work for idrac below that version and you have to upgrade to that version before progressing to later version. after you are on 3.30.30.30 you should be able to use the script to upgrade to the most recent (although the lower limit may change to 4.40.0.0 in the future, see T328593 for more info)

@Volans and I were discussing this on IRC today, some more observations with cp5019, that failed the first attempt but worked on the second.

For posterity, the versions of the iDRAC and the NIC firmware that we are looking for for the cp hosts bullseye upgrade and that we pass to the firmware cookbook/upload on the HTTP management interface: