While attempting to disable puppet, we found these hosts to be inaccessible via both cumin & ssh (ssh: connect to host wikikube-ctrl2004.codfw.wmnet port 22: Connection timed out) consistent also with the hosts being down on icinga [7].

Since we have yet to automate reverse DNS zone delegation for k8s pod IP ranges [4] (context T376291), I've manually added the delegations for these hosts here [5].

Codfw has been successfully repooled as of 13:41 UTC this morning. The cookbooks have mistakenly been tag logged a different task, so pasting the summaries below:

Much apologies, those runs were logged incorrectly above. Meant them for T413974 rather.

Resolving as this has been addressed in [0]

Hi folks, for visibility, we will be repooling codfw tomorrow, April 2nd at 14:00 UTC, rather than 15:00 UTC.

For visibility: during the UEFI reboot - sre.hosts.provision cookbook fails on both these hosts [0] while throwing the exception:

Thanks Scott~ Switchover docs have been updated to note the change. I've noted it under [0] however feel free to let me know if it should be more/less visible documentation wise.

For visibility, I've intentionally aborted the reimage due to the following [0] which I will investigate & re-attempt.

==> Unable to verify that the host is inside the Debian installer, please verify manually with: sudo install-console wikikube-ctrl2006.codfw.wmnet

In T400969#11529307, @Clement_Goubert wrote:

@jasmine_ please resolve this task when done with T409104 T409103

hosts have been added to the cluster :) thanks for the reviews @Raine & @Clement_Goubert!

Separately noted: We will follow this up by removing references in puppet, followed by removing the hosts discovery aliases.

In T383227#11251360, @Jclark-ctr wrote:

@Jasmin when you finish these up when it is ready Please assign over to me and add ops-eqiad tag

Thanks