Change #1147746 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Convert snapshot1017 into a dse-k8s-worker host

https://gerrit.wikimedia.org/r/1147746

Change #1147746 merged by Brouberol:

[operations/puppet@production] Convert snapshot1017 into a dse-k8s-worker host

https://gerrit.wikimedia.org/r/1147746

Cookbook cookbooks.sre.hosts.rename started by brouberol@cumin2002 from snapshot1017 to dse-k8s-worker1010 completed:

• snapshot1017 (PASS)
  • ✔️ Downtimed host on Icinga/Alertmanager
  • ✔️ Disabled puppet and its timer
  • ✔️ Disabled debmonitor-client timer
  • ✔️ Netbox updated
  • ✔️ BMC Hostname updated
  • ✔️ DNS updated
  • ✔️ Switch description updated
  • ✔️ Removed from DebMonitor
  • ✔️ Removed from Puppet master and PuppetDB
  • Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin2002 for host dse-k8s-worker1010.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin2002 for host dse-k8s-worker1010.eqiad.wmnet with OS bookworm completed:

• dse-k8s-worker1010 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505191320_brouberol_2073053_dse-k8s-worker1010.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Just checking: Should we evaluate the rack/network distance between the existing set of dse-k8s hosts and these snapshot servers? I ask because we could be creating a small network storm when collecting these into a chatty k8s cluster. No?

Change #1147768 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Convert snapshot1014 into a dse-k8s-worker host

https://gerrit.wikimedia.org/r/1147768

@xcollazo Good question. Looking into netbox, these nodes seem to be pretty well distributed between rows/racks.

In prior experiences, that was the ideal setup as it would allow you to have leeway for workloads requiring anti-affinity and rack-awareness (especially stateful workloads). I don't think the kubelet chatter would be enough to cause a network storm (for example, the eqiad k8s cluster has 247 workers locates across rows/racks:

brouberol@cumin2002:~$ sudo cumin wikikube-worker1*.eqiad.wmnet
247 hosts will be targeted:
wikikube-worker[1002-1007,1011-1012,1015-1016,1019-1168,1240-1289,1291-1327].eqiad.wmnet
DRY-RUN mode enabled, aborting

Data transfers might be something to keep in mind though. For example, large queries, large Ceph writes, etc. That being said, I don't know enough to be able to say more,

dse-k8s-worker1010 is live!

brouberol@dse-k8s-worker1010:~$ sudo crictl  ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
ad2bfc68d1dff       d4034e5d9070e       9 seconds ago       Running             base                0                   b6405f59bd51c       druid-load-editattemptstep-hourly-wait-for-event-edita-h1locd9d
b2b367ed274b1       d4034e5d9070e       6 minutes ago       Running             base                0                   29b86727cd730       webrequest-actor-metrics-hourly-wait-for-wmf-webreques-4rwqrviz
2b2d498d0846e       d4034e5d9070e       8 minutes ago       Running             base                0                   b970711d55ff7       gobblin-webrequest-frontend-webrequest-frontend-usl11iut
8502b2b8acd77       995e3fccdbd70       17 minutes ago      Running             csi-cephfsplugin    0                   3b9ad0d6da168       ceph-csi-cephfs-nodeplugin-4pgjn
480f0d446fba0       995e3fccdbd70       17 minutes ago      Running             csi-rbdplugin       0                   05782996e1522       ceph-csi-rbd-nodeplugin-swk6g
5085957dd995f       38c7f90a0cb64       17 minutes ago      Running             istio-proxy         0                   0b43390240597       istio-ingressgateway-65txg
40abe77e54439       8fe9e7575c40c       17 minutes ago      Running             driver-registrar    0                   3b9ad0d6da168       ceph-csi-cephfs-nodeplugin-4pgjn
b5d0fe3779db9       8fe9e7575c40c       17 minutes ago      Running             driver-registrar    0                   05782996e1522       ceph-csi-rbd-nodeplugin-swk6g
brouberol@dse-k8s-worker1010:~$

Change #1147768 merged by Brouberol:

[operations/puppet@production] Convert snapshot1014 into a dse-k8s-worker host

https://gerrit.wikimedia.org/r/1147768

Cookbook cookbooks.sre.hosts.rename started by brouberol@cumin2002 from snapshot1014 to dse-k8s-worker1011 completed:

• snapshot1014 (PASS)
  • ✔️ Downtimed host on Icinga/Alertmanager
  • ✔️ Disabled puppet and its timer
  • ✔️ Disabled debmonitor-client timer
  • ✔️ Netbox updated
  • ✔️ BMC Hostname updated
  • ✔️ DNS updated
  • ✔️ Switch description updated
  • ✔️ Removed from DebMonitor
  • ✔️ Removed from Puppet master and PuppetDB
  • Rename completed 👍 - now please run the re-image cookbook on the new name with --new

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin2002 for host dse-k8s-worker1011.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin2002 for host dse-k8s-worker1011.eqiad.wmnet with OS bookworm executed with errors:

• dse-k8s-worker1011 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Checked BIOS boot parameters are back to normal
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker1011.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin2002 for host dse-k8s-worker1011.eqiad.wmnet with OS bookworm

The renaming of snapshot1014 and snapshot1017 broke scap mediawiki deploys.

17:39:41 sudo -u mwdeploy -n -- /usr/bin/scap cdb-rebuild (ran as mwdeploy@snapshot1014.eqiad.wmnet) returned [255]: ssh: Could not resolve hostname snapshot1014.eqiad.wmnet: Name or service not known

17:39:41 sudo -u mwdeploy -n -- /usr/bin/scap cdb-rebuild (ran as mwdeploy@snapshot1017.eqiad.wmnet) returned [255]: ssh: Could not resolve hostname snapshot1017.eqiad.wmnet: Name or service not known

leading to errors here:

https://spiderpig.wikimedia.org/

Change #1147826 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] remove decommissioned snapshot10{14,17} from hiera configuration

https://gerrit.wikimedia.org/r/1147826

Change #1147826 merged by Brouberol:

[operations/puppet@production] remove decommissioned snapshot10{14,17} from hiera configuration

https://gerrit.wikimedia.org/r/1147826

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin2002 for host dse-k8s-worker1011.eqiad.wmnet with OS bookworm completed:

• dse-k8s-worker1011 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Add puppet_version metadata (7) to Debian installer
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505191758_brouberol_2207289_dse-k8s-worker1011.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

root@deploy1003:~# kubectl uncordon dse-k8s-worker1011.eqiad.wmnet
node/dse-k8s-worker1011.eqiad.wmnet uncordoned
root@deploy1003:~# kubectl get nodes
NAME                             STATUS   ROLES           AGE     VERSION
dse-k8s-ctrl1001.eqiad.wmnet     Ready    control-plane   2y85d   v1.23.14
dse-k8s-ctrl1002.eqiad.wmnet     Ready    control-plane   2y85d   v1.23.14
dse-k8s-worker1001.eqiad.wmnet   Ready    <none>          2y85d   v1.23.14
dse-k8s-worker1002.eqiad.wmnet   Ready    <none>          2y85d   v1.23.14
dse-k8s-worker1003.eqiad.wmnet   Ready    <none>          2y85d   v1.23.14
dse-k8s-worker1004.eqiad.wmnet   Ready    <none>          2y85d   v1.23.14
dse-k8s-worker1005.eqiad.wmnet   Ready    <none>          2y82d   v1.23.14
dse-k8s-worker1006.eqiad.wmnet   Ready    <none>          2y82d   v1.23.14
dse-k8s-worker1007.eqiad.wmnet   Ready    <none>          2y82d   v1.23.14
dse-k8s-worker1008.eqiad.wmnet   Ready    <none>          2y82d   v1.23.14
dse-k8s-worker1009.eqiad.wmnet   Ready    <none>          262d    v1.23.14
dse-k8s-worker1010.eqiad.wmnet   Ready    <none>          4h40m   v1.23.14
dse-k8s-worker1011.eqiad.wmnet   Ready    <none>          9m14s   v1.23.14

All done!

Change #1147887 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add dse-k8s-worker10[10-11] to the cluster

https://gerrit.wikimedia.org/r/1147887

Change #1147887 merged by Btullis:

[operations/puppet@production] Add dse-k8s-worker10[10-11] to the cluster

https://gerrit.wikimedia.org/r/1147887

Host rebooted by brouberol@cumin2002 with reason: Reboot of new dse-k8s-worker nodes post BGP setup

Host rebooted by brouberol@cumin2002 with reason: Reboot of new dse-k8s-worker nodes post BGP setup

After having enabled BGP on the hosts, run homer and added the nodes to the k8s cluster in puppet, the calico pods are now stable. I've uncordoned dse-k8s-worker1010 for now, and will uncordon the other if everything goes alright, in about 1h.

Change #1148284 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/cookbooks@master] Include the hostname in the phabricator message when rebooting a host

https://gerrit.wikimedia.org/r/1148284

New nodes have been uncordoned and are stable!

Change #1148284 merged by Brouberol:

[operations/cookbooks@master] Include the hostname in the phabricator message when rebooting a host

https://gerrit.wikimedia.org/r/1148284

Change #1148751 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] snapshot: Remove now unused Cumin alias

https://gerrit.wikimedia.org/r/1148751

Change #1148751 merged by Muehlenhoff:

[operations/puppet@production] snapshot: Remove now unused Cumin alias

https://gerrit.wikimedia.org/r/1148751