Page MenuHomePhabricator
Create Task
Maniphest T372878

Re-IP wikikube servers in codfw row A/B moving to per-rack subnets
Closed, ResolvedPublic
Actions

Description

Per T354869, we have to renumber various wikikube servers in codfw, namely in rows A/B. We 'll also utilize the opportunity for downtime to reimage and rename them.

Handle renumbering and vlan move for wikikube nodes, ideally before T370962: Southward Datacenter Switchover (September 2024)

Per-host tracking sheet:
https://docs.google.com/spreadsheets/d/1HvaScZNUH-toZrlYKNlLyHs5an6rXT5uUxQ1Nl49I5M/edit?usp=sharing

Procedure

Workers

Rename and renumber

  1. Locally, use add_k8s_node.py --puppet-dir $PUPPET_DIR --netbox-url https://netbox.wikimedia.org --netbox-token $NETBOX_TOKEN --task-id T372878 --rename-only --move-vlan $FQDNS (from serviceops kitchensink, make sure to pull main)
  2. Follow the instructions
  3. Make sure that the old names were correctly deactivated from puppetserver sudo puppet node deactivate $old_fqdn
  4. Run homer 'cr*codfw*' commit 'T372878' from cumin to remove the old BGP config

Note: When creating a relabel task, look if there isn't one already open and edit it to cut down on the number of small tasks for DC-Ops

Renumber only

From cumin, run:

  1. sudo cookbook sre.k8s.renumber-node -t T372878 $fqdn
  2. Follow the instructions from the cookbook for homer run (Running homer on the CR can technically be done as soon as the move-vlan part of the reimage cookbook is through)

Control plane

TODO

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
kubernetes: rename mw2427 -> wikikube-worker2127operations/puppetproduction+4 -7
kubernetes: rename mw2426 -> wikikube-worker2126operations/puppetproduction+5 -5
kubernetes: rename mw2425 -> wikikube-worker2125operations/puppetproduction+5 -5
kubernetes: rename mw2424 -> wikikube-worker2124operations/puppetproduction+5 -5
kubernetes: re-name / IP mw231[345]operations/puppetproduction+9 -9
kubernetes20[59-60], mw230[1-5] -> wikikube-worker21[14-20]operations/puppetproduction+16 -19
Rename mw239[0456789] to wikikube-worker21[07-13]operations/puppetproduction+16 -19
kubernetes: Rename two workersoperations/puppetproduction+7 -7
kubernetes: rename mw2431 to wikikube-worker2104operations/puppetproduction+5 -5
kubernetes: rename mw2430 to wikikube-worker2103operations/puppetproduction+5 -5
rename/renumber kubernetes2034 to wikikube-worker2102operations/puppetproduction+4 -4
kubernetes: Rename mw233[2-4]operations/puppetproduction+13 -9
k8s: rename mw232[012], kubernetes2031 to wikikube-workersoperations/puppetproduction+11 -14
k8s: rename mw232[012], kubernetes2031 for wikikube-workersoperations/puppetproduction+11 -14
rename/renumber kubernetes2020,2033 to wikikube-worker2093,2094operations/puppetproduction+6 -6
kubernetes: Rename mw242[01] to wikikube-worker...operations/puppetproduction+7 -7
Rename/Renumber mw2434,2435 to wikikube-worker2089,2090operations/puppetproduction+7 -7
Rename kubernets2009,2010,2035,2054, reimage to bookwormoperations/puppetproduction+10 -10
k8s: re-vlan, rename and reimage mw231[6789]operations/puppetproduction+11 -11
kubernetes: Rename mw2325 to wikikube-worker2079operations/puppetproduction+5 -9
kubernetes: Set wikikube-worker2079 insetupoperations/puppetproduction+5 -1
kubernetes: re-name/IP mw2260 and mw2312operations/puppetproduction+8 -8
kubernetes: Rename mw240[267] to wikikube-worker...operations/puppetproduction+8 -11
site: fix role for reimaged kubernetes hostsoperations/puppetproduction+1 -1
k8s: rename kubernetes20(28|55) and mw242[23] to wikikube-workersoperations/puppetproduction+11 -11
Make k8s/pool-depool-node work on control-planes as welloperations/cookbooksmaster+15 -14
k8s: rename mw238[6789] to use wikikube-worker hostnamesoperations/puppetproduction+10 -13
Rename kubernetes2008,2027 to wikikube-worker206[67]operations/puppetproduction+6 -6
kubernetes: re-name/IP kubernetes20(30|57) as wikikube-worker206[45]operations/puppetproduction+6 -6
k8s: rename mw238[345] to wikikube-worker206[012]operations/puppetproduction+9 -9
Rename mw237[789] to wikikube-worker205[789]operations/puppetproduction+9 -9
k8s: rename mw2382 to wikikube-worker2056operations/puppetproduction+5 -5
k8s: rename mw2381 to wikikube-worker2055operations/puppetproduction+5 -5
Rename mw229[567] to wikikube-worker205[234]operations/puppetproduction+8 -11
Rename mw229[567] to wikikube-worker205[123]operations/puppetproduction+8 -11
kubernetes: Rename mw2401 to wikikube-worker2051operations/puppetproduction+5 -5
k8s: rename mw2380 to wikikube-worker2050operations/puppetproduction+5 -5
kubernetes: re-name/IP kubernetes2029 as wikikube-worker2049operations/puppetproduction+4 -4
Rename mw2295 to wikikube-worker2048operations/puppetproduction+5 -5
Rename/Re-IP kubernetes2007 as wikikube-worker2047operations/puppetproduction+4 -4
kubernetes: re-name/IP kubernetes2026 as wikikube-worker2046operations/puppetproduction+4 -4
Rename mw2293 to wikikube-worker2045operations/puppetproduction+5 -5
Rename kubernetes2019 to wikikube-worker2044operations/puppetproduction+4 -4
Rename mw2292 to wikikube-worker2043operations/puppetproduction+5 -5
kubernetes: re-name/IP kubernetes2025 as wikikube-worker2042operations/puppetproduction+4 -4
kubernetes: rename + re-IP kubernetes2018operations/puppetproduction+4 -4
kubernetes: Rename mw2291 to wikikube-worker2040operations/puppetproduction+5 -5
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjijikiT371881 low rate of mw-memcached errors
OpenNoneT374366 Race condition in iptables rules during puppet runs on k8s nodes
 OpenNoneT354869 Re-IP codfw private baremetal hosts to new per-rack vlans/subnets
 ResolvedakosiarisT372878 Re-IP wikikube servers in codfw row A/B moving to per-rack subnets
 ResolvedJhancock.wmT372916 Relabel codfw kubernetes nodes
 ResolvedNoneT373401 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT373457 Relabel codfw kubernetes nodes
 DuplicateNoneT373491 Relabel codfw kubernetes nodes
 DuplicateNoneT373505 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT373591 Relabel codfw kubernetes nodes
 DuplicateNoneT373669 Relabel codfw kubernetes nodes mw2295,mw2296,mw2297
 DuplicateNoneT373699 Relabel codfw kubernetes nodes mw237[789]
 ResolvedMoritzMuehlenhoffT373819 Issues reimaging kubernetes workers due to user conflicts in systemd-timesyncd
ResolvedJhancock.wmT373916 Relabel codfw kubernetes nodes
 InvalidNoneT373934 Update iDRAC on mw2260.codfw.wmnet
 ResolvedPRODUCTION ERRORClement_GoubertT373982 wikikube-worker2080.codfw.wmnet can't auth to registry
 ResolvedJMeybohmT374019 kubernetes2035 (renamed to wikikube-worker2087) reporting "Comm Error: Backplane 0"
 ResolvedJhancock.wmT374249 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT374258 Comm Error: backplane 0 when reimaging wikikube-worker2095
 ResolvedJhancock.wmT374380 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT374622 Relabel codfw kubernetes nodes mw2390 and mw2394-mw2399
 ResolvedJhancock.wmT374733 Relabel codfw kubernetes nodes
 ResolvedJhancock.wmT375398 Relabel codfw kubernetes nodes mw2424 and mw2425
 ResolvedNoneT381244 Relabel codfw kubernetes nodes

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Sep 13 2024, 1:53 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2114.codfw.wmnet (PASS)
  • wikikube-worker2114 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2114.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 1:53 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet completed:

  • wikikube-worker2114.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2114.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2114 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2114.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2115.codfw.wmnet (PASS)
  • wikikube-worker2115 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2115.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 1:55 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet completed:

  • wikikube-worker2115.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2115.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2115 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2115.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 1:58 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet

ops-monitoring-bot added a comment.Sep 13 2024, 1:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 13 2024, 2:00 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet

ops-monitoring-bot added a comment.Sep 13 2024, 2:00 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet completed:

  • wikikube-worker2115.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2115.codfw.wmnet, sre.hosts.reimage returned 94
ops-monitoring-bot added a comment.Sep 13 2024, 2:00 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet

ops-monitoring-bot added a comment.Sep 13 2024, 2:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 13 2024, 2:01 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2114.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2114.codfw.wmnet (PASS)
  • wikikube-worker2114 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2114.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:01 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2114.codfw.wmnet completed:

  • wikikube-worker2114.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2114.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2114 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2114.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:01 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2115.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2115.codfw.wmnet (PASS)
  • wikikube-worker2115 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2115.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:01 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2115.codfw.wmnet completed:

  • wikikube-worker2115.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2115.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2115 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2115.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:08 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2116.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2116.codfw.wmnet (PASS)
  • wikikube-worker2116 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2116.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:08 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2116.codfw.wmnet completed:

  • wikikube-worker2116.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2116.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2116 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2116.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:18 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2119.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2119.codfw.wmnet (PASS)
  • wikikube-worker2119 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2119.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:18 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2119.codfw.wmnet completed:

  • wikikube-worker2119.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2119.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2119 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2119.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2118.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2118.codfw.wmnet (PASS)
  • wikikube-worker2118 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131411_akosiaris_242015_wikikube-worker2118.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Sep 13 2024, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T14:33:17Z] <akosiaris> homer cr*codfw* commit 'T372878'

Stashbot added a comment.Sep 13 2024, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T14:33:24Z] <akosiaris> homer lsw1-a6-codfw* commit 'T372878'

ops-monitoring-bot added a comment.Sep 13 2024, 2:39 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2117.codfw.wmnet with OS bullseye executed with errors:

  • wikikube-worker2117.codfw.wmnet (PASS)
  • wikikube-worker2117 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2117.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:39 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2117.codfw.wmnet completed:

  • wikikube-worker2117.codfw.wmnet (FAIL)
    • Failed to reimage node wikikube-worker2117.codfw.wmnet, sre.hosts.reimage returned 99
  • wikikube-worker2117 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console wikikube-worker2117.codfw.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 2:41 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2118.codfw.wmnet completed:

  • wikikube-worker2118.codfw.wmnet (FAIL)
    • Successfully reimaged node wikikube-worker2118.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Failed to pool and uncordon node wikikube-worker2118.codfw.wmnet, sre.k8s.pool-depool-node returned 99
  • wikikube-worker2118 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131411_akosiaris_242015_wikikube-worker2118.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2118.codfw.wmnet (PASS)
ops-monitoring-bot added a comment.Sep 13 2024, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wikikube-worker2120.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2120.codfw.wmnet (PASS)
  • wikikube-worker2120 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202409131409_akosiaris_237773_wikikube-worker2120.out, asking the operator what to do
    • First Puppet run failed and the operator skipped it
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Sep 13 2024, 3:01 PM

Change #1072762 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] kubernetes: re-name / IP mw231[345]

https://gerrit.wikimedia.org/r/1072762

gerritbot added a project: Patch-For-Review.Sep 13 2024, 3:01 PM
Stashbot added a comment.Sep 13 2024, 3:12 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T15:12:43Z] <akosiaris> homer lsw1-a6-codfw* commit T372878

ops-monitoring-bot added a comment.Sep 13 2024, 3:17 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2120.codfw.wmnet completed:

  • wikikube-worker2120.codfw.wmnet (FAIL)
    • Successfully reimaged node wikikube-worker2120.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2120.codfw.wmnet
  • wikikube-worker2120 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202409131409_akosiaris_237773_wikikube-worker2120.out, asking the operator what to do
    • First Puppet run failed and the operator skipped it
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2120.codfw.wmnet (PASS)
    • Host wikikube-worker2120.codfw.wmnet pooled in codfw
ops-monitoring-bot added a comment.Sep 13 2024, 3:17 PM

Cookbook cookbooks.sre.k8s.renumber-node started by akosiaris@cumin1002 Renumbering for host wikikube-worker2120.codfw.wmnet completed:

  • wikikube-worker2120.codfw.wmnet (FAIL)
    • Successfully reimaged node wikikube-worker2120.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2120.codfw.wmnet
  • wikikube-worker2120 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202409131409_akosiaris_237773_wikikube-worker2120.out, asking the operator what to do
    • First Puppet run failed and the operator skipped it
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2120.codfw.wmnet (PASS)
    • Host wikikube-worker2120.codfw.wmnet pooled in codfw
gerritbot added a comment.Sep 13 2024, 3:17 PM

Change #1072762 merged by Scott French:

[operations/puppet@production] kubernetes: re-name / IP mw231[345]

https://gerrit.wikimedia.org/r/1072762

ops-monitoring-bot added a comment.Sep 13 2024, 3:27 PM

Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from mw2313 to wikikube-worker2121 completed:

  • mw2313 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Maintenance_bot removed a project: Patch-For-Review.Sep 13 2024, 3:30 PM
ops-monitoring-bot added a comment.Sep 13 2024, 3:33 PM

Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from mw2314 to wikikube-worker2122 completed:

  • mw2314 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 13 2024, 3:39 PM

Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from mw2315 to wikikube-worker2123 completed:

  • mw2315 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 13 2024, 3:50 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by swfrench@cumin2002 Renumbering for host wikikube-worker2121.codfw.wmnet

ops-monitoring-bot added a comment.Sep 13 2024, 3:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2121.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 13 2024, 3:56 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by swfrench@cumin2002 Renumbering for host wikikube-worker2122.codfw.wmnet

ops-monitoring-bot added a comment.Sep 13 2024, 3:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2122.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 13 2024, 4:03 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by swfrench@cumin2002 Renumbering for host wikikube-worker2123.codfw.wmnet

ops-monitoring-bot added a comment.Sep 13 2024, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host wikikube-worker2123.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 13 2024, 4:35 PM

Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2121.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2121.codfw.wmnet (PASS)
  • wikikube-worker2121 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131616_swfrench_4028128_wikikube-worker2121.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Sep 13 2024, 4:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T16:38:54Z] <swfrench-wmf> running homer lsw1-b3-codfw* commit 'T372878'

ops-monitoring-bot added a comment.Sep 13 2024, 4:43 PM

Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2122.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2122.codfw.wmnet (PASS)
  • wikikube-worker2122 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131623_swfrench_4035684_wikikube-worker2122.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 13 2024, 4:46 PM

Cookbook cookbooks.sre.k8s.renumber-node started by swfrench@cumin2002 Renumbering for host wikikube-worker2121.codfw.wmnet completed:

  • wikikube-worker2121.codfw.wmnet (PASS)
    • Successfully reimaged node wikikube-worker2121.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2121.codfw.wmnet
  • wikikube-worker2121 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131616_swfrench_4028128_wikikube-worker2121.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2121.codfw.wmnet (PASS)
    • Host wikikube-worker2121.codfw.wmnet pooled in codfw
ops-monitoring-bot added a comment.Sep 13 2024, 4:50 PM

Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikikube-worker2123.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2123.codfw.wmnet (PASS)
  • wikikube-worker2123 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131631_swfrench_4043602_wikikube-worker2123.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 13 2024, 4:52 PM

Cookbook cookbooks.sre.k8s.renumber-node started by swfrench@cumin2002 Renumbering for host wikikube-worker2122.codfw.wmnet completed:

  • wikikube-worker2122.codfw.wmnet (PASS)
    • Successfully reimaged node wikikube-worker2122.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2122.codfw.wmnet
  • wikikube-worker2122 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131623_swfrench_4035684_wikikube-worker2122.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2122.codfw.wmnet (PASS)
    • Host wikikube-worker2122.codfw.wmnet pooled in codfw
Stashbot added a comment.Sep 13 2024, 4:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-13T16:57:49Z] <swfrench-wmf> running homer cr*codfw* commit 'T372878'

ops-monitoring-bot added a comment.Sep 13 2024, 4:59 PM

Cookbook cookbooks.sre.k8s.renumber-node started by swfrench@cumin2002 Renumbering for host wikikube-worker2123.codfw.wmnet completed:

  • wikikube-worker2123.codfw.wmnet (PASS)
    • Successfully reimaged node wikikube-worker2123.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2123.codfw.wmnet
  • wikikube-worker2123 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409131631_swfrench_4043602_wikikube-worker2123.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2123.codfw.wmnet (PASS)
    • Host wikikube-worker2123.codfw.wmnet pooled in codfw
Jhancock.wm closed subtask T374733: Relabel codfw kubernetes nodes as Resolved.Sep 16 2024, 2:58 PM
gerritbot added a comment.Sep 20 2024, 12:47 PM

Change #1074410 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kubernetes: mw2313 -> wikikube-worker2124

https://gerrit.wikimedia.org/r/1074410

gerritbot added a project: Patch-For-Review.Sep 20 2024, 12:47 PM
gerritbot added a comment.Sep 23 2024, 8:08 AM

Change #1074410 merged by Effie Mouzeli:

[operations/puppet@production] kubernetes: rename mw2424 -> wikikube-worker2124

https://gerrit.wikimedia.org/r/1074410

ops-monitoring-bot added a comment.Sep 23 2024, 8:21 AM

Cookbook cookbooks.sre.hosts.rename started by jiji@cumin1002 from mw2424 to wikikube-worker2124 completed:

  • mw2424 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 23 2024, 8:30 AM

Cookbook cookbooks.sre.k8s.renumber-node was started by jiji@cumin1002 Renumbering for host wikikube-worker2124.codfw.wmnet

ops-monitoring-bot added a comment.Sep 23 2024, 8:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host wikikube-worker2124.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2024, 8:31 AM
ops-monitoring-bot added a comment.Sep 23 2024, 9:48 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host wikikube-worker2124.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2124.codfw.wmnet (PASS)
  • wikikube-worker2124 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409230927_jiji_1054100_wikikube-worker2124.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Sep 23 2024, 9:53 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-23T09:53:02Z] <effie> homer cr*codfw* commit 'T372878'

ops-monitoring-bot added a comment.Sep 23 2024, 9:59 AM

Cookbook cookbooks.sre.k8s.renumber-node started by jiji@cumin1002 Renumbering for host wikikube-worker2124.codfw.wmnet completed:

  • wikikube-worker2124.codfw.wmnet (FAIL)
    • Successfully reimaged node wikikube-worker2124.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Failed to pool and uncordon node wikikube-worker2124.codfw.wmnet, sre.k8s.pool-depool-node returned 99
  • wikikube-worker2124 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409230927_jiji_1054100_wikikube-worker2124.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2124.codfw.wmnet (PASS)
Stashbot added a comment.Sep 23 2024, 10:25 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-23T10:25:41Z] <effie> homer lsw1-a6-codfw* commit 'T372878'

gerritbot added a comment.Sep 23 2024, 11:54 AM

Change #1074987 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kubernetes: rename mw2425 -> wikikube-worker2125

https://gerrit.wikimedia.org/r/1074987

gerritbot added a project: Patch-For-Review.Sep 23 2024, 11:54 AM
gerritbot added a comment.Sep 23 2024, 12:00 PM

Change #1074987 merged by Effie Mouzeli:

[operations/puppet@production] kubernetes: rename mw2425 -> wikikube-worker2125

https://gerrit.wikimedia.org/r/1074987

ops-monitoring-bot added a comment.Sep 23 2024, 12:18 PM

Cookbook cookbooks.sre.hosts.rename started by jiji@cumin1002 from mw2425 to wikikube-worker2125 completed:

  • mw2425 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 23 2024, 12:27 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by jiji@cumin1002 Renumbering for host wikikube-worker2125.codfw.wmnet

ops-monitoring-bot added a comment.Sep 23 2024, 12:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host wikikube-worker2125.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 23 2024, 1:18 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host wikikube-worker2125.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2125.codfw.wmnet (PASS)
  • wikikube-worker2125 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409231258_jiji_1227176_wikikube-worker2125.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Sep 23 2024, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-23T13:21:02Z] <effie> homer cr*codfw* commit 'T372878'

Stashbot added a comment.Sep 23 2024, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-23T13:21:24Z] <effie> homer lsw1-a6-codfw* commit 'T372878'

Maintenance_bot removed a project: Patch-For-Review.Sep 23 2024, 1:30 PM
ops-monitoring-bot added a comment.Sep 23 2024, 1:33 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jiji@cumin1002 Renumbering for host wikikube-worker2125.codfw.wmnet completed:

  • wikikube-worker2125.codfw.wmnet (FAIL)
    • Successfully reimaged node wikikube-worker2125.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on deployment servers
    • Failed to run puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2125.codfw.wmnet
  • wikikube-worker2125 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409231258_jiji_1227176_wikikube-worker2125.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2125.codfw.wmnet (PASS)
    • Host wikikube-worker2125.codfw.wmnet pooled in codfw
ops-monitoring-bot added a comment.Sep 23 2024, 1:33 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jiji@cumin1002 Renumbering for host wikikube-worker2125.codfw.wmnet completed:

  • wikikube-worker2125.codfw.wmnet (FAIL)
    • Successfully reimaged node wikikube-worker2125.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Failed to run puppet agent on deployment servers
    • Successfully ran puppet agent on deployment servers
    • Failed to run puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2125.codfw.wmnet
  • wikikube-worker2125 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409231258_jiji_1227176_wikikube-worker2125.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2125.codfw.wmnet (PASS)
    • Host wikikube-worker2125.codfw.wmnet pooled in codfw
gerritbot added a comment.Sep 24 2024, 9:39 AM

Change #1075149 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kubernetes: rename mw2426 -> wikikube-worker2126

https://gerrit.wikimedia.org/r/1075149

gerritbot added a project: Patch-For-Review.Sep 24 2024, 9:39 AM
gerritbot added a comment.Sep 24 2024, 9:43 AM

Change #1075149 merged by Effie Mouzeli:

[operations/puppet@production] kubernetes: rename mw2426 -> wikikube-worker2126

https://gerrit.wikimedia.org/r/1075149

ops-monitoring-bot added a comment.Sep 24 2024, 10:02 AM

Cookbook cookbooks.sre.hosts.rename started by jiji@cumin1002 from mw2426 to wikikube-worker2126 completed:

  • mw2426 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Sep 24 2024, 10:05 AM

Cookbook cookbooks.sre.k8s.renumber-node was started by jiji@cumin1002 Renumbering for host wikikube-worker2126.codfw.wmnet

ops-monitoring-bot added a comment.Sep 24 2024, 10:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host wikikube-worker2126.codfw.wmnet with OS bullseye

gerritbot added a comment.Sep 24 2024, 10:19 AM

Change #1075158 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):

[operations/puppet@production] kubernetes: rename mw2427 -> wikikube-worker2127

https://gerrit.wikimedia.org/r/1075158

ops-monitoring-bot added a comment.Sep 24 2024, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host wikikube-worker2126.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2126.codfw.wmnet (PASS)
  • wikikube-worker2126 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409241034_jiji_1729962_wikikube-worker2126.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Sep 24 2024, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-24T11:31:20Z] <effie> homer cr*codfw* commit 'T372878'

Stashbot added a comment.Sep 24 2024, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-24T11:31:31Z] <effie> homer lsw1-a6-codfw* commit 'T372878'

ops-monitoring-bot added a comment.Sep 24 2024, 11:38 AM

Cookbook cookbooks.sre.k8s.renumber-node started by jiji@cumin1002 Renumbering for host wikikube-worker2126.codfw.wmnet completed:

  • wikikube-worker2126.codfw.wmnet (PASS)
    • Successfully reimaged node wikikube-worker2126.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2126.codfw.wmnet
  • wikikube-worker2126 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409241034_jiji_1729962_wikikube-worker2126.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2126.codfw.wmnet (PASS)
    • Host wikikube-worker2126.codfw.wmnet pooled in codfw
gerritbot added a comment.Sep 24 2024, 12:05 PM

Change #1075158 merged by Effie Mouzeli:

[operations/puppet@production] kubernetes: rename mw2427 -> wikikube-worker2127

https://gerrit.wikimedia.org/r/1075158

ops-monitoring-bot added a comment.Sep 24 2024, 12:17 PM

Cookbook cookbooks.sre.hosts.rename started by jiji@cumin1002 from mw2427 to wikikube-worker2127 completed:

  • mw2427 (WARN)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ⚠️Rollback initiated but nothing to rollback (too soon or too late).⚠️
ops-monitoring-bot added a comment.Sep 24 2024, 12:30 PM

Cookbook cookbooks.sre.hosts.rename started by jiji@cumin1002 from mw2427 to wikikube-worker2127 completed:

  • mw2427 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
Maintenance_bot removed a project: Patch-For-Review.Sep 24 2024, 12:31 PM
ops-monitoring-bot added a comment.Sep 24 2024, 12:32 PM

Cookbook cookbooks.sre.k8s.renumber-node was started by jiji@cumin1002 Renumbering for host wikikube-worker2127.codfw.wmnet

ops-monitoring-bot added a comment.Sep 24 2024, 12:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host wikikube-worker2127.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 24 2024, 1:18 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host wikikube-worker2127.codfw.wmnet with OS bullseye completed:

  • wikikube-worker2127.codfw.wmnet (PASS)
  • wikikube-worker2127 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409241258_jiji_1805090_wikikube-worker2127.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Sep 24 2024, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-24T13:21:09Z] <effie> homer lsw1-a6-codfw* commit 'T372878

Stashbot added a comment.Sep 24 2024, 1:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-09-24T13:23:51Z] <kamila_> homer cr*codfw* commit T372878

ops-monitoring-bot added a comment.Sep 24 2024, 1:27 PM

Cookbook cookbooks.sre.k8s.renumber-node started by jiji@cumin1002 Renumbering for host wikikube-worker2127.codfw.wmnet completed:

  • wikikube-worker2127.codfw.wmnet (PASS)
    • Successfully reimaged node wikikube-worker2127.codfw.wmnet
    • Successfully set BGP to true in Netbox
    • Successfully ran puppet agent on deployment servers
    • Successfully ran puppet agent on registry servers
    • Pooled and uncordoned node wikikube-worker2127.codfw.wmnet
  • wikikube-worker2127 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409241258_jiji_1805090_wikikube-worker2127.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
  • wikikube-worker2127.codfw.wmnet (PASS)
    • Host wikikube-worker2127.codfw.wmnet pooled in codfw
Jhancock.wm closed subtask T375398: Relabel codfw kubernetes nodes mw2424 and mw2425 as Resolved.Sep 24 2024, 2:02 PM
JMeybohm mentioned this in T365571: Rename wikikube worker nodes during OS reimage.Sep 26 2024, 7:32 AM
akosiaris closed this task as Resolved.Oct 16 2024, 2:31 PM
akosiaris claimed this task.

I 'll resolve this. All hosts that could be renumbered have been renumbered. 6 hosts have been decomed instead. And then there is a number of jobrunners that will hopefully soon be reimaged to wikikube-worker nodes.

Jhancock.wm closed subtask T381244: Relabel codfw kubernetes nodes as Resolved.Dec 2 2024, 5:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits