Page MenuHomePhabricator
Create Task
Maniphest T380350

wikikube-worker13[13-27] implementation tracking
Closed, ResolvedPublic
Actions

Description

wikikube-worker13[13-27] implementation tracking

This task is to track the service implementation of serviceops host(s) listed in the task description.

Once the linked racking task has been resolved, this task can be implemented.

This sub-task creation/update is per the request of serviceops; this task is assigned at creation to the 'Sub-team Technical Contact' provided in the initial ordering task.

1.) Extend the hostname globs as appropriate in puppet/manifests/site.pp. Remove the entries for the pre-rename hostnames.
2.) Verify and commit changes to puppet repo, review, merge etc.
3.) Run the reimage cookbook:
Done:

sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1313
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1314
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1315
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1316
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1317
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1318
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1319
sudo cookbook sre.hosts.reimage --force-dhcp-tftp -t T380350 --os bookworm wikikube-worker1320
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1321
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1322
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1323
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1324
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1325
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1326
sudo cookbook sre.hosts.reimage -t T380350 --os bookworm wikikube-worker1327

4.) Update Netbox' (remember to run homer afterwards and !log your action on #wikimedia-operations):
./add_k8s_node.py --netbox-token $NETBOX_TOKEN --netbox-commit --task-id T380350 wikikube-worker13[13-27].eqiad.wmnet
5.) Pool the new nodes:
sudo cookbook sre.k8s.pool-depool-node --k8s-cluster wikikube-eqiad -t T380350 pool wikikube-worker13[13-27].eqiad.wmnet

Details

SubjectRepoBranchLines +/-
Revert "wikikube: Add wikikube-worker13[13-28]"operations/puppetproduction+6 -31
wikikube: Add wikikube-worker13[13-28]operations/puppetproduction+33 -6
wikikube: Remove wikikube-worker1328operations/puppetproduction+1 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT378185 Q2:rack/setup/install wikikube-worker13[13-28]
 ResolvedClement_GoubertT380350 wikikube-worker13[13-27] implementation tracking

Event Timeline

Clement_Goubert created this task.Wed, Nov 20, 11:00 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Nov 20, 11:00 AM
Clement_Goubert triaged this task as Medium priority.Wed, Nov 20, 11:01 AM
Clement_Goubert moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.
Clement_Goubert updated the task description. (Show Details)Thu, Nov 21, 2:23 PM
gerritbot added a comment.Fri, Nov 22, 10:51 AM

Change #1094381 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] wikikube: Add wikikube-worker13[13-28]

https://gerrit.wikimedia.org/r/1094381

gerritbot added a project: Patch-For-Review.Fri, Nov 22, 10:51 AM
gerritbot added a comment.Mon, Nov 25, 12:02 PM

Change #1094381 merged by Clément Goubert:

[operations/puppet@production] wikikube: Add wikikube-worker13[13-28]

https://gerrit.wikimedia.org/r/1094381

Clement_Goubert renamed this task from wikikube-worker13[13-28] implementation tracking to wikikube-worker13[13-27] implementation tracking.Mon, Nov 25, 12:06 PM
Clement_Goubert updated the task description. (Show Details)
gerritbot added a comment.Mon, Nov 25, 12:10 PM

Change #1097361 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] wikikube: Remove wikikube-worker1328

https://gerrit.wikimedia.org/r/1097361

gerritbot added a comment.Mon, Nov 25, 12:12 PM

Change #1097361 merged by Clément Goubert:

[operations/puppet@production] wikikube: Remove wikikube-worker1328

https://gerrit.wikimedia.org/r/1097361

Maintenance_bot removed a project: Patch-For-Review.Mon, Nov 25, 12:31 PM
ops-monitoring-bot added a comment.Mon, Nov 25, 12:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1313.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1314.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1315.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1316.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1317.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1318.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1319.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, Nov 25, 12:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1320.eqiad.wmnet with OS bookworm

JMeybohm mentioned this in T380731: Reboots of Bookworm systems which use 6.1.115.Mon, Nov 25, 12:50 PM
ops-monitoring-bot added a comment.Mon, Nov 25, 1:24 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1313.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1313 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251304_cgoubert_303715_wikikube-worker1313.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:27 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1316.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1316 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251308_cgoubert_303917_wikikube-worker1316.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:30 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1317.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1317 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251311_cgoubert_304019_wikikube-worker1317.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1315.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1315 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251313_cgoubert_303847_wikikube-worker1315.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1319.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1319 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251317_cgoubert_304197_wikikube-worker1319.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:41 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1314.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1314 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251321_cgoubert_303773_wikikube-worker1314.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:43 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1320.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1320 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251324_cgoubert_304292_wikikube-worker1320.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 25, 1:47 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1318.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1318 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251328_cgoubert_304102_wikikube-worker1318.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, Nov 25, 2:22 PM

Change #1097392 had a related patch set uploaded (by CDanis; author: CDanis):

[operations/puppet@production] Revert "wikikube: Add wikikube-worker13[13-28]"

https://gerrit.wikimedia.org/r/1097392

gerritbot added a project: Patch-For-Review.Mon, Nov 25, 2:22 PM
gerritbot added a comment.Mon, Nov 25, 2:35 PM

Change #1097392 merged by Clément Goubert:

[operations/puppet@production] Revert "wikikube: Add wikikube-worker13[13-28]"

https://gerrit.wikimedia.org/r/1097392

Maintenance_bot removed a project: Patch-For-Review.Mon, Nov 25, 3:31 PM
Clement_Goubert changed the task status from Open to Stalled.Mon, Nov 25, 4:34 PM

Because of T375845: WikiKube clusters close to exhausting Calico IPPool allocations, putting these nodes in production needs to wait for T379599: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters to be completed to have enough ip blocks to proceed
.

JMeybohm changed the task status from Stalled to Open.Tue, Nov 26, 9:24 AM
JMeybohm subscribed.

Hosts from T379599: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters have been decommissioned

Clement_Goubert updated the task description. (Show Details)Tue, Nov 26, 3:44 PM
Clement_Goubert updated the task description. (Show Details)
ops-monitoring-bot added a comment.Tue, Nov 26, 3:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 3:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 3:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 3:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 3:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:20 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1323 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1323.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1322 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1322.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:27 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1324 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1324.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:27 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1325 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1325.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:27 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1326 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1326.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:28 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1327 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1327.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:28 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1321 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1321.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm executed with errors:

  • wikikube-worker1321 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker1321.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 26, 4:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Nov 26, 5:03 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1323 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261645_cgoubert_618650_wikikube-worker1323.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Nov 26, 5:10 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1322 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261651_cgoubert_621385_wikikube-worker1322.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Nov 26, 5:13 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1327 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261654_cgoubert_621568_wikikube-worker1327.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Nov 26, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1324 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261658_cgoubert_621433_wikikube-worker1324.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Nov 26, 5:20 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1326 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261701_cgoubert_621524_wikikube-worker1326.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Nov 26, 5:23 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1325 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261704_cgoubert_621468_wikikube-worker1325.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Nov 26, 5:28 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm completed:

  • wikikube-worker1321 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261709_cgoubert_623223_wikikube-worker1321.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Tue, Nov 26, 5:31 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:31:55Z] <claime> homer 'lsw1-f7-eqiad*' commit 'T380350'

Stashbot added a comment.Tue, Nov 26, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:32:56Z] <claime> homer 'lsw1-e6-eqiad*' commit 'T380350'

Stashbot added a comment.Tue, Nov 26, 5:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:33:26Z] <claime> homer 'lsw1-e5-eqiad*' commit 'T380350'

Stashbot added a comment.Tue, Nov 26, 5:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:34:00Z] <claime> homer 'lsw1-f5-eqiad*' commit 'T380350'

Stashbot added a comment.Tue, Nov 26, 5:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:34:28Z] <claime> homer 'lsw1-f6-eqiad*' commit 'T380350'

Stashbot added a comment.Tue, Nov 26, 5:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:35:01Z] <claime> homer 'lsw1-e7-eqiad*' commit 'T380350'

Stashbot added a comment.Tue, Nov 26, 5:35 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-26T17:35:30Z] <claime> homer 'cr*eqiad*' commit 'T380350'

ops-monitoring-bot added a comment.Tue, Nov 26, 5:47 PM

pool host wikikube-worker[1313-1327].eqiad.wmnet by cgoubert@cumin1002 with reason: None

ops-monitoring-bot added a comment.Tue, Nov 26, 5:47 PM

Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1002 pool for host wikikube-worker[1313-1327].eqiad.wmnet completed:

  • wikikube-worker[1313-1327].eqiad.wmnet (PASS)
    • Host wikikube-worker[1313-1327].eqiad.wmnet pooled in wikikube-eqiad
Clement_Goubert closed this task as Resolved.Tue, Nov 26, 5:48 PM
Clement_Goubert claimed this task.
Clement_Goubert updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits