Page MenuHomePhabricator
Create Task
Maniphest T377008

wikikube-worker21[28-35] implementation tracking
Closed, ResolvedPublic
Actions

Description

wikikube-worker21[28-35] implementation tracking

This task is to track the service implementation of serviceops host(s) listed in the task description.

Once the linked racking task has been resolved, this task can be implemented.

This sub-task creation/update is per the request of serviceops; this task is assigned at creation to the 'Sub-team Technical Contact' provided in the initial ordering task.

Details

SubjectRepoBranchLines +/-
wikikube: Add wikikube-worker21[28-35]operations/puppetproduction+19 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Oct 11 2024, 3:23 PM
RobH unsubscribed.
Clement_Goubert triaged this task as Medium priority.Nov 8 2024, 3:47 PM
Clement_Goubert moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.
JMeybohm mentioned this in T377877: Migrate wikikube-codfw to containerd.Nov 11 2024, 11:19 AM
gerritbot added a comment.Nov 13 2024, 11:48 AM

Change #1090829 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] wikikube: Add wikikube-worker21[28-35]

https://gerrit.wikimedia.org/r/1090829

gerritbot added a project: Patch-For-Review.Nov 13 2024, 11:48 AM
gerritbot added a comment.Nov 13 2024, 12:01 PM

Change #1090829 merged by Clément Goubert:

[operations/puppet@production] wikikube: Add wikikube-worker21[28-35]

https://gerrit.wikimedia.org/r/1090829

ops-monitoring-bot added a comment.Nov 13 2024, 12:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 12:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2129.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 12:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2130.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2024, 12:31 PM
ops-monitoring-bot added a comment.Nov 13 2024, 12:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2131.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 12:44 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2128 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2128.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 13 2024, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 12:54 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2128 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2128.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 13 2024, 12:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2129.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2129 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2129.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 13 2024, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2128 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2128.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 13 2024, 2:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 2:36 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2130.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2130 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2130.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 13 2024, 2:36 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2131.codfw.wmnet with OS bookworm executed with errors:

  • wikikube-worker2131 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console wikikube-worker2131.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 13 2024, 2:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2129.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 3:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2130.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 3:14 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2128.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2128 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131455_cgoubert_2054197_wikikube-worker2128.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 3:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2131.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 3:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2129.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2129 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131518_cgoubert_2056052_wikikube-worker2129.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 3:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2132.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 3:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2133.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 3:53 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2130.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2130 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131534_cgoubert_2060994_wikikube-worker2130.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 3:56 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2131.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2131 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131537_cgoubert_2061402_wikikube-worker2131.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 4:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2134.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2135.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 13 2024, 4:24 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2132.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2132 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131605_cgoubert_2067437_wikikube-worker2132.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2133.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2133 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131608_cgoubert_2067710_wikikube-worker2133.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2134.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2134 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131629_cgoubert_2074334_wikikube-worker2134.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 13 2024, 4:50 PM

Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2135.codfw.wmnet with OS bookworm completed:

  • wikikube-worker2135 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411131631_cgoubert_2074467_wikikube-worker2135.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 13 2024, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T16:58:22Z] <claime> homer 'lsw1-b2-codfw*' commit T377008

Stashbot added a comment.Nov 13 2024, 5:01 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T17:01:15Z] <claime> homer 'lsw1-b4-codfw*' commit T377008

Stashbot added a comment.Nov 13 2024, 5:02 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T17:02:24Z] <claime> homer 'cr*codfw*' commit T377008

Stashbot added a comment.Nov 13 2024, 5:17 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T17:17:25Z] <claime> homer 'lsw1-c4-codfw*' commit 'T377008'

Stashbot added a comment.Nov 13 2024, 5:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T17:18:08Z] <claime> homer 'lsw1-d4-codfw*' commit 'T377008'

Stashbot added a comment.Nov 13 2024, 5:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T17:18:54Z] <claime> homer 'lsw1-c2-codfw*' commit 'T377008'

Stashbot added a comment.Nov 13 2024, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-13T17:20:18Z] <claime> homer 'lsw1-d2-codfw*' commit 'T377008'

ops-monitoring-bot added a comment.Nov 13 2024, 5:23 PM

pool host wikikube-worker[2128-2135].codfw.wmnet by cgoubert@cumin1002 with reason: New nodes

ops-monitoring-bot added a comment.Nov 13 2024, 5:23 PM

Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1002 pool for host wikikube-worker[2128-2135].codfw.wmnet completed:

  • wikikube-worker[2128-2135].codfw.wmnet (PASS)
    • Host wikikube-worker[2128-2135].codfw.wmnet pooled in wikikube-codfw
Clement_Goubert closed this task as Resolved.Nov 13 2024, 5:26 PM
Clement_Goubert claimed this task.

Nodes imaged and pooled, resolving. Decom of refreshed hosts will be tracked in T379788

Jhancock.wm closed subtask T379788: Decommission kubernetes20[07-14].codfw.wmnet as Resolved.Thu, Dec 12, 5:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits