kubernetes20[57-60].codfw.wmnet have been delivered by DC-Ops and need to be setup/added do the cluster.
https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_nodes
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Unknown Object (Task) | |||||
| Resolved | Jhancock.wm | T349873 Q2:rack/setup/install 4 parsoid hosts | |||
| Resolved | Clement_Goubert | T352369 setup/install kubernetes20[57-60] |
Event Timeline
Comment Actions
Change 979029 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/homer/public@master] kubernetes20[57-60]: Add to codfw.k8s
Comment Actions
Change 979030 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] wikikube: put kubernetes20[57-60] in production
Comment Actions
Change 979031 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):
[operations/puppet@production] wikikube: add kubernetes20[57-60] to LVS
Comment Actions
Change 979030 merged by Clément Goubert:
[operations/puppet@production] wikikube: put kubernetes20[57-60] in production
Comment Actions
Change 979029 merged by jenkins-bot:
[operations/homer/public@master] kubernetes20[57-60]: Add to codfw.k8s
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2057.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2058.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2059.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host kubernetes2060.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2057.codfw.wmnet with OS bullseye completed:
- kubernetes2057 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311301114_cgoubert_2208997_kubernetes2057.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2058.codfw.wmnet with OS bullseye completed:
- kubernetes2058 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311301122_cgoubert_2214688_kubernetes2058.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2060.codfw.wmnet with OS bullseye completed:
- kubernetes2060 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311301125_cgoubert_2214984_kubernetes2060.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host kubernetes2059.codfw.wmnet with OS bullseye completed:
- kubernetes2059 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311301128_cgoubert_2214804_kubernetes2059.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-11-30T12:06:07Z] <claime> Running homer 'cr*codfw*' commit T352369
Comment Actions
Change 979031 merged by Clément Goubert:
[operations/puppet@production] wikikube: add kubernetes20[57-60] to LVS
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-11-30T12:22:34Z] <claime> Pooling kubernetes20(5[4789]|60).codfw.wmnet - T352369
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2023-11-30T12:24:25Z] <claime> Uncordoning kubernetes20(5[4789]|60).codfw.wmnet - T352369