Page MenuHomePhabricator
Create Task
Maniphest T346714

Set up kubernetes10[27-56]
Closed, ResolvedPublic
Actions

Description

The new nods kubernetes10[27-56] have been handed over by DCops and should be added to the wikikube cluster

https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_nodes

Details

SubjectRepoBranchLines +/-
conftool: add new k8s nodesoperations/puppetproduction+30 -0
kubernetes: add kubernetes10[27-56] to wikikubeoperations/puppetproduction+31 -6
Add the configuration for the new wikikube hosts in eqiadoperations/homer/publicmaster+31 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedNoneT342533 Q1:rack/setup/install kubernetes10[27-56]
 ResolvedJMeybohmT346714 Set up kubernetes10[27-56]

Event Timeline

Joe created this task.Sep 19 2023, 6:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 19 2023, 6:16 AM
Joe triaged this task as High priority.Sep 19 2023, 6:26 AM
gerritbot added a comment.Sep 19 2023, 6:51 AM

Change 958809 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/homer/public@master] Add the configuration for the new wikikube hosts in eqiad

https://gerrit.wikimedia.org/r/958809

gerritbot added a project: Patch-For-Review.Sep 19 2023, 6:51 AM

Change 958810 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] kubernetes: add kubernetes10[27-56] to wikikube

https://gerrit.wikimedia.org/r/958810

gerritbot added a comment.Sep 19 2023, 6:52 AM

Change 958811 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] conftool: add new k8s nodes

https://gerrit.wikimedia.org/r/958811

JMeybohm mentioned this in T277876: Reserve resources for system daemons on kubernetes nodes.Sep 19 2023, 8:10 AM
gerritbot added a comment.Sep 25 2023, 7:47 AM

Change 958809 merged by jenkins-bot:

[operations/homer/public@master] Add the configuration for the new wikikube hosts in eqiad

https://gerrit.wikimedia.org/r/958809

gerritbot added a comment.Sep 25 2023, 8:24 AM

Change 958810 merged by Effie Mouzeli:

[operations/puppet@production] kubernetes: add kubernetes10[27-56] to wikikube

https://gerrit.wikimedia.org/r/958810

ops-monitoring-bot added a comment.Sep 25 2023, 9:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1030.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 9:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1031.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 9:20 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1032.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 9:53 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1030.eqiad.wmnet with OS bullseye completed:

  • kubernetes1030 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309250936_jiji_1441522_kubernetes1030.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 9:56 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1032.eqiad.wmnet with OS bullseye completed:

  • kubernetes1032 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309250938_jiji_1441630_kubernetes1032.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 9:59 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1031.eqiad.wmnet with OS bullseye completed:

  • kubernetes1031 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309250941_jiji_1441532_kubernetes1031.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1033.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1034.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1035.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1037.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1036 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1039.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1040.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:04 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1038 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1041.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1042.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1036 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:09 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1038 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1043.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1052.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1044.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1045.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1046.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1047.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1048.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1047.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1047 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:25 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1049.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1050.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1051.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1053.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1054.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1056.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1055.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1047.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 25 2023, 10:36 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1042.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1042 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251027_jiji_1458479_kubernetes1042.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:37 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1040.eqiad.wmnet with OS bullseye completed:

  • kubernetes1040 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251020_jiji_1458457_kubernetes1040.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:39 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1041.eqiad.wmnet with OS bullseye completed:

  • kubernetes1041 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251023_jiji_1458475_kubernetes1041.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:40 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1035.eqiad.wmnet with OS bullseye completed:

  • kubernetes1035 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251022_jiji_1458342_kubernetes1035.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:41 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1033.eqiad.wmnet with OS bullseye completed:

  • kubernetes1033 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251020_jiji_1458315_kubernetes1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:43 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1039.eqiad.wmnet with OS bullseye completed:

  • kubernetes1039 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251025_jiji_1458434_kubernetes1039.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:46 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1037.eqiad.wmnet with OS bullseye completed:

  • kubernetes1037 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251027_jiji_1458405_kubernetes1037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:48 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1034.eqiad.wmnet with OS bullseye completed:

  • kubernetes1034 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251027_jiji_1458324_kubernetes1034.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1051.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1051 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251047_jiji_1462081_kubernetes1051.out
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1055.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1055 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251048_jiji_1462477_kubernetes1055.out
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1045.eqiad.wmnet with OS bullseye completed:

  • kubernetes1045 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251043_jiji_1461941_kubernetes1045.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 10:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1043.eqiad.wmnet with OS bullseye completed:

  • kubernetes1043 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251040_jiji_1461932_kubernetes1043.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 11:03 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1056.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1056 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251053_jiji_1462481_kubernetes1056.out
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 25 2023, 11:06 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1048.eqiad.wmnet with OS bullseye completed:

  • kubernetes1048 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251050_jiji_1462028_kubernetes1048.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 25 2023, 11:07 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1044.eqiad.wmnet with OS bullseye completed:

  • kubernetes1044 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251048_jiji_1461937_kubernetes1044.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 11:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1049.eqiad.wmnet with OS bullseye completed:

  • kubernetes1049 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251045_jiji_1462033_kubernetes1049.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 25 2023, 11:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1052.eqiad.wmnet with OS bullseye completed:

  • kubernetes1052 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251045_jiji_1462133_kubernetes1052.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 25 2023, 11:09 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1046.eqiad.wmnet with OS bullseye completed:

  • kubernetes1046 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251049_jiji_1461960_kubernetes1046.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 11:12 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1050.eqiad.wmnet with OS bullseye completed:

  • kubernetes1050 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251047_jiji_1462058_kubernetes1050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 25 2023, 11:13 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1038.eqiad.wmnet with OS bullseye completed:

  • kubernetes1038 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251052_jiji_1475198_kubernetes1038.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 11:16 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1047.eqiad.wmnet with OS bullseye completed:

  • kubernetes1047 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251052_jiji_1471296_kubernetes1047.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 25 2023, 11:16 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1053.eqiad.wmnet with OS bullseye completed:

  • kubernetes1053 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251050_jiji_1462420_kubernetes1053.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 25 2023, 11:16 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1036.eqiad.wmnet with OS bullseye completed:

  • kubernetes1036 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251057_jiji_1475033_kubernetes1036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 25 2023, 11:20 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host kubernetes1054.eqiad.wmnet with OS bullseye completed:

  • kubernetes1054 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309251053_jiji_1462424_kubernetes1054.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
JMeybohm added a comment.Sep 25 2023, 11:37 AM

re-ran puppet on kubernetes[1027-1056].eqiad.wmnet and called the remove-downtime cookbook for them

gerritbot added a comment.Sep 25 2023, 11:41 AM

Change 958811 merged by JMeybohm:

[operations/puppet@production] conftool: add new k8s nodes

https://gerrit.wikimedia.org/r/958811

Stashbot added a comment.Sep 25 2023, 11:42 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T11:42:44Z] <jayme> running puppet on lvs in codfw - T346714

Stashbot added a comment.Sep 25 2023, 11:43 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T11:43:10Z] <jayme> running puppet on lvs in eqiad - T346714 (TYPO from above, did not run in codfw)

JMeybohm closed this task as Resolved.Sep 25 2023, 11:58 AM
JMeybohm claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 12:10 PM
Stashbot added a comment.Sep 25 2023, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-09-25T13:14:40Z] <jayme> ran homer "lsw1-*eqiad*" commit - T346714

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits