Page MenuHomePhabricator
Create Task
Maniphest T326420

Kafka-logging Bullseye Upgrades
Closed, ResolvedPublic
Actions

Description

We have 6 kafka-logging hosts needing an upgrade to Bullseye.

  • kafka-logging1001
  • kafka-logging1002
  • kafka-logging1003
  • kafka-logging2001
  • kafka-logging2002
  • kafka-logging2003

We can use the kafka-logging host expansion T326419 to ensure all works as expected on Bullseye, then reimage the above hosts in place as a rolling maintenance.

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedandrea.denisseT324725 Observability Bullseye upgrades
 ResolvedherronT326420 Kafka-logging Bullseye Upgrades

Event Timeline

herron created this task.Jan 6 2023, 3:45 PM
herron triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 6 2023, 3:45 PM
herron mentioned this in T324725: Observability Bullseye upgrades.Jan 6 2023, 3:47 PM
herron mentioned this in T326419: Expand kafka-logging using hosts kafka-logging[12]00[45].
herron added a parent task: T324725: Observability Bullseye upgrades.Jan 6 2023, 3:50 PM
• lmata edited projects, added SRE Observability (FY2022/2023-Q3); removed observability.Jan 13 2023, 4:24 AM
• lmata moved this task from Inbox to Prioritized on the Observability-Logging board.Jan 16 2023, 9:57 PM
ops-monitoring-bot added a comment.Mar 13 2023, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1001 for host kafka-logging2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 13 2023, 9:23 PM

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1001 for host kafka-logging2001.codfw.wmnet with OS bullseye completed:

  • kafka-logging2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303132038_herron_267887_kafka-logging2001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 14 2023, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1001 for host kafka-logging2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 14 2023, 3:59 PM

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1001 for host kafka-logging2003.codfw.wmnet with OS bullseye completed:

  • kafka-logging2003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303141518_herron_533415_kafka-logging2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 14 2023, 6:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1001 for host kafka-logging2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 14 2023, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1001 for host kafka-logging2002.codfw.wmnet with OS bullseye completed:

  • kafka-logging2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303141813_herron_569079_kafka-logging2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
herron updated the task description. (Show Details)Mar 14 2023, 6:52 PM
ops-monitoring-bot added a comment.Mar 15 2023, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1001 for host kafka-logging1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 15 2023, 1:51 PM

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1001 for host kafka-logging1003.eqiad.wmnet with OS bullseye completed:

  • kafka-logging1003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303151312_herron_779773_kafka-logging1003.out
    • Checked BIOS boot parameters are back to normal
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
herron updated the task description. (Show Details)Mar 15 2023, 3:08 PM
ops-monitoring-bot added a comment.Mar 15 2023, 3:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1001 for host kafka-logging1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 15 2023, 4:15 PM

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1001 for host kafka-logging1002.eqiad.wmnet with OS bullseye completed:

  • kafka-logging1002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303151534_herron_809035_kafka-logging1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 15 2023, 7:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1001 for host kafka-logging1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 15 2023, 8:15 PM

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1001 for host kafka-logging1001.eqiad.wmnet with OS bullseye completed:

  • kafka-logging1001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303151932_herron_854627_kafka-logging1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
herron closed this task as Resolved.Mar 15 2023, 9:09 PM
herron claimed this task.
herron updated the task description. (Show Details)
• lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q3) board.May 2 2023, 1:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits