Page MenuHomePhabricator
Create Task
Maniphest T321410

Upgrade logstash to bullseye
Closed, ResolvedPublic
Actions

Description

  • beta-logs
  • eqiad
  • codfw

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
install_server: set codfw logstash vms to install bullseyeoperations/puppetproduction+6 -0
install_server: set eqiad bullseye vms to install bullseyeoperations/puppetproduction+6 -0
Add bullseye support.operations/debs/prometheus-logstash-exportermaster+14 -4
scap: update logstash_host for beta scapoperations/puppetproduction+2 -2
beta-logs: transition jobs host assignment to bullseye hostoperations/puppetproduction+1 -1
beta-logs: allow bullseye logstash host access to lokioperations/puppetproduction+1 -0
beta-logs: add new hostsoperations/puppetproduction+41 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedandrea.denisseT324725 Observability Bullseye upgrades
 ResolvedcolewhiteT321410 Upgrade logstash to bullseye

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2022, 7:35 PM
colewhite moved this task from Inbox to Backlog on the Observability-Logging board.Oct 21 2022, 7:36 PM
colewhite claimed this task.Oct 21 2022, 7:42 PM
colewhite triaged this task as Medium priority.
gerritbot added a comment.Oct 21 2022, 9:02 PM

Change 844563 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: add new hosts

https://gerrit.wikimedia.org/r/844563

gerritbot added a project: Patch-For-Review.Oct 21 2022, 9:02 PM
gerritbot added a comment.Nov 4 2022, 8:00 PM

Change 844563 merged by Cwhite:

[operations/puppet@production] beta-logs: add new hosts

https://gerrit.wikimedia.org/r/844563

colewhite changed the task status from Open to In Progress.Nov 4 2022, 8:01 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 4 2022, 8:30 PM
colewhite updated the task description. (Show Details)Nov 4 2022, 10:45 PM
gerritbot added a comment.Nov 7 2022, 6:55 PM

Change 854106 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: allow bullseye logstash host access to loki

https://gerrit.wikimedia.org/r/854106

gerritbot added a project: Patch-For-Review.Nov 7 2022, 6:55 PM
gerritbot added a comment.Nov 7 2022, 7:02 PM

Change 854106 merged by Cwhite:

[operations/puppet@production] beta-logs: allow bullseye logstash host access to loki

https://gerrit.wikimedia.org/r/854106

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2022, 7:30 PM
gerritbot added a comment.Nov 7 2022, 9:51 PM

Change 854109 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] scap: update logstash_host for beta scap

https://gerrit.wikimedia.org/r/854109

gerritbot added a project: Patch-For-Review.Nov 7 2022, 9:51 PM
gerritbot added a comment.Nov 8 2022, 12:27 AM

Change 854111 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: transition jobs host assignment to bullseye host

https://gerrit.wikimedia.org/r/854111

gerritbot added a comment.Nov 15 2022, 6:19 PM

Change 857049 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/debs/prometheus-logstash-exporter@master] Add bullseye support.

https://gerrit.wikimedia.org/r/857049

gerritbot added a comment.Nov 15 2022, 6:29 PM

Change 854111 merged by Cwhite:

[operations/puppet@production] beta-logs: transition jobs host assignment to bullseye host

https://gerrit.wikimedia.org/r/854111

gerritbot added a comment.Nov 15 2022, 8:16 PM

Change 854109 merged by Cwhite:

[operations/puppet@production] scap: update logstash_host for beta scap

https://gerrit.wikimedia.org/r/854109

gerritbot added a comment.Nov 16 2022, 3:43 PM

Change 857049 merged by Cwhite:

[operations/debs/prometheus-logstash-exporter@master] Add bullseye support.

https://gerrit.wikimedia.org/r/857049

gerritbot added a comment.Nov 30 2022, 12:47 AM

Change 861871 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] install_server: set eqiad bullseye vms to install bullseye

https://gerrit.wikimedia.org/r/861871

gerritbot added a comment.Nov 30 2022, 12:47 AM

Change 861872 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] install_server: set codfw logstash vms to install bullseye

https://gerrit.wikimedia.org/r/861872

gerritbot added a comment.Nov 30 2022, 11:32 PM

Change 861871 merged by Cwhite:

[operations/puppet@production] install_server: set eqiad bullseye vms to install bullseye

https://gerrit.wikimedia.org/r/861871

colewhite updated the task description. (Show Details)Dec 2 2022, 12:07 AM
Stashbot added a comment.Dec 3 2022, 12:17 AM

Mentioned in SAL (#wikimedia-operations) [2022-12-03T00:17:36Z] <cwhite> draining shards from logstash1010, logstash1033, logstash1034, logstash1035 - T321410

ops-monitoring-bot added a comment.Dec 5 2022, 3:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1010.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 5 2022, 4:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1035.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 5 2022, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1010.eqiad.wmnet with OS bullseye completed:

  • logstash1010 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212051544_cwhite_3290570_logstash1010.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 5 2022, 4:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1034.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 5 2022, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1033.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 5 2022, 5:21 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1035.eqiad.wmnet with OS bullseye completed:

  • logstash1035 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212051638_cwhite_3301713_logstash1035.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 5 2022, 5:21 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1034.eqiad.wmnet with OS bullseye completed:

  • logstash1034 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212051641_cwhite_3302006_logstash1034.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 5 2022, 6:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1033.eqiad.wmnet with OS bullseye completed:

  • logstash1033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212051644_cwhite_3302469_logstash1033.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 6 2022, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1029.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 4:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1028.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 4:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1027.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 5:12 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1028.eqiad.wmnet with OS bullseye executed with errors:

  • logstash1028 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2022, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1029.eqiad.wmnet with OS bullseye executed with errors:

  • logstash1029 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2022, 5:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1028.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 5:29 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1027.eqiad.wmnet with OS bullseye executed with errors:

  • logstash1027 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2022, 6:18 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1028.eqiad.wmnet with OS bullseye executed with errors:

  • logstash1028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2022, 6:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1027.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 7:38 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1027.eqiad.wmnet with OS bullseye completed:

  • logstash1027 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212061856_cwhite_3609470_logstash1027.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 6 2022, 7:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1028.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 8:24 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1028.eqiad.wmnet with OS bullseye completed:

  • logstash1028 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212061940_cwhite_3619941_logstash1028.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 6 2022, 8:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1029.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 8:25 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1029.eqiad.wmnet with OS bullseye executed with errors:

  • logstash1029 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2022, 8:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1029.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 9:07 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1029.eqiad.wmnet with OS bullseye completed:

  • logstash1029 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212062025_cwhite_3632715_logstash1029.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 7 2022, 2:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1011.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 7 2022, 3:59 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1011.eqiad.wmnet with OS bullseye completed:

  • logstash1011 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212070249_cwhite_3699833_logstash1011.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 7 2022, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1026.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 7 2022, 5:26 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1026.eqiad.wmnet with OS bullseye executed with errors:

  • logstash1026 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 7 2022, 6:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1026.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 7 2022, 6:49 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1026.eqiad.wmnet with OS bullseye completed:

  • logstash1026 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212071806_cwhite_3867961_logstash1026.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 7 2022, 11:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1012.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 8 2022, 12:45 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1012.eqiad.wmnet with OS bullseye completed:

  • logstash1012 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212072338_cwhite_3929430_logstash1012.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• lmata mentioned this in T324725: Observability Bullseye upgrades.Dec 8 2022, 1:52 AM
colewhite updated the task description. (Show Details)Dec 8 2022, 3:50 PM
ops-monitoring-bot added a comment.Dec 13 2022, 9:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2035.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 13 2022, 9:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2034.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 13 2022, 9:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 13 2022, 10:12 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2034.codfw.wmnet with OS bullseye completed:

  • logstash2034 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212132130_cwhite_1169790_logstash2034.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 13 2022, 10:14 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2033.codfw.wmnet with OS bullseye completed:

  • logstash2033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212132132_cwhite_1170035_logstash2033.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 13 2022, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2035.codfw.wmnet with OS bullseye completed:

  • logstash2035 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212132123_cwhite_1166950_logstash2035.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 12:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2036.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 14 2022, 12:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2037.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 14 2022, 1:18 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2036.codfw.wmnet with OS bullseye completed:

  • logstash2036 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212140045_cwhite_1206345_logstash2036.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 1:22 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2037.codfw.wmnet with OS bullseye completed:

  • logstash2037 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212140046_cwhite_1206501_logstash2037.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2028.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 14 2022, 5:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2029.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 14 2022, 5:34 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2028.codfw.wmnet with OS bullseye completed:

  • logstash2028 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212141648_cwhite_1375376_logstash2028.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 6:03 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2029.codfw.wmnet with OS bullseye completed:

  • logstash2029 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212141718_cwhite_1380916_logstash2029.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 6:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 14 2022, 6:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2027.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 14 2022, 7:08 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2027.codfw.wmnet with OS bullseye completed:

  • logstash2027 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212141823_cwhite_1394473_logstash2027.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 7:30 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2001.codfw.wmnet with OS bullseye completed:

  • logstash2001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212141809_cwhite_1392921_logstash2001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 14 2022, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2026.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 15 2022, 12:14 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2026.codfw.wmnet with OS bullseye executed with errors:

  • logstash2026 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 15 2022, 12:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2026.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 15 2022, 12:19 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2026.codfw.wmnet with OS bullseye executed with errors:

  • logstash2026 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 15 2022, 12:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2026.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 15 2022, 1:45 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2026.codfw.wmnet with OS bullseye completed:

  • logstash2026 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212150019_cwhite_1457754_logstash2026.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Dec 15 2022, 4:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 15 2022, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2002.codfw.wmnet with OS bullseye completed:

  • logstash2002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212151654_cwhite_1630173_logstash2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Dec 15 2022, 8:34 PM

Change 861872 merged by Cwhite:

[operations/puppet@production] install_server: set codfw logstash vms to install bullseye

https://gerrit.wikimedia.org/r/861872

ops-monitoring-bot added a comment.Dec 19 2022, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2003.codfw.wmnet with OS bullseye

colewhite closed this task as Resolved.Dec 19 2022, 10:49 PM
colewhite updated the task description. (Show Details)
ops-monitoring-bot added a comment.Dec 19 2022, 11:05 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2003.codfw.wmnet with OS bullseye completed:

  • logstash2003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212191903_cwhite_2645190_logstash2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
herron added a parent task: T324725: Observability Bullseye upgrades.Jan 6 2023, 3:50 PM
colewhite added a project: SRE Observability (FY2022/2023-Q2).Feb 9 2023, 12:22 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 9 2023, 12:31 AM
• lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q2) board.Jul 18 2023, 5:01 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits