Page MenuHomePhabricator
Create Task
Maniphest T353912

Observability Bookworm upgrades
Open, Needs TriagePublic
Actions

Description

Tracking task for o11y Bookworm upgrades

  • Alert (2 hosts) T333615
  • Grafana (2 hosts) T352665
  • Centrallog (2 hosts) bookworm host at phi-syslog-01.o11y.eqiad1.wikimedia.cloud
  • Logstash (30 hosts)
  • Prometheus (4 hosts) bookworm host at phi-prometheus-01.o11y.eqiad1.wikimedia.cloud
  • T397756: Kafka-logging -> Bookworm Kafka-logging (6 hosts) bullseye host at phi-kafka-01.o11y.eqiad1.wikimedia.cloud
  • T397757: Kafkamon -> Bookworm Kafkamon (2 hosts) bookworm host at phi-kafkamon-01.o11y.eqiad1.wikimedia.cloud
  • mwlog (2 hosts). bookworm host at phi-mwlog-01.o11y.eqiad1.wikimedia.cloud
  • webperf (2 hosts). bookworm host at phi-arclamp-01.o11y.eqiad1.wikimedia.cloud — Ref T305460
  • arclamp (2 hosts). bookworm host at phi-webperf-01.o11y.eqiad1.wikimedia.cloud — Ref T305460

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Apr 3 2024, 12:25 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logging-hd2003.codfw.wmnet with OS bookworm completed:

  • logging-hd2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030003_cwhite_4011899_logging-hd2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 3 2024, 12:43 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logging-hd2002.codfw.wmnet with OS bookworm executed with errors:

  • logging-hd2002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" logging-hd2002.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Apr 3 2024, 12:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logging-hd2002.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Apr 3 2024, 1:40 AM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logging-hd2002.codfw.wmnet with OS bookworm completed:

  • logging-hd2002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404030119_cwhite_4083457_logging-hd2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
andrea.denisse updated the task description. (Show Details)Apr 3 2024, 3:35 AM
colewhite edited projects, added SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).Apr 3 2024, 11:08 PM
colewhite moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q4) board.
gerritbot added a comment.Jun 7 2024, 7:16 AM

Change #1039974 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] webperf: don't hardcode php version

https://gerrit.wikimedia.org/r/1039974

fgiunchedi updated the task description. (Show Details)Jun 7 2024, 7:16 AM
colewhite edited projects, added SRE Observability (FY2024/2025-Q1); removed SRE Observability (FY2023/2024-Q4).Jul 2 2024, 5:58 PM
fgiunchedi updated the task description. (Show Details)Jul 8 2024, 12:06 PM
fgiunchedi updated the task description. (Show Details)Jul 30 2024, 8:12 AM
fgiunchedi updated the task description. (Show Details)Jul 30 2024, 8:32 AM
fgiunchedi updated the task description. (Show Details)Aug 28 2024, 10:40 AM
fgiunchedi added subscribers: andrea.denisse, fgiunchedi.

The prometheus bookworm upgrade has been completed as part of T326657: Add prometheus-https load balancer, thank you @andrea.denisse for your help with this

fgiunchedi mentioned this in T374842: Retire anycast_healthchecker Icinga check.Sep 17 2024, 9:28 AM
Stashbot added a comment.Sep 18 2024, 8:41 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-18T08:41:40Z] <tappof> centrallog2002 upgrade to bookworm in progress https://phabricator.wikimedia.org/T353912

ops-monitoring-bot added a comment.Sep 18 2024, 9:11 AM

Host rebooted by tappof@cumin2002 with reason: upgrade to bookworm

Stashbot added a comment.Sep 24 2024, 5:55 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-24T05:55:18Z] <tappof> centrallog1002 upgrade to bookworm in progress https://phabricator.wikimedia.org/T353912

ops-monitoring-bot added a comment.Sep 24 2024, 6:35 AM

Host rebooted by tappof@cumin2002 with reason: upgrade to bookworm

tappof updated the task description. (Show Details)Sep 24 2024, 8:47 AM
tappof closed subtask T375085: mtail 3.0.0~rc50-1+b6 leaks memory on centrallog2002 as Resolved.Sep 24 2024, 9:30 AM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q1) board.Oct 31 2024, 9:24 PM
• lmata edited projects, added SRE Observability (FY2024/2025-Q2); removed SRE Observability (FY2024/2025-Q1).
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q2) board.Nov 5 2024, 5:06 PM
gerritbot added a comment.Jan 6 2025, 10:59 PM

Change #1108502 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add jvm options for openjdk-17 support

https://gerrit.wikimedia.org/r/1108502

gerritbot added a comment.Jan 6 2025, 11:07 PM

Change #1108502 merged by Cwhite:

[operations/puppet@production] logstash: add jvm options for openjdk-17 support

https://gerrit.wikimedia.org/r/1108502

gerritbot added a comment.Jan 9 2025, 12:18 AM

Change #1109188 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: update codfw jobs host to logging-sd2001

https://gerrit.wikimedia.org/r/1109188

gerritbot added a comment.Jan 9 2025, 12:19 AM

Change #1109189 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: update eqiad jobs host to logging-sd1001

https://gerrit.wikimedia.org/r/1109189

• lmata edited projects, added SRE Observability (FY2024/2025-Q3); removed SRE Observability (FY2024/2025-Q2).Jan 14 2025, 3:56 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q3) board.
gerritbot added a comment.Jan 14 2025, 6:05 PM

Change #1109188 merged by Cwhite:

[operations/puppet@production] logstash: update codfw jobs host to logging-sd2001

https://gerrit.wikimedia.org/r/1109188

fgiunchedi added a project: observability.Jan 30 2025, 2:29 PM
fgiunchedi moved this task from Inbox to Radar on the observability board.Jan 30 2025, 2:45 PM
gerritbot added a comment.Feb 24 2025, 10:40 PM

Change #1109189 merged by Cwhite:

[operations/puppet@production] logstash: update eqiad jobs host to logging-sd1001

https://gerrit.wikimedia.org/r/1109189

• lmata edited projects, added SRE Observability (FY2024/2025-Q4); removed SRE Observability (FY2024/2025-Q3).Apr 1 2025, 12:04 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q4) board.
Aklapper changed the task status from In Progress to Open.Apr 11 2025, 10:19 PM

Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than one year (see T380300). Feel free to set that status again, or rather break down into smaller subtasks.

herron updated the task description. (Show Details)Jun 24 2025, 6:49 PM
• lmata edited projects, added SRE Observability (FY2025/2026-Q1); removed SRE Observability (FY2024/2025-Q4).Jun 30 2025, 4:04 PM
• lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2025/2026-Q1) board.
gerritbot added a comment.Jul 21 2025, 8:30 PM

Change #1171272 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: provision logging-logstash-04

https://gerrit.wikimedia.org/r/1171272

gerritbot added a comment.Jul 21 2025, 8:34 PM

Change #1171272 merged by Cwhite:

[operations/puppet@production] beta-logs: provision logging-logstash-04

https://gerrit.wikimedia.org/r/1171272

ops-monitoring-bot added a comment.Jul 22 2025, 5:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2023.codfw.wmnet with OS bookworm

gerritbot added a comment.Jul 22 2025, 9:28 PM

Change #1171713 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] opensearch: curator instance config to follow $enable_curator

https://gerrit.wikimedia.org/r/1171713

ops-monitoring-bot added a comment.Jul 22 2025, 9:41 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2023.codfw.wmnet with OS bookworm completed:

  • logstash2023 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507221727_cwhite_390479_logstash2023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 22 2025, 9:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2024.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 22 2025, 10:47 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2024.codfw.wmnet with OS bookworm completed:

  • logstash2024 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507222212_cwhite_530206_logstash2024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 22 2025, 10:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2025.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 22 2025, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2025.codfw.wmnet with OS bookworm completed:

  • logstash2025 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507222314_cwhite_561046_logstash2025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 23 2025, 1:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2030.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 23 2025, 2:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2030.codfw.wmnet with OS bookworm completed:

  • logstash2030 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507231403_cwhite_993421_logstash2030.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 23 2025, 2:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2031.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 23 2025, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2031.codfw.wmnet with OS bookworm completed:

  • logstash2031 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507231503_cwhite_1024154_logstash2031.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 23 2025, 4:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2032.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 23 2025, 6:18 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2032.codfw.wmnet with OS bookworm completed:

  • logstash2032 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507231658_cwhite_1080985_logstash2032.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 23 2025, 7:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1023.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 23 2025, 8:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1023.eqiad.wmnet with OS bookworm completed:

  • logstash1023 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507232002_cwhite_1174671_logstash1023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 24 2025, 4:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1024.eqiad.wmnet with OS bookworm

gerritbot added a comment.Jul 24 2025, 5:26 PM

Change #1171713 merged by Cwhite:

[operations/puppet@production] opensearch: curator instance config to follow $enable_curator

https://gerrit.wikimedia.org/r/1171713

ops-monitoring-bot added a comment.Jul 24 2025, 5:34 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1024.eqiad.wmnet with OS bookworm completed:

  • logstash1024 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507241648_cwhite_1778704_logstash1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 24 2025, 5:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1025.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 6:29 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1025.eqiad.wmnet with OS bookworm completed:

  • logstash1025 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507241758_cwhite_1815978_logstash1025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 24 2025, 6:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1030.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 7:45 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1030.eqiad.wmnet with OS bookworm completed:

  • logstash1030 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507241908_cwhite_1849953_logstash1030.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 24 2025, 8:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1031.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1031.eqiad.wmnet with OS bookworm completed:

  • logstash1031 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507242023_cwhite_1890087_logstash1031.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 24 2025, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1032.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 24 2025, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1032.eqiad.wmnet with OS bookworm completed:

  • logstash1032 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507242123_cwhite_1920057_logstash1032.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 25 2025, 3:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2037.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 25 2025, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2037.codfw.wmnet with OS bookworm completed:

  • logstash2037 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507251603_cwhite_2455208_logstash2037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 25 2025, 9:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2036.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 25 2025, 10:07 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2036.codfw.wmnet with OS bookworm completed:

  • logstash2036 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507252144_cwhite_2623349_logstash2036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 29 2025, 10:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2035.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 29 2025, 11:10 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2035.codfw.wmnet with OS bookworm completed:

  • logstash2035 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507292248_cwhite_1248291_logstash2035.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 30 2025, 2:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2034.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 30 2025, 3:39 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2034.codfw.wmnet with OS bookworm completed:

  • logstash2034 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507301514_cwhite_1730735_logstash2034.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 30 2025, 5:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash2033.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 30 2025, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash2033.codfw.wmnet with OS bookworm completed:

  • logstash2033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507301823_cwhite_1825672_logstash2033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 30 2025, 7:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1037.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 30 2025, 8:23 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1037.eqiad.wmnet with OS bookworm completed:

  • logstash1037 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507302003_cwhite_1882817_logstash1037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 30 2025, 10:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1036.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 30 2025, 11:02 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1036.eqiad.wmnet with OS bookworm completed:

  • logstash1036 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507302244_cwhite_1963622_logstash1036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 31 2025, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1033.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 31 2025, 3:27 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1033.eqiad.wmnet with OS bookworm completed:

  • logstash1033 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507311508_cwhite_2444254_logstash1033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 31 2025, 4:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1034.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 31 2025, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1034.eqiad.wmnet with OS bookworm completed:

  • logstash1034 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507311714_cwhite_2508313_logstash1034.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jul 31 2025, 7:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwhite@cumin2002 for host logstash1035.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jul 31 2025, 8:07 PM

Cookbook cookbooks.sre.hosts.reimage started by cwhite@cumin2002 for host logstash1035.eqiad.wmnet with OS bookworm completed:

  • logstash1035 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202507311948_cwhite_2584295_logstash1035.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
colewhite updated the task description. (Show Details)Jul 31 2025, 8:09 PM
colewhite mentioned this in T324865: Upgrade prometheus-elasticsearch-exporter to 1.4.0 or greater..Aug 1 2025, 9:28 PM
Krinkle updated the task description. (Show Details)Sep 29 2025, 4:58 PM
Krinkle updated the task description. (Show Details)
andrea.denisse updated the task description. (Show Details)Oct 1 2025, 2:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits