Page MenuHomePhabricator
Create Task
Maniphest T364289

Reimage external store hosts with Bookworm
Closed, ResolvedPublic
Actions

Description

They were all migrated to 10.6, but still running Bullseye, they need to be migrated to bookworm

es4

  • es2020 master
  • es2021
  • es2022
  • es1020
  • es1021 master
  • es1022

es5

  • es2023 master
  • es2024
  • es2025
  • es1023
  • es1024 master
  • es1025

Details

SubjectRepoBranchLines +/-
site.pp: Reorganize es5 hostsoperations/puppetproduction+8 -1
es1024: Remove package declarationoperations/puppetproduction+0 -1
es1024: Disable notificationsoperations/puppetproduction+1 -0
es1021: Remove package declarationoperations/puppetproduction+0 -1
es1021: Disable notificationsoperations/puppetproduction+1 -0
es1022: Disable notificationsoperations/puppetproduction+1 -0
es2022: Remove package declarationoperations/puppetproduction+0 -1
es2021: Remove package declarationoperations/puppetproduction+0 -1
es1020: Remove package declarationoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Mon, May 6, 6:34 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, May 6, 6:34 AM
Marostegui triaged this task as Medium priority.Mon, May 6, 6:34 AM
Marostegui moved this task from Triage to In progress on the DBA board.
ops-monitoring-bot added a comment.Mon, May 6, 7:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1020.eqiad.wmnet with OS bookworm

Marostegui claimed this task.Mon, May 6, 7:14 AM
Marostegui updated the task description. (Show Details)Mon, May 6, 7:47 AM
ops-monitoring-bot added a comment.Mon, May 6, 7:56 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1020.eqiad.wmnet with OS bookworm completed:

  • es1020 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405060732_marostegui_577385_es1020.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mon, May 6, 8:02 AM

Change #1028365 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1020: Remove package declaration

https://gerrit.wikimedia.org/r/1028365

gerritbot added a project: Patch-For-Review.Mon, May 6, 8:02 AM

Change #1028365 merged by Marostegui:

[operations/puppet@production] es1020: Remove package declaration

https://gerrit.wikimedia.org/r/1028365

Stashbot added a comment.Mon, May 6, 8:04 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-06T08:04:24Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1025 T364289', diff saved to https://phabricator.wikimedia.org/P61890 and previous config saved to /var/cache/conftool/dbconfig/20240506-080423-root.json

ops-monitoring-bot added a comment.Mon, May 6, 8:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1025.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Mon, May 6, 8:06 AM
Maintenance_bot removed a project: Patch-For-Review.Mon, May 6, 8:31 AM
Marostegui updated the task description. (Show Details)Mon, May 6, 8:38 AM
ops-monitoring-bot added a comment.Mon, May 6, 8:48 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1025.eqiad.wmnet with OS bookworm completed:

  • es1025 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405060822_marostegui_585872_es1025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, May 6, 10:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2024.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, May 6, 11:08 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2024.codfw.wmnet with OS bookworm completed:

  • es2024 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405061044_marostegui_641087_es2024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Mon, May 6, 11:36 AM
ops-monitoring-bot added a comment.Mon, May 6, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2021.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Mon, May 6, 2:07 PM
ops-monitoring-bot added a comment.Mon, May 6, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2021.codfw.wmnet with OS bookworm completed:

  • es2021 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405061350_marostegui_830494_es2021.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
gerritbot added a comment.Mon, May 6, 2:28 PM

Change #1028513 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2021: Remove package declaration

https://gerrit.wikimedia.org/r/1028513

gerritbot added a project: Patch-For-Review.Mon, May 6, 2:28 PM

Change #1028513 merged by Marostegui:

[operations/puppet@production] es2021: Remove package declaration

https://gerrit.wikimedia.org/r/1028513

Maintenance_bot removed a project: Patch-For-Review.Mon, May 6, 2:30 PM
ops-monitoring-bot added a comment.Wed, May 8, 5:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2025.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Wed, May 8, 5:46 AM
ops-monitoring-bot added a comment.Wed, May 8, 5:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2022.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 8, 5:53 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2025.codfw.wmnet with OS bookworm completed:

  • es2025 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405080528_marostegui_1246868_es2025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Wed, May 8, 6:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2022.codfw.wmnet with OS bookworm completed:

  • es2022 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405080611_marostegui_1254735_es2022.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 8, 6:35 AM
Marostegui closed subtask T364443: Switchover es5 codfw master (es2023 -> es2024) as Resolved.Wed, May 8, 8:09 AM
ops-monitoring-bot added a comment.Wed, May 8, 8:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2023.codfw.wmnet with OS bookworm

gerritbot added a comment.Wed, May 8, 8:19 AM

Change #1029120 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es2022: Remove package declaration

https://gerrit.wikimedia.org/r/1029120

gerritbot added a project: Patch-For-Review.Wed, May 8, 8:19 AM

Change #1029120 merged by Marostegui:

[operations/puppet@production] es2022: Remove package declaration

https://gerrit.wikimedia.org/r/1029120

Maintenance_bot removed a project: Patch-For-Review.Wed, May 8, 8:30 AM
Marostegui updated the task description. (Show Details)Wed, May 8, 8:52 AM
ops-monitoring-bot added a comment.Wed, May 8, 8:58 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2023.codfw.wmnet with OS bookworm completed:

  • es2023 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405080835_marostegui_1274691_es2023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Wed, May 8, 9:06 AM

Change #1029133 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1022: Disable notifications

https://gerrit.wikimedia.org/r/1029133

gerritbot added a project: Patch-For-Review.Wed, May 8, 9:06 AM
Stashbot added a comment.Wed, May 8, 9:06 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-08T09:06:22Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1022 T364289', diff saved to https://phabricator.wikimedia.org/P62038 and previous config saved to /var/cache/conftool/dbconfig/20240508-090621-root.json

gerritbot added a comment.Wed, May 8, 9:07 AM

Change #1029133 merged by Marostegui:

[operations/puppet@production] es1022: Disable notifications

https://gerrit.wikimedia.org/r/1029133

ops-monitoring-bot added a comment.Wed, May 8, 9:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1022.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Wed, May 8, 9:30 AM
ops-monitoring-bot added a comment.Wed, May 8, 9:49 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1022.eqiad.wmnet with OS bookworm completed:

  • es1022 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405080925_marostegui_1284070_es1022.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 8, 9:54 AM
Marostegui closed subtask T364451: Switchover es4 codfw master (es2020 -> es2021) as Resolved.Thu, May 9, 6:22 AM
ops-monitoring-bot added a comment.Thu, May 9, 6:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es2020.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 9, 7:14 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es2020.codfw.wmnet with OS bookworm completed:

  • es2020 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405090650_marostegui_1672825_es2020.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Thu, May 9, 7:31 AM
Marostegui closed subtask T364816: Switchover es4 master (es1021 -> es1020) as Resolved.Thu, May 16, 6:05 AM
Marostegui closed subtask T365094: Switchover es5 master (es1024 -> es1025) as Resolved.Thu, May 16, 7:44 AM
Stashbot added a comment.Thu, May 16, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T07:48:37Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1021 T364289', diff saved to https://phabricator.wikimedia.org/P62454 and previous config saved to /var/cache/conftool/dbconfig/20240516-074837-root.json

gerritbot added a comment.Thu, May 16, 7:49 AM

Change #1032385 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1021: Disable notifications

https://gerrit.wikimedia.org/r/1032385

gerritbot added a project: Patch-For-Review.Thu, May 16, 7:49 AM

Change #1032385 merged by Marostegui:

[operations/puppet@production] es1021: Disable notifications

https://gerrit.wikimedia.org/r/1032385

ops-monitoring-bot added a comment.Thu, May 16, 7:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1021.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 16, 8:26 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1021.eqiad.wmnet with OS bookworm completed:

  • es1021 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405160808_marostegui_361726_es1021.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Thu, May 16, 8:31 AM
Marostegui updated the task description. (Show Details)Thu, May 16, 8:47 AM
gerritbot added a comment.Thu, May 16, 9:33 AM

Change #1032403 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1021: Remove package declaration

https://gerrit.wikimedia.org/r/1032403

gerritbot added a project: Patch-For-Review.Thu, May 16, 9:33 AM

Change #1032403 merged by Marostegui:

[operations/puppet@production] es1021: Remove package declaration

https://gerrit.wikimedia.org/r/1032403

Maintenance_bot removed a project: Patch-For-Review.Thu, May 16, 10:31 AM
gerritbot added a comment.Thu, May 16, 1:11 PM

Change #1032479 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1024: Disable notifications

https://gerrit.wikimedia.org/r/1032479

gerritbot added a project: Patch-For-Review.Thu, May 16, 1:11 PM
Stashbot added a comment.Thu, May 16, 1:11 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T13:11:12Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool es1024 T364289', diff saved to https://phabricator.wikimedia.org/P62501 and previous config saved to /var/cache/conftool/dbconfig/20240516-131111-root.json

gerritbot added a comment.Thu, May 16, 1:11 PM

Change #1032479 merged by Marostegui:

[operations/puppet@production] es1024: Disable notifications

https://gerrit.wikimedia.org/r/1032479

ops-monitoring-bot added a comment.Thu, May 16, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host es1024.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Thu, May 16, 1:31 PM
ops-monitoring-bot added a comment.Thu, May 16, 1:48 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host es1024.eqiad.wmnet with OS bookworm completed:

  • es1024 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405161331_marostegui_407297_es1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Fri, May 17, 4:43 AM

Change #1032620 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1024: Remove package declaration

https://gerrit.wikimedia.org/r/1032620

gerritbot added a project: Patch-For-Review.Fri, May 17, 4:43 AM

Change #1032620 merged by Marostegui:

[operations/puppet@production] es1024: Remove package declaration

https://gerrit.wikimedia.org/r/1032620

gerritbot added a comment.Fri, May 17, 5:10 AM

Change #1032622 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Reorganize es5 hosts

https://gerrit.wikimedia.org/r/1032622

gerritbot added a comment.Fri, May 17, 5:11 AM

Change #1032622 merged by Marostegui:

[operations/puppet@production] site.pp: Reorganize es5 hosts

https://gerrit.wikimedia.org/r/1032622

Marostegui updated the task description. (Show Details)Fri, May 17, 5:17 AM
Maintenance_bot removed a project: Patch-For-Review.Fri, May 17, 5:31 AM
Marostegui closed this task as Resolved.Fri, May 17, 6:12 AM

All done

Maintenance_bot moved this task from In progress to Done on the DBA board.Fri, May 17, 6:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL