Page MenuHomePhabricator
Create Task
Maniphest T351864

Migrate dbproxy hosts to Bookworm
Closed, ResolvedPublic
Actions

Description

We need to test first if they work as expected.
Probably start with codfw hosts as they are ununsed.

  • dbproxy1020
  • dbproxy1021
  • dbproxy1022
  • dbproxy1023
  • dbproxy1024
  • dbproxy1025
  • dbproxy1026
  • dbproxy1027
  • dbproxy2001
  • dbproxy2002
  • dbproxy2003
  • dbproxy2004

Note: dbproxy1018 and dbproxy1019 are owned by WMCS so they should be tracked in a different track

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui updated the task description. (Show Details)Nov 24 2023, 6:06 AM
ops-monitoring-bot added a comment.Nov 24 2023, 6:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy2003.codfw.wmnet with OS bookworm

Marostegui moved this task from Ready to In progress on the DBA board.Nov 24 2023, 6:22 AM
Marostegui updated the task description. (Show Details)Nov 24 2023, 6:54 AM
ops-monitoring-bot added a comment.Nov 24 2023, 6:55 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy2003.codfw.wmnet with OS bookworm completed:

  • dbproxy2003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311240633_marostegui_2534377_dbproxy2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 24 2023, 6:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy2002.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Nov 24 2023, 7:29 AM
ops-monitoring-bot added a comment.Nov 24 2023, 7:32 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Nov 24 2023, 7:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy2002.codfw.wmnet with OS bookworm completed:

  • dbproxy2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311240715_marostegui_2555096_dbproxy2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Nov 24 2023, 8:04 AM
ops-monitoring-bot added a comment.Nov 24 2023, 8:11 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy2001.codfw.wmnet with OS bookworm completed:

  • dbproxy2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311240750_marostegui_2572256_dbproxy2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Nov 28 2023, 8:36 AM

Change 977993 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1024: Disable notifications

https://gerrit.wikimedia.org/r/977993

gerritbot added a project: Patch-For-Review.Nov 28 2023, 8:36 AM

Change 977993 merged by Marostegui:

[operations/puppet@production] dbproxy1024: Disable notifications

https://gerrit.wikimedia.org/r/977993

ops-monitoring-bot added a comment.Nov 28 2023, 8:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1024.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Nov 28 2023, 9:08 AM
Maintenance_bot removed a project: Patch-For-Review.Nov 28 2023, 9:10 AM
ops-monitoring-bot added a comment.Nov 28 2023, 9:13 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1024.eqiad.wmnet with OS bookworm completed:

  • dbproxy1024 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311280855_marostegui_843008_dbproxy1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Nov 29 2023, 8:43 AM

Change 978468 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Failover m1-master

https://gerrit.wikimedia.org/r/978468

gerritbot added a project: Patch-For-Review.Nov 29 2023, 8:43 AM
gerritbot added a comment.Nov 29 2023, 8:53 AM

Change 978468 merged by Marostegui:

[operations/dns@master] wmnet: Failover m1-master

https://gerrit.wikimedia.org/r/978468

Stashbot added a comment.Nov 29 2023, 8:54 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-29T08:54:00Z] <marostegui> Failover m1-master from dbproxy1022 to dbproxy1024 T351864

Marostegui added a comment.Nov 29 2023, 8:55 AM

m1-master has been failed over to dbproxy1024, which is running bookworm. I am going to let connections move on their own rather than forcing them

gerritbot added a comment.Nov 29 2023, 9:03 AM

Change 978471 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1025: Disable notifications

https://gerrit.wikimedia.org/r/978471

ops-monitoring-bot added a comment.Nov 29 2023, 9:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1025.eqiad.wmnet with OS bookworm

gerritbot added a comment.Nov 29 2023, 9:04 AM

Change 978471 merged by Marostegui:

[operations/puppet@production] dbproxy1025: Disable notifications

https://gerrit.wikimedia.org/r/978471

Maintenance_bot removed a project: Patch-For-Review.Nov 29 2023, 9:11 AM
ops-monitoring-bot added a comment.Nov 29 2023, 9:42 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1025.eqiad.wmnet with OS bookworm completed:

  • dbproxy1025 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290921_marostegui_1512579_dbproxy1025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Nov 29 2023, 10:33 AM
gerritbot added a comment.Nov 30 2023, 8:43 AM

Change 979021 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1025: Disable notifications

https://gerrit.wikimedia.org/r/979021

gerritbot added a project: Patch-For-Review.Nov 30 2023, 8:43 AM

Change 979021 merged by Marostegui:

[operations/puppet@production] dbproxy1025: Disable notifications

https://gerrit.wikimedia.org/r/979021

ops-monitoring-bot added a comment.Nov 30 2023, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1025.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Nov 30 2023, 9:11 AM
ops-monitoring-bot added a comment.Nov 30 2023, 9:25 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1025.eqiad.wmnet with OS bookworm completed:

  • dbproxy1025 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311300902_marostegui_2148326_dbproxy1025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Dec 1 2023, 7:12 AM

Change 979199 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1026: Disable notifications

https://gerrit.wikimedia.org/r/979199

gerritbot added a project: Patch-For-Review.Dec 1 2023, 7:12 AM
ops-monitoring-bot added a comment.Dec 1 2023, 7:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1026.eqiad.wmnet with OS bookworm

gerritbot added a comment.Dec 1 2023, 7:12 AM

Change 979199 merged by Marostegui:

[operations/puppet@production] dbproxy1026: Disable notifications

https://gerrit.wikimedia.org/r/979199

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2023, 7:30 AM
Marostegui updated the task description. (Show Details)Dec 1 2023, 7:43 AM
ops-monitoring-bot added a comment.Dec 1 2023, 7:47 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1026.eqiad.wmnet with OS bookworm completed:

  • dbproxy1026 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312010729_marostegui_2769990_dbproxy1026.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
gerritbot added a comment.Dec 4 2023, 7:54 AM

Change 979676 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1022: Disable notifications

https://gerrit.wikimedia.org/r/979676

gerritbot added a project: Patch-For-Review.Dec 4 2023, 7:54 AM
ops-monitoring-bot added a comment.Dec 4 2023, 7:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1022.eqiad.wmnet with OS bookworm

gerritbot added a comment.Dec 4 2023, 7:54 AM

Change 979676 merged by Marostegui:

[operations/puppet@production] dbproxy1022: Disable notifications

https://gerrit.wikimedia.org/r/979676

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2023, 8:10 AM
ops-monitoring-bot added a comment.Dec 4 2023, 8:10 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1022.eqiad.wmnet with OS bookworm executed with errors:

  • dbproxy1022 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 4 2023, 8:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 4 2023, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1022.eqiad.wmnet with OS bookworm executed with errors:

  • dbproxy1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 4 2023, 8:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1022.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 4 2023, 10:17 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1022.eqiad.wmnet with OS bookworm completed:

  • dbproxy1022 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312041002_marostegui_444718_dbproxy1022.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Dec 4 2023, 10:21 AM
Marostegui added a comment.Dec 4 2023, 10:44 AM

dbproxy1022 has had issues during the reimage (eternal PXE reboot), but it has been fixed by Riccardo via sudo cookbook sre.hosts.provision --no-dhcp --no-users --no-switch dbproxy1022

gerritbot added a comment.Dec 4 2023, 11:38 AM

Change 979917 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1027: Disable notifications

https://gerrit.wikimedia.org/r/979917

gerritbot added a project: Patch-For-Review.Dec 4 2023, 11:38 AM

Change 979917 merged by Marostegui:

[operations/puppet@production] dbproxy1027: Disable notifications

https://gerrit.wikimedia.org/r/979917

ops-monitoring-bot added a comment.Dec 4 2023, 11:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1027.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2023, 12:10 PM
Marostegui updated the task description. (Show Details)Dec 4 2023, 12:12 PM
ops-monitoring-bot added a comment.Dec 4 2023, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1027.eqiad.wmnet with OS bookworm completed:

  • dbproxy1027 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312041156_marostegui_519861_dbproxy1027.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
gerritbot added a comment.Dec 5 2023, 6:55 AM

Change 980087 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Failover m2-master

https://gerrit.wikimedia.org/r/980087

gerritbot added a project: Patch-For-Review.Dec 5 2023, 6:55 AM
gerritbot added a comment.Dec 5 2023, 8:26 AM

Change 980087 merged by Marostegui:

[operations/dns@master] wmnet: Failover m2-master

https://gerrit.wikimedia.org/r/980087

Stashbot added a comment.Dec 5 2023, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-05T08:26:29Z] <marostegui> Failover m2-master dbproxy1023.eqiad.wmnet -> dbproxy1025.eqiad.wmnet T351864

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2023, 8:30 AM
gerritbot added a comment.Dec 5 2023, 10:53 AM

Change 980350 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1023: Disable notifications

https://gerrit.wikimedia.org/r/980350

gerritbot added a project: Patch-For-Review.Dec 5 2023, 10:53 AM

Change 980350 merged by Marostegui:

[operations/puppet@production] dbproxy1023: Disable notifications

https://gerrit.wikimedia.org/r/980350

ops-monitoring-bot added a comment.Dec 5 2023, 10:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1023.eqiad.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2023, 11:10 AM
ops-monitoring-bot added a comment.Dec 5 2023, 11:30 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1023.eqiad.wmnet with OS bookworm completed:

  • dbproxy1023 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312051112_marostegui_1125298_dbproxy1023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Dec 5 2023, 11:30 AM
gerritbot added a comment.Dec 7 2023, 6:35 AM

Change 980978 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Failover m5-master to dbproxy1027

https://gerrit.wikimedia.org/r/980978

gerritbot added a project: Patch-For-Review.Dec 7 2023, 6:35 AM
Stashbot added a comment.Dec 7 2023, 6:35 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-07T06:35:55Z] <marostegui> Failover m5-master from dbproxy1021 to dbproxy1027 T351864

gerritbot added a comment.Dec 7 2023, 6:36 AM

Change 980978 merged by Marostegui:

[operations/dns@master] wmnet: Failover m5-master to dbproxy1027

https://gerrit.wikimedia.org/r/980978

Maintenance_bot removed a project: Patch-For-Review.Dec 7 2023, 7:10 AM
gerritbot added a comment.Dec 11 2023, 6:59 AM

Change 981716 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Failover m3-master

https://gerrit.wikimedia.org/r/981716

gerritbot added a project: Patch-For-Review.Dec 11 2023, 6:59 AM
gerritbot added a comment.Dec 11 2023, 7:12 AM

Change 981716 merged by Marostegui:

[operations/dns@master] wmnet: Failover m3-master

https://gerrit.wikimedia.org/r/981716

Stashbot added a comment.Dec 11 2023, 7:12 AM

Mentioned in SAL (#wikimedia-operations) [2023-12-11T07:12:50Z] <marostegui> Failvoer m3-master from dbproxy1020 to dbproxy1026 T351864

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2023, 7:30 AM
gerritbot added a comment.Dec 13 2023, 5:53 AM

Change 982511 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1021: Disable notifications

https://gerrit.wikimedia.org/r/982511

gerritbot added a project: Patch-For-Review.Dec 13 2023, 5:53 AM
ops-monitoring-bot added a comment.Dec 13 2023, 5:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1021.eqiad.wmnet with OS bookworm

gerritbot added a comment.Dec 13 2023, 5:54 AM

Change 982511 merged by Marostegui:

[operations/puppet@production] dbproxy1021: Disable notifications

https://gerrit.wikimedia.org/r/982511

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2023, 6:10 AM
Marostegui updated the task description. (Show Details)Dec 13 2023, 6:28 AM
gerritbot added a comment.Dec 13 2023, 6:29 AM

Change 982516 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1020: Disable notifications

https://gerrit.wikimedia.org/r/982516

gerritbot added a project: Patch-For-Review.Dec 13 2023, 6:29 AM
ops-monitoring-bot added a comment.Dec 13 2023, 6:29 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1021.eqiad.wmnet with OS bookworm completed:

  • dbproxy1021 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312130610_marostegui_2075111_dbproxy1021.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Dec 13 2023, 6:30 AM

Change 982516 merged by Marostegui:

[operations/puppet@production] dbproxy1020: Disable notifications

https://gerrit.wikimedia.org/r/982516

Maintenance_bot removed a project: Patch-For-Review.Dec 13 2023, 7:10 AM
ops-monitoring-bot added a comment.Dec 13 2023, 7:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host dbproxy1020.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Dec 13 2023, 8:11 AM
ops-monitoring-bot added a comment.Dec 13 2023, 8:15 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host dbproxy1020.eqiad.wmnet with OS bookworm completed:

  • dbproxy1020 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312130756_marostegui_2121454_dbproxy1020.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui closed this task as Resolved.Dec 13 2023, 8:16 AM

This is all done

Maintenance_bot moved this task from In progress to Done on the DBA board.Dec 13 2023, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits