Page MenuHomePhabricator
Create Task
Maniphest T300600

Upgrade s3 to Bullseye
Closed, ResolvedPublic
Actions

Description

  • dbstore1007 (T299481)
  • db2149
  • db2139 (backup T299876)
  • db2127
  • db2109
  • db2105 (master)
  • db2094 (sanitarium host)
  • db2074 (sanitarium master)
  • db1179
  • db1175
  • db1166
  • db1157 (master)
  • db1154 (sanitarium host)
  • db1145 (backup T299876)
  • db1123
  • db1112 (sanitarium master)
  • db1102 (backup T299876)
  • clouddb1021 (T299480)
  • clouddb1017 (T299480)
  • clouddb1013 (T299480)

Details

SubjectRepoBranchLines +/-
db1112: Disable notificationsoperations/puppetproduction+1 -0
db1175: Disable notificationsoperations/puppetproduction+1 -0
db1179: Disable notificationsoperations/puppetproduction+1 -0
db1166: Disable notificationsoperations/puppetproduction+1 -0
db1123: Disable notificationsoperations/puppetproduction+1 -0
db2105: Disable notificationsoperations/puppetproduction+1 -0
s3 codfw db*: Disable notificationsoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Feb 1 2022, 8:03 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 1 2022, 8:03 AM
Marostegui triaged this task as Medium priority.Feb 1 2022, 8:03 AM
Marostegui moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Feb 1 2022, 8:04 AM

Change 758781 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] s3 codfw db*: Disable notifications

https://gerrit.wikimedia.org/r/758781

gerritbot added a project: Patch-For-Review.Feb 1 2022, 8:04 AM

Change 758781 merged by Marostegui:

[operations/puppet@production] s3 codfw db*: Disable notifications

https://gerrit.wikimedia.org/r/758781

ops-monitoring-bot added a comment.Feb 1 2022, 8:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2074.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 8:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2109.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Feb 1 2022, 8:10 AM
Marostegui updated the task description. (Show Details)Feb 1 2022, 8:37 AM
ops-monitoring-bot added a comment.Feb 1 2022, 8:39 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2109.codfw.wmnet with OS bullseye completed:

  • db2109 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010807_marostegui_15345_db2109.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 1 2022, 8:40 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2074.codfw.wmnet with OS bullseye completed:

  • db2074 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010806_marostegui_15232_db2074.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 1 2022, 8:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2149.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 8:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2127.codfw.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Feb 1 2022, 9:11 AM
ops-monitoring-bot added a comment.Feb 1 2022, 9:16 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2149.codfw.wmnet with OS bullseye completed:

  • db2149 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010842_marostegui_24228_db2149.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Feb 1 2022, 9:20 AM
ops-monitoring-bot added a comment.Feb 1 2022, 9:21 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2127.codfw.wmnet with OS bullseye completed:

  • db2127 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010846_marostegui_24588_db2127.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Feb 1 2022, 10:20 AM

Change 758797 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2105: Disable notifications

https://gerrit.wikimedia.org/r/758797

gerritbot added a project: Patch-For-Review.Feb 1 2022, 10:20 AM

Change 758797 merged by Marostegui:

[operations/puppet@production] db2105: Disable notifications

https://gerrit.wikimedia.org/r/758797

ops-monitoring-bot added a comment.Feb 1 2022, 10:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2105.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2105.codfw.wmnet with OS bullseye completed:

  • db2105 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202011024_marostegui_14888_db2105.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Feb 1 2022, 10:57 AM

codfw core hosts are now upgraded.

Maintenance_bot removed a project: Patch-For-Review.Feb 1 2022, 11:10 AM
gerritbot added a comment.Mar 9 2022, 6:38 AM

Change 769278 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1123: Disable notifications

https://gerrit.wikimedia.org/r/769278

gerritbot added a project: Patch-For-Review.Mar 9 2022, 6:38 AM

Change 769278 merged by Marostegui:

[operations/puppet@production] db1123: Disable notifications

https://gerrit.wikimedia.org/r/769278

ops-monitoring-bot added a comment.Mar 9 2022, 6:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1123.eqiad.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Mar 9 2022, 7:07 AM
ops-monitoring-bot added a comment.Mar 9 2022, 7:09 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1123.eqiad.wmnet with OS bullseye completed:

  • db1123 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203090643_marostegui_2894_db1123.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Mar 9 2022, 7:10 AM
Stashbot added a comment.Mar 9 2022, 7:33 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-09T07:33:39Z] <marostegui> dbmaint on db1123 s3@eqiad T300600

Marostegui added a subtask: T301850: Switchover s3 master (db1157 -> db1123).Mar 11 2022, 6:20 AM
gerritbot added a comment.Mar 15 2022, 6:25 AM

Change 770831 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1166: Disable notifications

https://gerrit.wikimedia.org/r/770831

gerritbot added a project: Patch-For-Review.Mar 15 2022, 6:25 AM
Stashbot added a comment.Mar 15 2022, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-15T06:26:12Z] <marostegui> dbmaint on s3@eqiad T300600

gerritbot added a comment.Mar 15 2022, 6:26 AM

Change 770831 merged by Marostegui:

[operations/puppet@production] db1166: Disable notifications

https://gerrit.wikimedia.org/r/770831

ops-monitoring-bot added a comment.Mar 15 2022, 6:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1166.eqiad.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Mar 15 2022, 6:52 AM
ops-monitoring-bot added a comment.Mar 15 2022, 6:57 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1166.eqiad.wmnet with OS bullseye completed:

  • db1166 (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203150628_marostegui_1166092_db1166.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 18 2022, 5:38 AM

Change 771765 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1179: Disable notifications

https://gerrit.wikimedia.org/r/771765

Stashbot added a comment.Mar 18 2022, 5:38 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-18T05:38:32Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1179 reimage T300600', diff saved to https://phabricator.wikimedia.org/P22803 and previous config saved to /var/cache/conftool/dbconfig/20220318-053832-marostegui.json

Stashbot added a comment.Mar 18 2022, 5:39 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-18T05:39:14Z] <marostegui> dbmaint on s3@eqiad T300600

gerritbot added a comment.Mar 18 2022, 5:39 AM

Change 771765 merged by Marostegui:

[operations/puppet@production] db1179: Disable notifications

https://gerrit.wikimedia.org/r/771765

ops-monitoring-bot added a comment.Mar 18 2022, 5:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1179.eqiad.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)Mar 18 2022, 6:09 AM
ops-monitoring-bot added a comment.Mar 18 2022, 6:12 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1179.eqiad.wmnet with OS bullseye completed:

  • db1179 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203180542_marostegui_1863100_db1179.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 21 2022, 5:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-21T05:52:03Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1175 reimage T300600', diff saved to https://phabricator.wikimedia.org/P22860 and previous config saved to /var/cache/conftool/dbconfig/20220321-055202-marostegui.json

Stashbot added a comment.Mar 21 2022, 5:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-21T05:52:19Z] <marostegui> dbmaint s5@eqiad T300600

gerritbot added a comment.Mar 21 2022, 5:52 AM

Change 772058 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1175: Disable notifications

https://gerrit.wikimedia.org/r/772058

gerritbot added a comment.Mar 21 2022, 5:53 AM

Change 772058 merged by Marostegui:

[operations/puppet@production] db1175: Disable notifications

https://gerrit.wikimedia.org/r/772058

ops-monitoring-bot added a comment.Mar 21 2022, 5:54 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1175.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2022, 6:19 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1175.eqiad.wmnet with OS bullseye executed with errors:

  • db1175 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Mar 21 2022, 6:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1175.eqiad.wmnet with OS bullseye

Marostegui added a comment.Mar 21 2022, 6:24 AM

db1175 isn't booting up and the idrac doesn't show anything, created T304280 for eqiad's DC Ops.

ops-monitoring-bot added a comment.Mar 21 2022, 6:43 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1175.eqiad.wmnet with OS bullseye executed with errors:

  • db1175 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson closed subtask T304280: db1175 not booting up as Resolved.Mar 21 2022, 9:11 PM
ops-monitoring-bot added a comment.Mar 22 2022, 5:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1175.eqiad.wmnet with OS bullseye

Marostegui added a comment.Mar 22 2022, 6:09 AM

db1175 was reimaged fine after Chris fixed it on-site

Marostegui updated the task description. (Show Details)Mar 22 2022, 6:09 AM
ops-monitoring-bot added a comment.Mar 22 2022, 6:12 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1175.eqiad.wmnet with OS bullseye completed:

  • db1175 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203220541_marostegui_2587373_db1175.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 22 2022, 12:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-22T12:24:49Z] <marostegui> dbmaint s3@eqiad T300600

gerritbot added a comment.Mar 23 2022, 6:06 AM

Change 773120 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1112: Disable notifications

https://gerrit.wikimedia.org/r/773120

gerritbot added a comment.Mar 23 2022, 6:07 AM

Change 773120 merged by Marostegui:

[operations/puppet@production] db1112: Disable notifications

https://gerrit.wikimedia.org/r/773120

ops-monitoring-bot added a comment.Mar 23 2022, 6:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1112.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 6:34 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1112.eqiad.wmnet with OS bullseye completed:

  • db1112 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203230609_marostegui_2812305_db1112.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Mar 23 2022, 6:38 AM

All these hosts are done - pending only the master swap T301850

Marostegui closed subtask T301850: Switchover s3 master (db1157 -> db1123) as Resolved.Mar 29 2022, 6:12 AM
ops-monitoring-bot added a comment.Mar 29 2022, 9:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1157.eqiad.wmnet with OS bullseye

Marostegui closed this task as Resolved.Mar 29 2022, 9:47 AM
Marostegui updated the task description. (Show Details)

Old master was reimaged - this is all done (apart from the hosts that have their own task)

ops-monitoring-bot added a comment.Mar 29 2022, 9:53 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1157.eqiad.wmnet with OS bullseye completed:

  • db1157 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203290924_marostegui_4105547_db1157.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot moved this task from In progress to Done on the DBA board.Mar 29 2022, 10:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits