Page MenuHomePhabricator
Create Task
Maniphest T342176

Q1:rack/setup/install db12[26-33]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db12[26-33]

Hostname / Racking / Installation Details

Hostnames: db1226, db1227, db1228, db1229, db1230, db1231, db1232, db1233
Racking Proposal: Different racks (and if possible spread across rows too)
Networking Setup: # of Connections: 1, Speed:1G Vlan: Private AAAA records: N
Partitioning/Raid: HW Raid: Y, Partman recipe and/or desired Raid Level: 10 @Marostegui will assign the correct partman recipe in puppet
OS Distro: Bullseye
Sub-team Technical Contact: @Marostegui

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

db1226:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1227:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1228:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1229:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1230:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1231:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1232:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1233:
  • - receive in system on procurement task 341274 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
add db1229 to site.ppJclark-ctroperations/puppetproduction+1 -1
Add new servers db1226-33Jclark-ctroperations/puppetproduction+5 -0
site.pp: Add db12[26-33]Marosteguioperations/puppetproduction+1 -1
install_server: Allow install db12[26-33]Marosteguioperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
RobH renamed this task from Q1:rack/setup/install X to Q1:rack/setup/install db12[26-33].Jul 18 2023, 9:15 PM
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jul 18 2023, 9:29 PM
gerritbot added a comment.Aug 3 2023, 7:24 AM

Change 945536 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Allow install db12[26-33]

https://gerrit.wikimedia.org/r/945536

gerritbot added a project: Patch-For-Review.Aug 3 2023, 7:24 AM

Change 945536 merged by Marostegui:

[operations/puppet@production] install_server: Allow install db12[26-33]

https://gerrit.wikimedia.org/r/945536

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2023, 7:30 AM
Jclark-ctr updated the task description. (Show Details)Aug 7 2023, 1:51 PM
gerritbot added a comment.Aug 9 2023, 6:24 AM

Change 947200 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add db12[26-33]

https://gerrit.wikimedia.org/r/947200

gerritbot added a project: Patch-For-Review.Aug 9 2023, 6:24 AM

Change 947200 merged by Marostegui:

[operations/puppet@production] site.pp: Add db12[26-33]

https://gerrit.wikimedia.org/r/947200

Maintenance_bot removed a project: Patch-For-Review.Aug 9 2023, 6:30 AM
VRiley-WMF subscribed.EditedAug 31 2023, 3:29 PM

db1226 - A 3. U 14. Port 18 CableID 1859
db1227 - A 5. U 8. Port 25 CableID 0023000145
db1228 - B 5. U 23. Port 12 CableID 3809

VRiley-WMF added a comment.Sep 6 2023, 4:57 PM

db1229 - B 6. U 29. Port 23 CableID 1896
db1230 - C 3. U 23. Port 8 CableID 3310
db1231 - C 6. U 11. Port 15 CableID 3221
db1232 - D 3. U 14. Port 22 CableID 3687
db1233 - D 6. U 8. Port 7 CableID 1189

Marostegui mentioned this in T346454: Master and candidate master of s5 and s8 in eqiad are in the same row.Sep 18 2023, 10:43 AM
Jclark-ctr updated the task description. (Show Details)Sep 19 2023, 9:29 PM
Jclark-ctr updated the task description. (Show Details)Sep 19 2023, 11:23 PM
Jclark-ctr updated the task description. (Show Details)Sep 19 2023, 11:26 PM
Jclark-ctr updated the task description. (Show Details)Sep 19 2023, 11:40 PM
Jclark-ctr updated the task description. (Show Details)Sep 20 2023, 12:13 AM
gerritbot added a comment.Sep 20 2023, 11:23 PM

Change 959359 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] Add new servers db1226-33

https://gerrit.wikimedia.org/r/959359

gerritbot added a project: Patch-For-Review.Sep 20 2023, 11:23 PM
Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Sep 20 2023, 11:40 PM
gerritbot added a comment.Sep 21 2023, 2:02 AM

Change 959359 merged by Andrew Bogott:

[operations/puppet@production] Add new servers db1226-33

https://gerrit.wikimedia.org/r/959359

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2023, 2:10 AM
ops-monitoring-bot added a comment.Sep 26 2023, 1:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1226.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1227.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:02 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1228.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1230.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1231.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1232.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1233.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 2:30 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1226.eqiad.wmnet with OS bullseye completed:

  • db1226 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309260214_jhancock_508910_db1226.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2023, 2:39 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1230.eqiad.wmnet with OS bullseye completed:

  • db1230 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309260220_jhancock_509856_db1230.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2023, 2:39 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1233.eqiad.wmnet with OS bullseye completed:

  • db1233 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309260223_jhancock_510426_db1233.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Sep 26 2023, 2:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1231.eqiad.wmnet with OS bullseye completed:

  • db1231 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309260225_jhancock_510057_db1231.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2023, 2:43 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1232.eqiad.wmnet with OS bullseye completed:

  • db1232 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309260227_jhancock_510199_db1232.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Sep 26 2023, 2:54 AM
Jhancock.wm added subscribers: Jclark-ctr, Jhancock.wm.

@Jclark-ctr or @VRiley-WMF can you check the cables on 1227 and 1228? they're showing as connected but won't pxe. it could be the port on the switch side, the cable, or sfp if a 10G switch. Message me on irc or @ me here when you find it.

1229 has an issue I'm still working out.

1226, 1230-1233 are complete

ops-monitoring-bot added a comment.Sep 26 2023, 3:06 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 26 2023, 3:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 4:09 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 28 2023, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1228.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1228.eqiad.wmnet with OS bullseye executed with errors:

  • db1228 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 28 2023, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 2 2023, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1227.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 2 2023, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1228.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 2 2023, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 2 2023, 1:36 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 2 2023, 1:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1227.eqiad.wmnet with OS bullseye completed:

  • db1227 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021339_jhancock_2238365_db1227.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Oct 2 2023, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1228.eqiad.wmnet with OS bullseye completed:

  • db1228 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310021341_jhancock_2238382_db1228.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Oct 2 2023, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye

Jhancock.wm updated the task description. (Show Details)Oct 2 2023, 1:59 PM
ops-monitoring-bot added a comment.Oct 2 2023, 3:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 6 2023, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2023, 3:31 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
Jhancock.wm added a comment.Oct 6 2023, 3:34 PM

db1229 has an image but is not passing part of the reimage process. here is the latest error. provision script was run again before this run and bios was checked. Confirmed the DNS records on netbox.

[49/50, retrying in 147.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title db1229 not found yet
Exception raised while executing cookbook sre.hosts.reimage:
Traceback (most recent call last):

File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 212, in run
  raw_ret = runner.run()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 586, in run
  self._populate_puppetdb()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 428, in _populate_puppetdb
  poll_puppetdb()
File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 210, in wrapper
  return func(*args, **kwargs)
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 423, in poll_puppetdb
  raise SpicerackError(f'Nagios_host resource with title {self.host} not found yet')

spicerack.exceptions.SpicerackError: Nagios_host resource with title db1229 not found yet
The reimage failed, see the cookbook logs for the details
Reimage executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 10 2023, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 10 2023, 2:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
Jclark-ctr added a subscriber: Papaul.Oct 10 2023, 2:17 PM

@Papaul Just got same error on db1229

Execution of preseeded command "wget -O /tmp/late_command │ │
│ │ http://apt.wikimedia.org/autoinstall/scripts/late_command.sh && sh │ │
│ │ /tmp/late_command" failed with exit code 1.

Marostegui added a subtask: T344036: Productionize db12[26-49].Oct 17 2023, 7:09 AM
Papaul added a comment.Oct 17 2023, 1:18 PM

@Jclark-ctr this is now fixed. You can try running the re-image again

ops-monitoring-bot added a comment.Oct 18 2023, 7:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 18 2023, 7:30 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 18 2023, 7:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 18 2023, 7:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • The reimage failed, see the cookbook logs for the details
Jclark-ctr added a comment.Oct 18 2023, 7:35 PM

@Papaul this is still failing

[25/50, retrying in 75.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title db1229 not found yet
Exception raised while executing cookbook sre.hosts.reimage:

ops-monitoring-bot added a comment.Oct 18 2023, 7:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

Volans subscribed.Oct 18 2023, 7:40 PM

@Jclark-ctr:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, No puppet role has been assigned to this node. (file: /etc/puppet/manifests/site.pp, line: 2939, column: 9) on node db1229.eqiad.wmnet
Jclark-ctr added a comment.Oct 18 2023, 7:56 PM

thanks @Volans

gerritbot added a comment.Oct 18 2023, 7:56 PM

Change 966923 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] add db1229 to site.pp

https://gerrit.wikimedia.org/r/966923

gerritbot added a project: Patch-For-Review.Oct 18 2023, 7:56 PM

Change 966923 merged by Jclark-ctr:

[operations/puppet@production] add db1229 to site.pp

https://gerrit.wikimedia.org/r/966923

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 8:10 PM
ops-monitoring-bot added a comment.Oct 18 2023, 8:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 18 2023, 8:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 18 2023, 8:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 18 2023, 8:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 18 2023, 8:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye executed with errors:

  • db1229 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 18 2023, 8:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 18 2023, 9:23 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host db1229.eqiad.wmnet with OS bullseye completed:

  • db1229 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310182102_jclark_2941298_db1229.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Oct 18 2023, 9:23 PM
Jclark-ctr claimed this task.
Jclark-ctr updated the task description. (Show Details)
• ABran-WMF changed the status of subtask T344036: Productionize db12[26-49] from Open to In Progress.Dec 20 2023, 3:43 PM
• ABran-WMF closed subtask T344036: Productionize db12[26-49] as Resolved.Feb 8 2024, 10:59 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits