Page MenuHomePhabricator
Create Task
Maniphest T342166

Q1:rack/setup/install db12[34-49]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db12[34-49]

Hostname / Racking / Installation Details

Hostnames: db1234, db1235, db1236, db1237, db1238, db1239, db1240, db1241, db1242, db1243, db1244, db1245, db1246, db1247, db1248, db1249
Racking Proposal: Different racks (and if possible spread across rows too)
Networking Setup: # of Connections: 1, Speed:1G Vlan: Private AAAA records: N
Partitioning/Raid: HW Raid: Y, Partman recipe and/or desired Raid Level: 10 @Marostegui will assign the correct partman recipe in puppet
OS Distro: Bullseye
Sub-team Technical Contact: @Marostegui

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

db1234:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1235:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1236:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1237:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1238:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1239:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1240:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1241:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1242:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1243:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1244:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1245:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1246:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1247:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1248:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1249:
  • - receive in system on procurement task T341269 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
install_server: Add db12[34-49] to reimage listMarosteguioperations/puppetproduction+1 -1
db12[34-49]: Add with notifications disabledMarosteguioperations/puppetproduction+48 -0
site.pp: New hosts db12[34-49]Marosteguioperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jul 18 2023, 8:04 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH mentioned this in Unknown Object (Task).
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jul 18 2023, 8:29 PM
gerritbot added a comment.Aug 4 2023, 6:28 AM

Change 945700 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: New hosts db12[34-49]

https://gerrit.wikimedia.org/r/945700

gerritbot added a project: Patch-For-Review.Aug 4 2023, 6:28 AM
gerritbot added a comment.Aug 4 2023, 7:09 AM

Change 945700 merged by Marostegui:

[operations/puppet@production] site.pp: New hosts db12[34-49]

https://gerrit.wikimedia.org/r/945700

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2023, 7:10 AM
gerritbot added a comment.Aug 7 2023, 5:59 AM

Change 946340 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db12[34-49]: Add with notifications disabled

https://gerrit.wikimedia.org/r/946340

gerritbot added a project: Patch-For-Review.Aug 7 2023, 5:59 AM
gerritbot added a comment.Aug 7 2023, 6:03 AM

Change 946340 merged by Marostegui:

[operations/puppet@production] db12[34-49]: Add with notifications disabled

https://gerrit.wikimedia.org/r/946340

Marostegui updated the task description. (Show Details)Aug 7 2023, 6:04 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 6:10 AM
gerritbot added a comment.Aug 9 2023, 6:14 AM

Change 947199 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Add db12[34-49] to reimage list

https://gerrit.wikimedia.org/r/947199

gerritbot added a project: Patch-For-Review.Aug 9 2023, 6:14 AM

Change 947199 merged by Marostegui:

[operations/puppet@production] install_server: Add db12[34-49] to reimage list

https://gerrit.wikimedia.org/r/947199

Maintenance_bot removed a project: Patch-For-Review.Aug 9 2023, 6:30 AM
Marostegui changed the status of subtask T344036: Productionize db12[26-49] from Open to Stalled.Aug 11 2023, 6:08 AM
Jclark-ctr updated the task description. (Show Details)Aug 23 2023, 2:59 PM
Jclark-ctr added subscribers: wiki_willy, RobH, Jclark-ctr.Aug 31 2023, 1:42 PM

@wiki_willy @Marostegui @RobH can we get some clarification on racking. ticket list Speed:1G Vlan. but came with 10g cards and on procurement doc list 10g. We did start racking these in 1g but have stopped until we get confirmation on 10g or 1g.

Marostegui added a comment.Aug 31 2023, 1:52 PM

We order them with 10G just in case, but we only use the 1G one.

VRiley-WMF subscribed.EditedAug 31 2023, 3:39 PM

db1234 - A 3. U 39. port 33 CableID 1945
db1235 - A 3. U 40. port 34 CableID 1903

RobH unsubscribed.Aug 31 2023, 5:43 PM
VRiley-WMF added a comment.Sep 12 2023, 3:15 PM

db1236 - A 5. U 27. port 45 CableID 3160
db1237 - A 5. U 33. port 33 CableID 1962
db1238 - B 5. U 14. port 23 CableID 4048
db1239 - B 5. U 15. port 09 CableID 3793

VRiley-WMF added a comment.Sep 12 2023, 6:24 PM

db1240 - B 6. U 03. port 01 CableID 1170
db1241 - B 6. U 04. port 05 CableID 1274
db1242 - C 3. U 12. port 25 CableID 5090
db1243 - C 3. U 13. port 04 CableID 2871

VRiley-WMF added a comment.Sep 13 2023, 4:28 PM

db1244 - C 6. U 09. port 6 CableID 3188
db1245 - C 6. U 10. port 7 CableID 3189
db1246 - D 3. U 03. port 0 CableID 3377
db1247 - D 3. U 04. port 1 CableID 3378
db1248 - D 8. U 04. port 0 CableID 3593
db1249 - D 8. U 07. port 3 CableID 3385

Jclark-ctr added a comment.Sep 21 2023, 1:48 AM

@VRiley-WMF db1247 Fails to provision please check cables / serial number in morning

Jclark-ctr updated the task description. (Show Details)Sep 21 2023, 2:12 AM
Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.
VRiley-WMF added a comment.Sep 21 2023, 4:31 PM

@Jclark-ctr

db1247 - Verified serial numbers. Verified port and CableID. All seem to be correct.

Swapped out cable with a new cable.

old: db1247 - D 3. U 04. port 1 CableID 3378
new: db1247 - D 3. U 04. port 1 CableID 2013339101755

CableID has been updated as well.

Jclark-ctr updated the task description. (Show Details)Sep 22 2023, 3:24 PM
Jclark-ctr updated the task description. (Show Details)Sep 22 2023, 3:32 PM
ops-monitoring-bot added a comment.Sep 27 2023, 2:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1234.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1235.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1236.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1237.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1238.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:09 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1239.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1240.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1241.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 27 2023, 2:38 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1234.eqiad.wmnet with OS bullseye completed:

  • db1234 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270221_jhancock_793116_db1234.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:40 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1235.eqiad.wmnet with OS bullseye completed:

  • db1235 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270223_jhancock_793297_db1235.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1237.eqiad.wmnet with OS bullseye completed:

  • db1237 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270226_jhancock_793689_db1237.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:44 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1238.eqiad.wmnet with OS bullseye completed:

  • db1238 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270228_jhancock_793804_db1238.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:46 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1236.eqiad.wmnet with OS bullseye completed:

  • db1236 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270230_jhancock_793477_db1236.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:47 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1239.eqiad.wmnet with OS bullseye completed:

  • db1239 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270232_jhancock_793986_db1239.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:50 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1240.eqiad.wmnet with OS bullseye completed:

  • db1240 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270233_jhancock_794146_db1240.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 27 2023, 2:51 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1241.eqiad.wmnet with OS bullseye completed:

  • db1241 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309270233_jhancock_794320_db1241.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Sep 27 2023, 2:52 AM
ops-monitoring-bot added a comment.Sep 28 2023, 1:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1242.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1243.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1244.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1245.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1246.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1247.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1248.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host db1249.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2023, 2:31 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1243.eqiad.wmnet with OS bullseye completed:

  • db1243 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280214_jhancock_1072100_db1243.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 28 2023, 2:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1242.eqiad.wmnet with OS bullseye completed:

  • db1242 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280215_jhancock_1071894_db1242.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 28 2023, 2:35 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1244.eqiad.wmnet with OS bullseye completed:

  • db1244 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280217_jhancock_1072357_db1244.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 28 2023, 2:39 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1246.eqiad.wmnet with OS bullseye completed:

  • db1246 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280220_jhancock_1072751_db1246.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 28 2023, 2:39 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1248.eqiad.wmnet with OS bullseye completed:

  • db1248 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280219_jhancock_1073141_db1248.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Sep 28 2023, 2:40 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1247.eqiad.wmnet with OS bullseye completed:

  • db1247 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280223_jhancock_1072953_db1247.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 28 2023, 2:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1249.eqiad.wmnet with OS bullseye completed:

  • db1249 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280225_jhancock_1073374_db1249.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 28 2023, 2:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host db1245.eqiad.wmnet with OS bullseye completed:

  • db1245 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309280225_jhancock_1072549_db1245.out
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Sep 28 2023, 2:48 AM
Jhancock.wm closed this task as Resolved.Sep 28 2023, 2:54 AM
Jhancock.wm claimed this task.
Jhancock.wm subscribed.

@Marostegui this is completed

Marostegui added a comment.Sep 28 2023, 5:18 AM

Thank you!!

Marostegui changed the status of subtask T344036: Productionize db12[26-49] from Stalled to Open.Sep 28 2023, 5:54 AM
• ABran-WMF changed the status of subtask T344036: Productionize db12[26-49] from Open to In Progress.Dec 20 2023, 3:43 PM
• ABran-WMF closed subtask T344036: Productionize db12[26-49] as Resolved.Feb 8 2024, 10:59 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits