Page MenuHomePhabricator
Create Task
Maniphest T342174

Q1:rack/setup/install db21[88-95]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db21[88-95]

Hostname / Racking / Installation Details

Hostnames: db2188, db2189, db2190, db2191, db2192, db2193, db2194, db2195
Racking Proposal: Different racks (and if possible spread across rows too)
Networking Setup: of Connections: 1, Speed:1G Vlan: Private AAAA records: N
Partitioning/Raid: HW Raid: Y, Partman recipe and/or desired Raid Level: 10 @Marostegui will assign the correct partman recipe in puppet
OS Distro: Bullseye
Sub-team Technical Contact: @Marostegui

Per host setup checklist

db2188: RACK: B5 - U:39 PORT:38
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2189: RACK: B8 - U:37 PORT: 36
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2190: RACK: C1 - U:8 PORT:7
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2191: RACK: C3 - U:43 PORT:42
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2192: RACK: C5 - U:20 PORT:19
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2193: RACK: D5 - U:30 PORT:29
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2194: RACK: D6 - U:27 PORT:26
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2195: RACK: D8 - U:19 PORT:18
  • - receive in system on procurement task T341273 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
db21[88-95]: Notifications disabledoperations/puppetproduction+16 -0
site.pp: Add db21[88-95]operations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jul 18 2023, 8:47 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2023, 8:47 PM
RobH mentioned this in Unknown Object (Task).Jul 18 2023, 8:47 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jul 18 2023, 9:29 PM
Jhancock.wm claimed this task.Jul 31 2023, 9:56 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Aug 1 2023, 4:22 PM
gerritbot added a comment.Aug 2 2023, 6:36 AM

Change 944765 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add db21[88-95]

https://gerrit.wikimedia.org/r/944765

gerritbot added a project: Patch-For-Review.Aug 2 2023, 6:36 AM
gerritbot added a comment.Aug 2 2023, 7:14 AM

Change 944765 merged by Marostegui:

[operations/puppet@production] site.pp: Add db21[88-95]

https://gerrit.wikimedia.org/r/944765

Maintenance_bot removed a project: Patch-For-Review.Aug 2 2023, 7:32 AM
Jhancock.wm updated the task description. (Show Details)Aug 3 2023, 5:38 PM
Papaul updated the task description. (Show Details)Aug 3 2023, 10:36 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Aug 3 2023, 11:08 PM
ops-monitoring-bot added a comment.Aug 3 2023, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2188.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Aug 3 2023, 11:26 PM
ops-monitoring-bot added a comment.Aug 3 2023, 11:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2189.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 3 2023, 11:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Aug 3 2023, 11:41 PM
ops-monitoring-bot added a comment.Aug 3 2023, 11:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2191.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 12:09 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2188.codfw.wmnet with OS bullseye completed:

  • db2188 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308032350_pt1979_1718504_db2188.out
    • Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet,puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 4 2023, 12:18 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2189.codfw.wmnet with OS bullseye completed:

  • db2189 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308032359_pt1979_1728463_db2189.out
    • Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet,puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul updated the task description. (Show Details)Aug 4 2023, 12:20 AM
ops-monitoring-bot added a comment.Aug 4 2023, 12:25 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2191.codfw.wmnet with OS bullseye completed:

  • db2191 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308040009_pt1979_1738173_db2191.out
    • Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet,puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 4 2023, 12:33 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye executed with errors:

  • db2190 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Papaul updated the task description. (Show Details)Aug 4 2023, 12:37 AM
ops-monitoring-bot added a comment.Aug 4 2023, 12:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 12:39 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye executed with errors:

  • db2190 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
Papaul updated the task description. (Show Details)Aug 4 2023, 12:52 AM
Papaul subscribed.EditedAug 4 2023, 12:54 AM

@Jhancock.wm while installing the OS on db2190 and db2193 for some reason i lost connection to the server. When i checked the switch port the link was showing down. when back on site can you please check the network cable? Thanks

show interfaces descriptions | match db2190
ge-1/0/7        up    down db2190
papaul@asw-d-codfw> show interfaces descriptions ge-5/0/29
Interface       Admin Link Description
ge-5/0/29       up    down db2193
ops-monitoring-bot added a comment.Aug 4 2023, 2:22 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2192.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 2:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2193.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 2:26 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2193.codfw.wmnet with OS bullseye executed with errors:

  • db2193 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 4 2023, 2:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2194.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 3:03 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2192.codfw.wmnet with OS bullseye completed:

  • db2192 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308040246_pt1979_1905896_db2192.out
    • Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet,puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 4 2023, 3:20 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2194.codfw.wmnet with OS bullseye completed:

  • db2194 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308040256_pt1979_1918881_db2194.out
    • Unable to run puppet on config-master2001.codfw.wmnet,config-master1001.eqiad.wmnet,puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul updated the task description. (Show Details)Aug 4 2023, 3:22 AM
Jhancock.wm added a comment.Aug 4 2023, 1:25 PM

@Papaul db2190 and db2193 have active links now.

ops-monitoring-bot added a comment.Aug 4 2023, 1:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2193.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye executed with errors:

  • db2190 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 4 2023, 1:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye

Papaul added a comment.Aug 4 2023, 1:49 PM

@Jhancock.wm thank you.

ops-monitoring-bot added a comment.Aug 4 2023, 2:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2195.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 4 2023, 2:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2193.codfw.wmnet with OS bullseye completed:

  • db2193 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308041408_pt1979_2624100_db2193.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 4 2023, 2:31 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2190.codfw.wmnet with OS bullseye completed:

  • db2190 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308041411_pt1979_2626632_db2190.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 4 2023, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2195.codfw.wmnet with OS bullseye completed:

  • db2195 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308041428_pt1979_2642433_db2195.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul closed this task as Resolved.Aug 4 2023, 4:01 PM
Papaul updated the task description. (Show Details)

@Marostegui all your's. Have fun

gerritbot added a comment.Aug 7 2023, 4:12 AM

Change 945963 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db21[88-95]: Notifications disabled

https://gerrit.wikimedia.org/r/945963

gerritbot added a project: Patch-For-Review.Aug 7 2023, 4:12 AM

Change 945963 merged by Marostegui:

[operations/puppet@production] db21[88-95]: Notifications disabled

https://gerrit.wikimedia.org/r/945963

Marostegui added a comment.Aug 7 2023, 4:15 AM
In T342174#9069908, @Papaul wrote:

@Marostegui all your's. Have fun

Thank you!!

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 4:30 AM
ABran-WMF changed the status of subtask T343674: Productionize db21[88-95] from Open to In Progress.Dec 20 2023, 3:43 PM
ABran-WMF closed subtask T343674: Productionize db21[88-95] as Resolved.Feb 20 2024, 8:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits