Page MenuHomePhabricator
Create Task
Maniphest T299608

Q3:(Need By: TBD) rack/setup/install elastic20[73-86]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of elastic20[73-86]

Hostname / Racking / Installation Details

Hostnames: elastic2073-elastic2086

Racking Proposal:

Current state (before 14 new servers; 36 total number):
Row A: elastic2037|elastic2038|elastic2039|elastic2040|elastic2055|elastic2056|elastic2061|elastic2062|elastic2069
Row B: elastic2041|elastic2042|elastic2043|elastic2044|elastic2057|elastic2058|elastic2063|elastic2064|elastic2070
Row C:
elastic2045|elastic2046|elastic2047|elastic2048|elastic2049|elastic2059|elastic2065|elastic2066|elastic2071
Row D:
elastic2050|elastic2051|elastic2052|elastic2053|elastic2054|elastic2060|elastic2067|elastic2068|elastic2072

We want to balance the total fleet between the four rows A-D as best as possible, but we need to make sure the new hosts are allocated into 10G racks.

Here's our proposed racking, but if there's insufficient row space, just try to balance as best as possible across rows while maintaining the invariant that every host has a 10G port:

Proposed new state (after adding 14 new servers; 50 total servers):

Row A: elastic2037|elastic2038|elastic2039|elastic2040|elastic2055|elastic2056|elastic2061|elastic2062|elastic2069|elastic2073|elastic2074|elastic2075|elastic2076
Row B: elastic2041|elastic2042|elastic2043|elastic2044|elastic2057|elastic2058|elastic2063|elastic2064|elastic2070|elastic2077|elastic2078|elastic2079|elastic2080
Row C:
elastic2045|elastic2046|elastic2047|elastic2048|elastic2049|elastic2059|elastic2065|elastic2066|elastic2071|elastic2081|elastic2082|elastic2083
Row D:
elastic2050|elastic2051|elastic2052|elastic2053|elastic2054|elastic2060|elastic2067|elastic2068|elastic2072|elastic2084|elastic2085|elastic2086

Networking/Subnet/VLAN/IP: 10G, same subnet as existing elastic2* servers (private VLAN)
Partitioning/Raid: RAID0 software (elasticsearch-raid0.cfg - already configured for elastic* in netboot.cfg)
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

elastic2073
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2074
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2075
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2076
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2077
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2078
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2079
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2080
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2081
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2082
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2083
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2084
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2085
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
elastic2086
  • - receive in system on procurement task T297642 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add elastic208[3-6] to site.ppoperations/puppetproduction+1 -1
Add new elastic2073 to elastic2082 to site.ppoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT294806 [Epic] Search platform - Hardware requests for 2021-2022
Unknown Object (Task)
 ResolvedPapaulT299608 Q3:(Need By: TBD) rack/setup/install elastic20[73-86]

Event Timeline

RobH created this task.Jan 19 2022, 10:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 19 2022, 10:51 PM
RobH renamed this task from (Need By: TBD) rack/setup/install elastic20[73-86] to Q3:(Need By: TBD) rack/setup/install elastic20[73-86].Jan 19 2022, 10:51 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
MPhamWMF moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.Jan 31 2022, 4:40 PM
Papaul updated the task description. (Show Details)Feb 15 2022, 11:12 PM
Papaul updated the task description. (Show Details)Feb 16 2022, 3:05 PM
Papaul updated the task description. (Show Details)Feb 17 2022, 7:30 PM
Volans updated the task description. (Show Details)Feb 22 2022, 7:21 PM
gerritbot added a comment.Feb 22 2022, 9:16 PM

Change 764897 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new elastic2073 to elastic2082 to site.pp

https://gerrit.wikimedia.org/r/764897

gerritbot added a project: Patch-For-Review.Feb 22 2022, 9:16 PM
gerritbot added a comment.Feb 22 2022, 9:20 PM

Change 764897 merged by Papaul:

[operations/puppet@production] Add new elastic2073 to elastic2082 to site.pp

https://gerrit.wikimedia.org/r/764897

Maintenance_bot removed a project: Patch-For-Review.Feb 22 2022, 10:10 PM
Volans updated the task description. (Show Details)Feb 23 2022, 1:14 PM
Volans updated the task description. (Show Details)Feb 23 2022, 7:06 PM
Papaul updated the task description. (Show Details)Feb 24 2022, 12:05 AM
ops-monitoring-bot added a comment.Feb 24 2022, 12:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2073.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Feb 24 2022, 12:27 AM
ops-monitoring-bot added a comment.Feb 24 2022, 12:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2074.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 12:51 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2073.codfw.wmnet with OS bullseye completed:

  • elastic2073 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202240021_pt1979_779236_elastic2073.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 12:53 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2075.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 12:59 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2074.codfw.wmnet with OS bullseye completed:

  • elastic2074 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202240028_pt1979_779753_elastic2074.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 1:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2075.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 1:01 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2075.codfw.wmnet with OS bullseye executed with errors:

  • elastic2075 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2022, 1:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2076.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 1:27 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2075.codfw.wmnet with OS bullseye completed:

  • elastic2075 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202240052_pt1979_784324_elastic2075.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 1:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2077.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 1:38 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2076.codfw.wmnet with OS bullseye completed:

  • elastic2076 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202240107_pt1979_788706_elastic2076.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 1:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 1:57 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2077.codfw.wmnet with OS bullseye completed:

  • elastic2077 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202240127_pt1979_791362_elastic2077.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 2:15 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2078.codfw.wmnet with OS bullseye completed:

  • elastic2078 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202240139_pt1979_795457_elastic2078.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul updated the task description. (Show Details)Feb 24 2022, 2:16 AM
ops-monitoring-bot added a comment.Feb 24 2022, 4:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2079.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 4:50 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2079.codfw.wmnet with OS bullseye completed:

  • elastic2079 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202241619_pt1979_903085_elastic2079.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 4:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2080.codfw.wmnet with OS bullseye

Volans updated the task description. (Show Details)Feb 24 2022, 5:11 PM
ops-monitoring-bot added a comment.Feb 24 2022, 5:30 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2080.codfw.wmnet with OS bullseye completed:

  • elastic2080 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202241659_pt1979_907926_elastic2080.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 5:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2081.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 6:13 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2081.codfw.wmnet with OS bullseye completed:

  • elastic2081 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202241743_pt1979_915279_elastic2081.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2082.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Feb 24 2022, 6:23 PM
ops-monitoring-bot added a comment.Feb 24 2022, 6:53 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2082.codfw.wmnet with OS bullseye completed:

  • elastic2082 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202241822_pt1979_920787_elastic2082.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 6:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2083.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 7:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2084.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 8:10 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2083.codfw.wmnet with OS bullseye executed with errors:

  • elastic2083 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2022, 8:14 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2084.codfw.wmnet with OS bullseye executed with errors:

  • elastic2084 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Feb 24 2022, 8:48 PM

Change 765600 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add elastic208[3-6] to site.pp

https://gerrit.wikimedia.org/r/765600

gerritbot added a project: Patch-For-Review.Feb 24 2022, 8:48 PM

Change 765600 merged by Papaul:

[operations/puppet@production] Add elastic208[3-6] to site.pp

https://gerrit.wikimedia.org/r/765600

ops-monitoring-bot added a comment.Feb 24 2022, 8:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2084.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 8:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2083.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Feb 24 2022, 9:10 PM
ops-monitoring-bot added a comment.Feb 24 2022, 9:11 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2084.codfw.wmnet with OS bullseye completed:

  • elastic2084 (WARN)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202242050_pt1979_939993_elastic2084.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 9:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2085.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 9:18 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2083.codfw.wmnet with OS bullseye completed:

  • elastic2083 (WARN)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202242059_pt1979_940567_elastic2083.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 9:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host elastic2086.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2022, 9:42 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2085.codfw.wmnet with OS bullseye completed:

  • elastic2085 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202242113_pt1979_945997_elastic2085.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 24 2022, 9:50 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host elastic2086.codfw.wmnet with OS bullseye completed:

  • elastic2086 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202242119_pt1979_946434_elastic2086.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Feb 24 2022, 9:53 PM
Papaul updated the task description. (Show Details)

@Gehel @RKemper all yours

RKemper mentioned this in T300943: Service implementation for elastic20[61-86].codfw.wmnet.Jun 2 2022, 8:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL