Page MenuHomePhabricator

Q3:(Need By: TBD) rack/setup/install restbase-dev200[123].codfw.wmnet
Closed, ResolvedPublic

Description

This task will track the racking, setup, and OS installation of restbase-dev200[123].codfw.wmnet

Hostname / Racking / Installation Details

Hostnames: restbase-dev200[123].codfw.wmnet - when configuring in netbox ensure to specify 2 cassandra instances
Racking Proposal: restbase-dev2001 in row b, restbase-dev2002 in row c, restbase-dev2003 in row d
Networking/Subnet/VLAN/IP: match other restbase, (1) 1g private1 vlan - when configuring in netbox ensure to specify 2 cassandra instances
Partitioning/Raid: Match other restbase-dev instances (partman/raid0.cfg partman/raid0-4dev.cfg)
OS Distro: buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

restbase-dev2001 B5 U21 ge-5/0/25
  • - receive in system on procurement task T297154 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
restbase-dev2002 C3 U38 ge-3/0/37
  • - receive in system on procurement task T297154 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
restbase-dev2003 D3 U10 ge-3/0/9
  • - receive in system on procurement task T297154 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Event Timeline

RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH edited subscribers, added: hnowlan; removed: RobH.

Change 763583 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add restbase-dev200[1-3] to site.pp and netboot

https://gerrit.wikimedia.org/r/763583

Change 763583 merged by Papaul:

[operations/puppet@production] Add restbase-dev200[1-3] to site.pp and netboot

https://gerrit.wikimedia.org/r/763583

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host restbase-dev2001.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host restbase-dev2002.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host restbase-dev2001.codfw.wmnet with OS buster completed:

  • restbase-dev2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202171850_pt1979_3896529_restbase-dev2001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host restbase-dev2003.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host restbase-dev2002.codfw.wmnet with OS buster completed:

  • restbase-dev2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202171904_pt1979_3897524_restbase-dev2002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host restbase-dev2003.codfw.wmnet with OS buster completed:

  • restbase-dev2003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202171924_pt1979_3901924_restbase-dev2003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged