install bast2003
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Apr 7 2023, 1:56 PM

Description

This task will track the racking, setup, and OS installation of bast2003

Hostname / Racking / Installation Details

Hostnames: bast2003
Racking Proposal: Replacing bast2002, no restrictions.
Networking Setup: # of Connections:1, Speed:1G, Vlan:Public, AAAA records:Y, Additional IP records (Cassandra)?: N
Partitioning/Raid: HW Raid:N, Partman recipe and/or desired Raid Level: sw raid 1 standard 2 dev
OS Distro: Bullseye
Sub-team Technical Contact: Moritz/JBond

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

bast2003: RACK: D8-U18 Port: 17

- receive in system on procurement task T333352 & in coupa
- rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
- add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
- network port setup via netbox, run homer from an active cumin host to commit
- bios/drac/serial setup/testing, see Lifecycle Steps & [x[ https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Automatic_setup | Automatic BIOS setup details ]]
- firmware update (idrac, bios, network, raid controller)
- operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::infrastructure_foundations
- OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Subject	Repo	Branch	Lines +/-
Remove bast2002 from bastion hiera entries	operations/puppet	production	+0 -2
wmf-laptop-sre: Add bast2003	operations/debs/wmf-sre-laptop	master	+2 -2
Add bast2003 to Bastion hiera settings	operations/puppet	production	+2 -0
Assign bastion role to bast2003	operations/puppet	production	+1 -1
Add bast2003 to site.pp	operations/puppet	production	+4 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Unknown Object (Task)
		Resolved		Papaul	T334287 Q3:rack/setup/install bast2003

Event Timeline

RobH created this task.Apr 7 2023, 1:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 7 2023, 1:56 PM

RobH added a parent task: Unknown Object (Task).Apr 7 2023, 1:56 PM

RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.

Jhancock.wm updated the task description. (Show Details)Apr 19 2023, 1:53 PM

RobH unsubscribed.Apr 19 2023, 1:58 PM

Papaul updated the task description. (Show Details)Apr 20 2023, 4:17 PM

Change 910558 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add bast2003 to site.pp

https://gerrit.wikimedia.org/r/910558

gerritbot added a project: Patch-For-Review.Apr 20 2023, 5:16 PM

Change 910558 merged by Papaul:

[operations/puppet@production] Add bast2003 to site.pp

https://gerrit.wikimedia.org/r/910558

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host bast2003.wikimedia.org with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2023, 5:29 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host bast2003.wikimedia.org with OS bullseye completed:

bast2003 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304201724_pt1979_4124136_bast2003.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

@MoritzMuehlenhoff all yours

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast2003.wikimedia.org with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast2003.wikimedia.org with OS bookworm completed:

bast2003 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202305040633_jmm_2267133_bast2003.out
- Checked BIOS boot parameters are back to normal
- Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB