install sretest1002
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Apr 10 2023, 1:40 PM

Project Tags

Referenced Files

None

Subscribers

Description

This task will track the racking, setup, and OS installation of sretest1002

Hostname / Racking / Installation Details

Hostnames: sretest1002
Racking Proposal: anywhere
Networking Setup: # of Connections:1 , Speed:10G. Vlan:Private AAAA records:Y, Additional IP records: N
Partitioning/Raid: will have hardware raid and be unicorn, can give to jbond/volans for imaging
OS Distro: Bullseye
Sub-team Technical Contact: jbond/volans

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

sretest1003:

- receive in system on procurement task T333355 & in coupa
- rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
- add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
- network port setup via netbox, run homer from an active cumin host to commit
- bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
- firmware update (idrac, bios, network, raid controller)
- operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::infrastructure_foundations
- OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Add sretest1003 to site.pp and netboot.cfg	Papaul	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
					Unknown Object (Task)
		Resolved		Jhancock.wm	T334393 Q3:rack/setup/install sretest1002

Event Timeline

RobH created this task.Apr 10 2023, 1:40 PM

RobH added a parent task: Unknown Object (Task).

RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

RobH mentioned this in Unknown Object (Task).

RobH unsubscribed.

Jclark-ctr claimed this task.Apr 26 2023, 10:46 PM

Jclark-ctr updated the task description. (Show Details)

sretest1002 @jbond @Volans this server name is already in use can this be named sretest1003?

sretest1002 A4 U31 PORT 45 CABLEID 230304500290

Jclark-ctr updated the task description. (Show Details)Apr 27 2023, 3:31 PM

Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.

Jclark-ctr updated the task description. (Show Details)

Jclark-ctr updated the task description. (Show Details)Apr 27 2023, 3:37 PM

In T334393#8810727, @Jclark-ctr wrote:

sretest1002 @jbond @Volans this server name is already in use can this be named sretest1003?

@Jclark-ctr correct, this is the HW refresh for sretest1002 so yes sretest1003 is fine as hostname, thanks and sorry for the confusion.

sretest1003 A4 U31 PORT 45 CABLEID 230304500290

Jclark-ctr updated the task description. (Show Details)Apr 28 2023, 11:57 AM

Jclark-ctr updated the task description. (Show Details)Apr 28 2023, 12:08 PM

Jhancock.wm claimed this task.Jun 8 2023, 2:04 PM

Jhancock.wm added a subscriber: Jclark-ctr.

Jhancock.wm updated the task description. (Show Details)Jun 12 2023, 4:46 PM

Jhancock.wm updated the task description. (Show Details)Jun 12 2023, 4:55 PM

@Jhancock.wm you can proceed with the OS install

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host sretest1003.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host sretest1003.eqiad.wmnet with OS bullseye executed with errors:

sretest1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- The reimage failed, see the cookbook logs for the details

Change 929418 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add sretest1003 to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/929418

Change 929418 merged by Papaul:

[operations/puppet@production] Add sretest1003 to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/929418

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host sretest1003.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2023, 9:31 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host sretest1003.eqiad.wmnet with OS bullseye completed:

sretest1003 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306122224_jhancock_3301583_sretest1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active
- The sre.puppet.sync-netbox-hiera cookbook was run successfully

Jhancock.wm updated the task description. (Show Details)Jun 12 2023, 10:43 PM

@jbond or @Volans finished this. all yours.

Q3:rack/setup/install sretest1002Closed, ResolvedPublicActions