Maniphest T319428

install arclamp2001.codfw.wmnet
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Oct 5 2022, 3:12 PM

Description

This task will track the racking, setup, and OS installation of arclamp2001.codfw.wmnet

Hostname / Racking / Installation Details

Hostnames: arclamp2001.codfw.wmnet
Racking Proposal: row D if possible
Networking Setup1 1G connection with AAAA records and no additional IPs on private1-d-codfw / 10.192.48.0/22
Partitioning/Raid: sw raid raid1-2dev.cfg
OS Distro: Bullseye
Sub-team Technical Contact:** @akosiaris @MoritzMuehlenhoff or @Krinkle

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

arclamp2001.codfw.wmnet:

- receive in system on procurement task T316907 & in coupa
- rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
- add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
- network port setup via netbox, run homer from an active cumin host to commit
- bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
- firmware update (idrac, bios, network, raid controller)
- operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
- OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

	Subject	Repo	Branch	Lines +/-
	Add new node to site.pp and to netboot	operations/puppet	production	+18 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T200108 Increase retention of ArcLamp SVGs to 2 years
Resolved	fgiunchedi	T316223 Expand RAM on arclamp hosts and move them to baremetal
		Unknown Object (Task)
Resolved	Papaul	T319428 Q2:rack/setup/install arclamp2001.codfw.wmnet
Resolved	fgiunchedi	T319429 service implementation tracking: arclamp2001.codfw.wmnet

Event Timeline

RobH created this task.Oct 5 2022, 3:12 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2022, 3:12 PM

RobH updated the task description. (Show Details)Oct 5 2022, 3:13 PM

RobH moved this task from Backlog to Procurement on the ops-codfw board.

RobH mentioned this in Unknown Object (Task).

RobH added a parent task: Unknown Object (Task).

RobH added subscribers: MoritzMuehlenhoff, Krinkle.

RobH mentioned this in T319429: service implementation tracking: arclamp2001.codfw.wmnet.Oct 5 2022, 3:15 PM

jijiki moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.Oct 10 2022, 8:54 AM

RobH unsubscribed.Oct 10 2022, 4:46 PM

Dzahn changed the status of subtask T319429: service implementation tracking: arclamp2001.codfw.wmnet from Open to Stalled.Oct 12 2022, 7:52 PM

Krinkle renamed this task from Q2:rack/setup/install webperf2005.codfw.wmnet to Q2:rack/setup/install arclamp2001.codfw.wmnet.Oct 15 2022, 10:20 PM

Krinkle updated the task description. (Show Details)

Dzahn changed the status of subtask T319429: service implementation tracking: arclamp2001.codfw.wmnet from Stalled to Open.Nov 9 2022, 5:57 PM

Dzahn mentioned this in T316223: Expand RAM on arclamp hosts and move them to baremetal.Nov 9 2022, 6:02 PM

Papaul updated the task description. (Show Details)Nov 14 2022, 6:09 PM

updated subteam contacts based on T316223#8381863

Change 856651 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new node to site.pp and to netboot

https://gerrit.wikimedia.org/r/856651

Change 856651 merged by Papaul:

[operations/puppet@production] Add new node to site.pp and to netboot

https://gerrit.wikimedia.org/r/856651

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2022, 7:30 PM

Papaul updated the task description. (Show Details)Nov 14 2022, 11:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host arclamp2001.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host arclamp2001.codfw.wmnet with OS bullseye completed:

arclamp2001 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211142352_pt1979_3797446_arclamp2001.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
- Updated Netbox status planned -> active

This is complete

fgiunchedi closed subtask T319429: service implementation tracking: arclamp2001.codfw.wmnet as Resolved.Jan 19 2023, 11:26 AM

Q2:rack/setup/install arclamp2001.codfw.wmnetClosed, ResolvedPublicActions