Page MenuHomePhabricator
Create Task
Maniphest T342164

Q1:rack/setup/install pc101[56]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of pc101[56]

Hostname / Racking / Installation Details

Hostnames: pc1015, pc1016
Racking Proposal: Any rack (NOT in A1, B1, C5, D6), as long as they are different rows.
Networking Setup: # Connections:1, Speed: 1G Vlan: Private AAAA records: N
Partitioning/Raid: HW Raid: Y, Raid Level: 10 and @Marostegui will assign the partman recipe on puppet
OS Distro: Bullseye
Sub-team Technical Contact: @Marostegui

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

pc1015:
  • - receive in system on procurement task T341271 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
pc1016:
  • - receive in system on procurement task T341271 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
add pc10(15-16) T342164operations/puppetproduction+5 -0
site: Add pc101[56] as in setupoperations/puppetproduction+6 -1
pc1015,pc1016: New hosts to be set upoperations/puppetproduction+6 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT342164 Q1:rack/setup/install pc101[56]
 ResolvedNoneT343408 Productionize pc1015, p1016, pc2015 and pc2016
 ResolvedMarosteguiT350526 Upgrade pc4 to Debian Bookworm

Event Timeline

RobH created this task.Jul 18 2023, 7:45 PM
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
Maintenance_bot added a project: SRE.Jul 18 2023, 7:45 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.Jul 18 2023, 7:45 PM
RobH unsubscribed.
gerritbot added a comment.Jul 25 2023, 6:21 AM

Change 941042 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] pc1015,pc1016: New hosts to be set up

https://gerrit.wikimedia.org/r/941042

gerritbot added a project: Patch-For-Review.Jul 25 2023, 6:21 AM

Change 941042 merged by Marostegui:

[operations/puppet@production] pc1015,pc1016: New hosts to be set up

https://gerrit.wikimedia.org/r/941042

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2023, 6:30 AM
Marostegui added a subtask: T343408: Productionize pc1015, p1016, pc2015 and pc2016.Aug 3 2023, 7:09 AM
gerritbot added a comment.Aug 3 2023, 7:11 AM

Change 945535 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site: Add pc101[56] as in setup

https://gerrit.wikimedia.org/r/945535

gerritbot added a project: Patch-For-Review.Aug 3 2023, 7:11 AM

Change 945535 merged by Marostegui:

[operations/puppet@production] site: Add pc101[56] as in setup

https://gerrit.wikimedia.org/r/945535

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2023, 7:30 AM
Jclark-ctr updated the task description. (Show Details)Aug 10 2023, 1:55 PM
Jclark-ctr subscribed.EditedAug 24 2023, 6:28 PM

pc1015 A6 U33
pc1016. C6 U31

Jclark-ctr claimed this task.Aug 24 2023, 6:28 PM
VRiley-WMF subscribed.Aug 31 2023, 2:20 PM

pc1015 - A 6. U 33. Port 32. Cableid: 2839

VRiley-WMF added a comment.Sep 18 2023, 3:45 PM

pc1016 - C 6. U 31. port 30 CableID 3252

VRiley-WMF added a comment.Sep 19 2023, 7:41 PM

pc1016 - C 6. U 31. port 30 CableID 3252 is having issues, will recheck cabling

Jclark-ctr added a comment.Sep 20 2023, 11:54 PM

@VRiley-WMF Serial was entered into netbox incorrectly if you are not onsite sometimes you can look at procurement ticket packing slip that is attached.

Jclark-ctr updated the task description. (Show Details)Sep 21 2023, 12:10 AM

@VRiley-WMF if you have a screen going please check to see if its in process of doing something on pc1015
spicerack.dhcp.DHCPError: Snippet /etc/dhcp/automation/mgmt-eqiad/pc1015.mgmt.eqiad.wmnet.conf already exists, is there another operation in progress for the same device? If not you delete it and retry.

Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Sep 21 2023, 2:13 AM
VRiley-WMF added a comment.EditedSep 21 2023, 6:18 PM

@Jclark-ctr corrected the problem with on pc1015. Also, I did close my screen on this device. Would you be able to try again?

Jclark-ctr updated the task description. (Show Details)Sep 22 2023, 2:50 PM
gerritbot added a comment.Sep 26 2023, 7:14 PM

Change 961191 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] add pc10(15-16) T342164

https://gerrit.wikimedia.org/r/961191

gerritbot added a project: Patch-For-Review.Sep 26 2023, 7:14 PM
Jclark-ctr updated the task description. (Show Details)Sep 26 2023, 7:21 PM
gerritbot added a comment.Sep 26 2023, 7:24 PM

Change 961191 abandoned by Jclark-ctr:

[operations/puppet@production] add pc10(15-16) T342164

Reason:

Already added by Marostegui

https://gerrit.wikimedia.org/r/961191

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 7:32 PM
ops-monitoring-bot added a comment.Sep 26 2023, 7:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host pc1016.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 7:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host pc1015.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 26 2023, 8:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host pc1015.eqiad.wmnet with OS bullseye completed:

  • pc1015 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309262002_jhancock_719761_pc1015.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 26 2023, 8:31 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host pc1016.eqiad.wmnet with OS bullseye completed:

  • pc1016 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309262010_jhancock_719766_pc1016.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Sep 26 2023, 8:32 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm subscribed.

@Marostegui finished up

Marostegui changed the status of subtask T343408: Productionize pc1015, p1016, pc2015 and pc2016 from Stalled to Open.Sep 27 2023, 8:44 AM
Marostegui added a comment.Sep 27 2023, 8:52 AM

Thank you!

Marostegui closed subtask T343408: Productionize pc1015, p1016, pc2015 and pc2016 as Resolved.Nov 2 2023, 10:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL