Page MenuHomePhabricator
Create Task
Maniphest T299466

Q3:(Need By: TBD) rack/setup/install stat1009
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of stat1009

Hostname / Racking / Installation Details

Hostnames: stat100* (next available) stat1009
Racking Proposal: Replacing stat1006, no restrictions.
Networking/Subnet/VLAN/IP: Single 1G private1 vlan connection
Partitioning/Raid: hw raid10 of hdd, hwraid1 of ssds (same as other config I hosts)
OS Distro: buster

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

stat1009
  • - receive in system on procurement task T297738 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add a new partman recipe for the new H750 based stat serversoperations/puppetproduction+71 -0
updating site.pp for stat1009 and stat1010operations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jan 18 2022, 11:14 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Jan 19 2022, 5:06 PM
RobH renamed this task from (Need By: TBD) rack/setup/install stat1009 to Q3:(Need By: TBD) rack/setup/install stat1009.Jan 19 2022, 5:20 PM
RobH unsubscribed.
odimitrijevic moved this task from Incoming (new tickets) to Ops Week on the Data-Engineering board.Mar 11 2022, 4:46 PM
Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.EditedJun 2 2022, 8:28 PM
Jclark-ctr subscribed.

stat1009 B1 U17 cableid 1181 port 5

@Cmjohnson port 1 is damaged missing plastic clip to retain network cable it is connected to port 2

Jclark-ctr updated the task description. (Show Details)Jun 2 2022, 8:29 PM
• Cmjohnson added a comment.Jun 26 2022, 10:05 PM

@Jclark-ctr any word on getting this server fixed?

• Cmjohnson updated the task description. (Show Details)Jun 26 2022, 10:12 PM
gerritbot added a comment.Jun 26 2022, 11:51 PM

Change 808547 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] updating site.pp for stat1009 and stat1010

https://gerrit.wikimedia.org/r/808547

gerritbot added a project: Patch-For-Review.Jun 26 2022, 11:51 PM

Change 808547 merged by Cmjohnson:

[operations/puppet@production] updating site.pp for stat1009 and stat1010

https://gerrit.wikimedia.org/r/808547

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2022, 12:30 AM
• Cmjohnson moved this task from Racking Tasks to Blocked on the ops-eqiad board.Jun 27 2022, 12:56 AM
gerritbot added a comment.Jun 27 2022, 11:32 AM

Change 808870 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add a new partman recipe for the new H750 based stat servers

https://gerrit.wikimedia.org/r/808870

gerritbot added a project: Patch-For-Review.Jun 27 2022, 11:32 AM
gerritbot added a comment.Jun 27 2022, 12:53 PM

Change 808870 merged by Btullis:

[operations/puppet@production] Add a new partman recipe for the new H750 based stat servers

https://gerrit.wikimedia.org/r/808870

Maintenance_bot removed a project: Patch-For-Review.Jun 27 2022, 1:30 PM
ops-monitoring-bot added a comment.Jun 30 2022, 1:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host stat1009.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jun 30 2022, 1:33 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host stat1009.eqiad.wmnet with OS buster executed with errors:

  • stat1009 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 30 2022, 1:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host stat1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 30 2022, 1:48 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host stat1009.eqiad.wmnet with OS bullseye executed with errors:

  • stat1009 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson updated the task description. (Show Details)Jun 30 2022, 1:48 PM
• Cmjohnson added subscribers: Papaul, RobH.Jun 30 2022, 1:52 PM

@BTullis @RobH @Papaul I set the raid up so the raid 1 ssds were first and used the install script for buster. Buster fails to see to the disks, so I attempted to install with Bullseye, the entire install with through as it should and the server rebooted. However, it did not boot the hard drive and defaulted to pxe boot again. I did verify bios boot settings have hard drive first and pxe second. I have seen something similar on the an-presto servers as well. I am assuming this has something to do with new raid controller cards. Have any of you experienced this yet and do you have any suggestions?

• Cmjohnson added a comment.Jun 30 2022, 1:54 PM

@BTullis just read your response on an-presto and see that you're experiencing this with stat1010. Thank you for digging into it more.

BTullis added a subscriber: fgiunchedi.Jun 30 2022, 2:02 PM

Thanks @Cmjohnson - yes I think that this is very likely to be the same issue. That's useful that you've experienced exactly the same outcome on this as I did with stat1010.
Let's stick to bullseye for these servers from now on (if we can). Hopefully we'll have a working partmand recipe or BIOS/RAID/whatever configuration that works soon. @fgiunchedi and I are hoping to get to the bottom of it soon.

Ottomata added a comment.Jun 30 2022, 2:29 PM

We will have to rebuild hadoop for bullsye, eh? T310643: Build Bigtop 1.5 Hadoop packages for Bullseye

BTullis added a comment.Jul 1 2022, 11:07 AM
In T299466#8040485, @Ottomata wrote:

We will have to rebuild hadoop for bullsye, eh? T310643: Build Bigtop 1.5 Hadoop packages for Bullseye

Yep, looks that way.

BTullis added a comment.Jul 1 2022, 11:09 AM

@Cmjohnson I think that this should now work if you tweak the RAID controller configuration as described here: T297913#8041258

Let me know if it doesn't behave and I'll look into it again. Thanks.

ops-monitoring-bot added a comment.Jul 1 2022, 9:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host stat1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:48 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host stat1009.eqiad.wmnet with OS bullseye completed:

  • stat1009 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012118_cmjohnson_1378019_stat1009.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
• Cmjohnson closed this task as Resolved.Jul 1 2022, 10:03 PM
• Cmjohnson updated the task description. (Show Details)

resolved

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits