Page MenuHomePhabricator

Q3:rack/setup/install snapshot1016 & snapshot1017
Closed, ResolvedPublic

Description

This task will track the racking, setup, and OS installation of snapshot101[67]

Hostname / Racking / Installation Details

Hostnames: snapshot1016, 1017 will be the new names, following the standard naming convention for these boxes.
Racking Proposal: Please put these in rows with fewer existing snapshot hosts (except 1008/9 which are to be replaed by these). Additionally, please put in racks without snapshot boxes in them already, if possible.
Networking Setup: 1G Nic, private vlan, just like the previous snapshot hosts please.
Partitioning/Raid: As with the previous boxes, software raid 1 of the two 480GB SSDs, using the same partman recipe combo: partman/standard.cfg partman/raid1-2dev.cfg
OS Distro: We're not yet on bullseye with our other hosts, so let's do buster; if we get the upgrade done before these hosts are ready for the base install, the task will get updated.
Sub-team Technical Contact: @ArielGlenn will be your contact for this time around. Thank you for choosing ATG services.

Per host setup checklist

snapshot1016:
  • - receive in system on procurement task T333363 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
snapshot1017:
  • - receive in system on procurement task T333363 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Event Timeline

RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH mentioned this in Unknown Object (Task).

snapshot1016 C6 U26 port 24 cableid 3230
snapshot1017 D6 U39. port 39 Cableid. 23000051

Change 928877 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add snapshot101[6-7] to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/928877

Change 928877 merged by Papaul:

[operations/puppet@production] Add snapshot101[6-7] to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/928877

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Installing Buster on those 2 server is giving the error below. Buster is not able ti detect the driver and the controlled used in those serves is Dell HBA3555i

─────────────────────────┤ [!] Detect disks ├───────────────────────────┐
  │                                                                         │
  │ No disk drive was detected. If you know the name of the driver needed   │
  │ by your disk drive, you can select it from the list.                    │
  │                                                                         │
  │ Driver needed for your disk drive:

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS bullseye

I tried to install Bullseye on one of the node it did install with no issues so the problem is from our Debian Buster installer

The driver is is simply not present in the Linux kernel present in Buster, so the problem isn't in the Buster installer per se :-)

However, in https://phabricator.wikimedia.org/T319067#8300254 I have created an installer setup which allows to install Buster with a Linux 5.10 backport, which can be used here.

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

@ArielGlenn i am still working on those servers after @MoritzMuehlenhoff show me the fix on installing Buster on those servers i tried it on the first node and the install complete the servers boots and put me in the busy box mode. when i do df -h i don't see any disk listed so i am still troubleshooting.

Volume group "vg0" not found
  Cannot process volume group vg0
ALERT!  /dev/mapper/vg0-root does not exist.  Dropping to a shell!
BusyBox v1.30.1 (Debian 1:1.30.1-4) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

In the busybox shell, what does "uname -a" show as the running kernel version?

@MoritzMuehlenhoff

(initramfs) uname -a
Linux (none) 4.19.0-24-amd64 #1 SMP Debian 4.19.282-1 (2023-04-29) x86_64 GNU/Linux
(initramfs)

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details

@MoritzMuehlenhoff

(initramfs) uname -a
Linux (none) 4.19.0-24-amd64 #1 SMP Debian 4.19.282-1 (2023-04-29) x86_64 GNU/Linux
(initramfs)

If this shows Linux 4.19, then it didn't pick up the Buster installer environment with the 5.10 backport, but standard Buster. Did you pass --pxe-media installer510 to the reimage cookbook?

@MoritzMuehlenhoff yes i did pass -pxe-media installer510 to the reimage cookbook. I can tray again.

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Change 929732 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Install Linux 5.10 for snapshot1016/1017

https://gerrit.wikimedia.org/r/929732

Change 929732 merged by Muehlenhoff:

[operations/puppet@production] Install Linux 5.10 for snapshot1016/1017

https://gerrit.wikimedia.org/r/929732

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1017.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1017.eqiad.wmnet with OS buster

Change 929767 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add snapshot101[67]

https://gerrit.wikimedia.org/r/929767

Change 929767 merged by Papaul:

[operations/puppet@production] Add snapshot101[67]

https://gerrit.wikimedia.org/r/929767

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster executed with errors:

  • snapshot1016 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306131555_pt1979_238435_snapshot1016.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster

Change 929772 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add snapshot101[67] to site.pp with role insetup

https://gerrit.wikimedia.org/r/929772

Change 929772 merged by Papaul:

[operations/puppet@production] Add snapshot101[67] to site.pp with role insetup

https://gerrit.wikimedia.org/r/929772

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1016.eqiad.wmnet with OS buster completed:

  • snapshot1016 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202306131758_pt1979_385666_snapshot1016.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202306131802_pt1979_385666_snapshot1016.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306131806_pt1979_385666_snapshot1016.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host snapshot1017.eqiad.wmnet with OS buster completed:

  • snapshot1017 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202306131656_pt1979_305359_snapshot1017.out, asking the operator what to do
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202306131733_pt1979_305359_snapshot1017.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306131806_pt1979_305359_snapshot1017.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully

@MoritzMuehlenhoff thanks the patch works
@ArielGlenn I had to change the role to insetup::core_platform since puppet was failing with the the role dumps::generation::worker::testbed since on site.pp it said

# NOTE: new snapshot hosts must also be manually added to
# hieradata/common.yaml:dumps_nfs_clients for dump nfs mount,

so I will let you change the role when you are ready to put the server in service. The 2 nodes are ready. Thank you

... The 2 nodes are ready. Thank you

Thank you, we'll take 'em! :-)