Page MenuHomePhabricator
Create Task
Maniphest T299435

Q3:(Need By: TBD) rack/setup/install ml-cache100[1-3]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ml-cache100[1-3]

Hostname / Racking / Installation Details

Hostnames: ml-cache100[1-3]
Racking Proposal: Spread in rows as much as possible, and bonus point if they don't share anything with the ml-serve100* nodes. If Rows E/F are available, utilize them to accomplish this.
Networking/Subnet/VLAN/IP: Single 10G private1 vlan connection.
Partitioning/Raid: default raid1 2device
OS Distro: Bullseye (default unless otherwise specified)

Per host setup checklist

ml-cache1001
  • - receive in system on procurement task T297638 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.

[]x - network port setup via netbox, run homer to commit

  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ml-cache1002
  • - receive in system on procurement task T297638 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ml-cache1003
  • - receive in system on procurement task T297638 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Adding ml-cache1001-3 to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 Resolved• CmjohnsonT299435 Q3:(Need By: TBD) rack/setup/install ml-cache100[1-3]

Event Timeline

RobH assigned this task to Jclark-ctr.Jan 18 2022, 6:03 PM
RobH created this task.
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH edited subscribers, added: calbon, elukey; removed: RobH.
Maintenance_bot added a project: SRE.Jan 18 2022, 6:45 PM
elukey moved this task from Unsorted to Backlog/SRE on the Machine-Learning-Team board.Jan 19 2022, 6:12 PM
Jclark-ctr updated the task description. (Show Details)Feb 9 2022, 11:15 PM
Jclark-ctr updated the task description. (Show Details)Feb 18 2022, 2:16 PM

ml-cache1001 E1 U23
ml-cache1002 E2 U23
ml-cache1003 F1 U23

Jclark-ctr added a comment.Feb 25 2022, 4:17 PM
namerack_nameportcableid

ml-cache1001 E1 23 20220147
ml-cache1002 E2 23 20220137
ml-cache1003 F1 23 20220125 |

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Mar 2 2022, 6:49 PM
Jclark-ctr subscribed.
gerritbot added a comment.Mar 24 2022, 3:21 PM

Change 773558 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] Adding ml-cache1001-3 to site.pp

https://gerrit.wikimedia.org/r/773558

gerritbot added a project: Patch-For-Review.Mar 24 2022, 3:21 PM
gerritbot added a comment.Mar 24 2022, 3:28 PM

Change 773558 merged by Cmjohnson:

[operations/puppet@production] Adding ml-cache1001-3 to site.pp

https://gerrit.wikimedia.org/r/773558

• Cmjohnson updated the task description. (Show Details)Mar 24 2022, 3:32 PM
• Cmjohnson updated the task description. (Show Details)
ops-monitoring-bot added a comment.Mar 24 2022, 3:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host ml-cache1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2022, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host ml-cache1001.eqiad.wmnet with OS bullseye completed:

  • ml-cache1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203241539_cmjohnson_3126180_ml-cache1001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Mar 24 2022, 4:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2022, 4:12 PM
ops-monitoring-bot added a comment.Mar 24 2022, 4:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host ml-cache1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2022, 4:35 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye completed:

  • ml-cache1002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203241608_cmjohnson_3131163_ml-cache1002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Mar 24 2022, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host ml-cache1003.eqiad.wmnet with OS bullseye completed:

  • ml-cache1003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203241618_cmjohnson_3132050_ml-cache1003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
• Cmjohnson closed this task as Resolved.Mar 24 2022, 5:55 PM
• Cmjohnson updated the task description. (Show Details)

Completed

elukey reopened this task as Open.Mar 25 2022, 7:16 AM

Hi Chris! I noticed that we have two nodes on the same ROW, would it be possible to move one elsewhere? We are going to host a Cassandra cluster on the nodes, and having 2/3 of the cluster unavailable for a ROW failure is a big risk. I can help with reimages etc.. if needed, sorry for the trouble!

• Cmjohnson added a comment.Mar 25 2022, 4:57 PM

@elukey I moved ml-cache1002 to row/rack C4.

ops-monitoring-bot added a comment.Mar 25 2022, 5:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 25 2022, 5:42 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye executed with errors:

  • ml-cache1002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Mar 28 2022, 4:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 28 2022, 4:59 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye executed with errors:

  • ml-cache1002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson added a comment.Mar 28 2022, 5:26 PM

@elukey Since moving the server, I cannot get it to install the OS correctly, can you please take a look. Thanks

elukey added a comment.Mar 29 2022, 9:43 AM

I tried to PXE boot and it didn't work, so I checked netbox and the interface listed looks weird: xe-4/0/010 https://netbox.wikimedia.org/dcim/interfaces/24986/

elukey@asw2-c-eqiad> show interfaces descriptions | match ml-cache 
xe-4/0/10       down  down ml-cache1002 {#4778}
elukey added a comment.Mar 29 2022, 9:56 AM

So I have deleted the /10 interface in netbox, and renamed the /010 to /10. Now homer offers me this diff:

Changes for 1 devices: ['asw2-c-eqiad.mgmt.eqiad.wmnet']

[edit interfaces interface-range disabled]
-    member xe-4/0/10;
[edit interfaces interface-range vlan-private1-c-eqiad]
     member xe-4/0/7 { ... }
+    member xe-4/0/10;
     member xe-4/0/11 { ... }
[edit interfaces interface-range vlan-private1-c-eqiad]
-    member xe-4/0/010;

Not entirely sure if the following is removed though:

elukey@asw2-c-eqiad> show interfaces descriptions xe-4/0/010  
Interface       Admin Link Description
xe-4/0/010                 ml-cache1002 {#4778}
ayounsi subscribed.Mar 29 2022, 9:59 AM

Looks good to deploy!

elukey added a comment.Mar 29 2022, 10:02 AM
elukey@asw2-c-eqiad> show interfaces descriptions xe-4/0/10  
Interface       Admin Link Description
xe-4/0/10       up    up   ml-cache1002 {#4778}

Way better now!

elukey added a comment.Mar 29 2022, 10:13 AM

@Cmjohnson I tried to reimage the node but I got spicerack.dhcp.DHCPError: target file ttyS1-115200/ml-cache1002.conf exists, I think that you have already a cookbook open for ml-cache1002, so if you can close it we can complete the task :)

• Cmjohnson added a comment.Apr 5 2022, 2:54 PM

@elukey closed

ops-monitoring-bot added a comment.Apr 6 2022, 7:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 6 2022, 7:31 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host ml-cache1002.eqiad.wmnet with OS bullseye executed with errors:

  • ml-cache1002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
elukey closed this task as Resolved.Apr 7 2022, 9:14 AM

Host reimaged correctly, all done!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL