Page MenuHomePhabricator
Create Task
Maniphest T372432

Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x)
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of X

Hostname / Racking / Installation Details

Hostnames: ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x)
Racking Proposal: the ml-serve ones should have rack diversity with the rest of ml-serve1xxx, ml-lab can be anywhere but not both on the same rack, dse-k8s-worker1009 should have rack diversity with the rest of dse-k8s-worker1xxx
Networking Setup: # of Connections: 1x10G, 1x management
Partitioning/Raid:No HW raid, ml-serve1xxx shouldbe covered by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1057177, The DSE host will be covered by an upcoming change. ML-Lab we haven't decided yet, no need to install them, just racked and having power (but being off) is fine.
OS Distro: Bookworm
Sub-team Technical Contact: @klausman

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ml-serve1009
  • Receive in system on procurement task T366522 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ml-serve1010
  • Receive in system on procurement task T366522 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ml-serve1011
  • Receive in system on procurement task T366522 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ml-lab1001
  • Receive in system on procurement task T366522 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ml-lab1002
  • Receive in system on procurement task T366522 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
dse-k8s-worker1009
  • Receive in system on procurement task T366522 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
BGP peers: add lsw1-e5-eqiad and lsw1-f5-eqiadoperations/deployment-chartsmaster+8 -0
manifests: move new GPU hosts in eqiad from insetup to worker roleoperations/puppetproduction+2 -10
preseed: Add ml-lab machines and dse-k8s-worker1009operations/puppetproduction+4 -1
manifest/hiera/conftool: Add new ML GPU hosts in eqiadoperations/puppetproduction+13 -0
site.pp: Add ml-labs machine entries for setupoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
RobH renamed this task from Q#:rack/setup/install X to Q1:rack/setup/install ml-serve1009-1011 (3x), ml-lab1001-1002 (2x), dse-k8s-worker1009 (1x).Aug 13 2024, 7:41 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to klausman.Aug 13 2024, 7:44 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

@klausman: Would you, or someone on your team, please update the puppet repo for these new hosts for insetup entries in site.pp as well as their preseed.yml parittion entries?

Please reference the patchset with this task, then unassign yourself once merged, thanks!

Maintenance_bot added a project: SRE.Aug 13 2024, 8:29 PM
klausman added a comment.EditedAug 14 2024, 10:02 AM
gerritbot added a comment.Aug 14 2024, 10:07 AM

Change #1062688 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] manifest/hiera/conftool: Add new ML GPU hosts in eqiad

https://gerrit.wikimedia.org/r/1062688

gerritbot added a project: Patch-For-Review.Aug 14 2024, 10:07 AM

Change #1062667 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] site.pp: Add ml-labs machine entries for setup

https://gerrit.wikimedia.org/r/1062667

RobH unsubscribed.Aug 14 2024, 6:15 PM
klausman moved this task from Unsorted to In Progress on the Machine-Learning-Team board.Aug 15 2024, 2:15 PM
gerritbot added a comment.Aug 16 2024, 1:54 PM

Change #1062667 merged by Klausman:

[operations/puppet@production] site.pp: Add ml-labs machine entries for setup

https://gerrit.wikimedia.org/r/1062667

gerritbot added a comment.Aug 19 2024, 8:11 AM

Change #1062688 abandoned by Klausman:

[operations/puppet@production] manifest/hiera/conftool: Add new ML GPU hosts in eqiad

Reason:

Not needed anymore, changes were covered by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1057177

https://gerrit.wikimedia.org/r/1062688

klausman removed klausman as the assignee of this task.Aug 19 2024, 8:14 AM
VRiley-WMF updated the task description. (Show Details)Aug 19 2024, 6:24 PM
VRiley-WMF subscribed.Aug 26 2024, 4:46 PM

ml-serve1009
Rack A2
U19
CableID 4897
Port 7

ml-serve1010
Rack E5
U3
CableID 230304500090
Port 19

ml-serve1011
Rack F5
U3
CableID 230304500083
Port 19

ml-lab1001
Rack E5
U5
CableID 20220176
Port 21

ml-lab1002
Rack F5
U5
CableID 20220175
Port 21

dse-k8s-worker1009
Rack A2
U 23
CableID 5340
Port 18

VRiley-WMF updated the task description. (Show Details)Aug 26 2024, 5:07 PM
ops-monitoring-bot added a comment.Aug 28 2024, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-serve1010.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 28 2024, 2:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 28 2024, 2:36 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-serve1010.eqiad.wmnet with OS bookworm completed:

  • ml-serve1010 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408281421_jclark_717144_ml-serve1010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 28 2024, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-serve1011.eqiad.wmnet with OS bookworm

Jclark-ctr updated the task description. (Show Details)Aug 28 2024, 2:54 PM
ops-monitoring-bot added a comment.Aug 28 2024, 3:23 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-serve1011.eqiad.wmnet with OS bookworm completed:

  • ml-serve1011 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408281505_jclark_727958_ml-serve1011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 28 2024, 3:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 28 2024, 3:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-serve1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 28 2024, 3:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 28 2024, 4:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 28 2024, 4:20 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-serve1009.eqiad.wmnet with OS bookworm completed:

  • ml-serve1009 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408281602_jclark_742664_ml-serve1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Aug 28 2024, 4:20 PM
ops-monitoring-bot added a comment.Aug 28 2024, 4:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker1009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" dse-k8s-worker1009.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 28 2024, 4:38 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 28 2024, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 28 2024, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

Jclark-ctr updated the task description. (Show Details)Aug 28 2024, 5:08 PM
Jclark-ctr assigned this task to klausman.Aug 28 2024, 5:19 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr subscribed.

@klausman. If you can update preseed.yaml file for these servers they continue to fail since file is not updated

ml-lab1001, ml-lab1002, dse-k8s-worker1009

ops-monitoring-bot added a comment.Aug 28 2024, 5:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker1009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" dse-k8s-worker1009.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 28 2024, 6:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
gerritbot added a comment.Aug 29 2024, 7:24 AM

Change #1068656 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] preseed: Add ml-lab machines and dse-k8s-worker1009

https://gerrit.wikimedia.org/r/1068656

gerritbot added a comment.Aug 29 2024, 7:29 AM

Change #1068657 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/puppet@production] manifests: move new ML GPU hosts in eqiad from insetup to worker role

https://gerrit.wikimedia.org/r/1068657

ops-monitoring-bot added a comment.Aug 29 2024, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 1:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm

Jclark-ctr claimed this task.Aug 29 2024, 1:37 PM
gerritbot added a comment.Aug 29 2024, 1:43 PM

Change #1068656 merged by Klausman:

[operations/puppet@production] preseed: Add ml-lab machines and dse-k8s-worker1009

https://gerrit.wikimedia.org/r/1068656

ops-monitoring-bot added a comment.Aug 29 2024, 1:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1002.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 1:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 1:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker1009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" dse-k8s-worker1009.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 1:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 2:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 2:38 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1002.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 2:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker1009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" dse-k8s-worker1009.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 2:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 3:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm completed:

  • ml-lab1002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408291502_jclark_985537_ml-lab1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 29 2024, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
Jclark-ctr updated the task description. (Show Details)Aug 29 2024, 8:41 PM
ops-monitoring-bot added a comment.Aug 29 2024, 8:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 9:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 9:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm completed:

  • dse-k8s-worker1009 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202408292110_jclark_1054840_dse-k8s-worker1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 29 2024, 9:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 10:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2024, 10:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2024, 11:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" ml-lab1001.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
gerritbot added a comment.Aug 30 2024, 1:42 PM

Change #1068657 merged by Klausman:

[operations/puppet@production] manifests: move new GPU hosts in eqiad from insetup to worker role

https://gerrit.wikimedia.org/r/1068657

gerritbot added a comment.Aug 30 2024, 2:28 PM

Change #1069225 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] BGP peers: add lsw1-e5-eqiad and lsw1-f5-eqiad

https://gerrit.wikimedia.org/r/1069225

gerritbot added a comment.Aug 30 2024, 2:57 PM

Change #1069225 merged by jenkins-bot:

[operations/deployment-charts@master] BGP peers: add lsw1-e5-eqiad and lsw1-f5-eqiad

https://gerrit.wikimedia.org/r/1069225

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2024, 3:30 PM
ops-monitoring-bot added a comment.Sep 4 2024, 1:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 4 2024, 2:36 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console ml-lab1001.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 4 2024, 3:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 4 2024, 7:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console ml-lab1001.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 5 2024, 7:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 5 2024, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host dse-k8s-worker1009.eqiad.wmnet with OS bookworm completed:

  • dse-k8s-worker1009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409051936_jclark_2650778_dse-k8s-worker1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Jclark-ctr updated the task description. (Show Details)Sep 5 2024, 8:38 PM
Jclark-ctr added a subscriber: Papaul.Sep 11 2024, 11:17 PM

@Papaul i have updated bmc and bios with no change to server. can you assist with this last server?

Papaul added a comment.Sep 12 2024, 12:56 AM

Some notes here:
I checked console redirect, it was working for me and the issue i found was that the server was not able to PXE boot made the changes below.
In the BIOS under "Boot"

  • Boot mode select was set to UEFI should be legacy
  • Fixed boot order priorities for Boot option 2: should be network it was CD/DVD

the server is now trying to pxe boot

You can resume with the install

Screenshot from 2024-09-11 19-54-34.png (304×814 px, 19 KB)

ops-monitoring-bot added a comment.Sep 13 2024, 8:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by klausman@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2024, 8:43 AM

Cookbook cookbooks.sre.hosts.reimage started by klausman@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console ml-lab1001.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 8:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by klausman@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2024, 9:09 AM

Cookbook cookbooks.sre.hosts.reimage started by klausman@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm executed with errors:

  • ml-lab1001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "sudo install-console ml-lab1001.eqiad.wmnet" to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 13 2024, 9:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by klausman@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 13 2024, 9:41 AM

Cookbook cookbooks.sre.hosts.reimage started by klausman@cumin1002 for host ml-lab1001.eqiad.wmnet with OS bookworm completed:

  • ml-lab1001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202409130928_klausman_200521_ml-lab1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
klausman updated the task description. (Show Details)Sep 13 2024, 9:42 AM
Jclark-ctr closed this task as Resolved.Sep 13 2024, 3:05 PM
klausman moved this task from In Progress to 2025-2026 Q1 Done on the Machine-Learning-Team board.Sep 17 2024, 2:54 PM
ops-monitoring-bot added a comment.Oct 21 2024, 12:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by klausman@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 21 2024, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage started by klausman@cumin1002 for host ml-lab1002.eqiad.wmnet with OS bookworm completed:

  • ml-lab1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202410211224_klausman_1749379_ml-lab1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• isarantopoulos moved this task from 2025-2026 Q1 Done to Task Archive on the Machine-Learning-Team board.Nov 4 2024, 8:07 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits