Page MenuHomePhabricator
Create Task
Maniphest T416245

Q3:rack/setup/install ms-fe102[14]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ms-fe102[14]

Hostname / Racking / Installation Details

Hostnames: ms-fe102[1-4]
Racking Proposal: Where should these systems be racked? Avoid existing ms-fe nodes, please
Networking Setup: # of Connections:1- Speed:10G. - VLAN:Private
OS Distro: bullseye
Boot Method: UEFI
Sub-team Technical Contact: @MatthewVernon
Please add SRE-swift-storage to subsequent racking tasks

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-fe1021
  • Receive in system on procurement task T413266 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-fe1022
  • Receive in system on procurement task T413266 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-fe1023
  • Receive in system on procurement task T413266 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-fe1024
  • Receive in system on procurement task T413266 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Other Assignee
Jclark-ctr
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
installserver: new ms-fe nodes are UEFI bootedoperations/puppetproduction+1 -1
swift: add 4 new eqiad frontends ms-fe102[1-4]operations/puppetproduction+13 -1
installserver: fix preseed config for ms-fe102[1-4]operations/puppetproduction+1 -1
installserver: add EFI preseed config for ms-fe102[14]operations/puppetproduction+3 -0
Customize query in gerrit

Related Objects

Event Timeline

Jhancock.wm created this task.Feb 2 2026, 9:19 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptFeb 2 2026, 9:19 PM
Jhancock.wm mentioned this in Unknown Object (Task).Feb 2 2026, 9:20 PM
Maintenance_bot added a project: SRE.Feb 2 2026, 9:29 PM
Jclark-ctr moved this task from Backlog to Procurement on the ops-eqiad board.Feb 2 2026, 9:55 PM
Jclark-ctr moved this task from Procurement to Racking Tasks on the ops-eqiad board.
Jclark-ctr claimed this task.Feb 3 2026, 12:00 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr updated the task description. (Show Details)Feb 3 2026, 1:16 PM
Jclark-ctr added a subscriber: elukey.Feb 3 2026, 1:21 PM

@elukey These are failing as well, just like backup1015 in T414725. I haven’t made any changes to the server — it’s still using the default user and password.

elukey added a comment.Feb 3 2026, 4:23 PM

@Jclark-ctr sadly from Redfish I don't see any LinkUp:

>>> pprint(r.request("GET", f"{r.system_manager}/EthernetInterfaces/NIC.Integrated.1-2-1").json()['LinkStatus'])
None
>>> pprint(r.request("GET", f"{r.system_manager}/EthernetInterfaces/NIC.Integrated.1-1-1").json()['LinkStatus'])
None

It seems iDRAC 9 + firmware version 7.20.80.50, not sure why it behaves like that. I think that the best action to unblock this is to manually select NIC.Integrated.1-1-1, I don't have other ideas :(

Jclark-ctr added a comment.Feb 3 2026, 7:07 PM

@MatthewVernon can you help with update preseed.yaml for efi booting?

Screenshot 2026-02-03 at 2.06.59 PM.png (878×1 px, 263 KB)

Jclark-ctr reassigned this task from Jclark-ctr to MatthewVernon.Feb 3 2026, 7:25 PM
Jclark-ctr updated Other Assignee, added: Jclark-ctr.
Jclark-ctr subscribed.
gerritbot added a comment.Feb 4 2026, 8:38 AM

Change #1236671 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] installserver: add EFI preseed config for ms-fe102[14]

https://gerrit.wikimedia.org/r/1236671

gerritbot added a project: Patch-For-Review.Feb 4 2026, 8:38 AM
gerritbot added a comment.Feb 4 2026, 8:45 AM

Change #1236671 merged by Elukey:

[operations/puppet@production] installserver: add EFI preseed config for ms-fe102[14]

https://gerrit.wikimedia.org/r/1236671

elukey added a comment.Feb 4 2026, 8:47 AM

@Jclark-ctr Matthew is out this week, I just merged a change that should unblock you. Lemme know how it goes!

Maintenance_bot removed a project: Patch-For-Review.Feb 4 2026, 9:31 AM
ops-monitoring-bot added a comment.Feb 4 2026, 12:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1021.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 4 2026, 12:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1023.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 4 2026, 12:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1024.eqiad.wmnet with OS bullseye

Jclark-ctr claimed this task.Feb 4 2026, 12:44 PM
Jclark-ctr updated the task description. (Show Details)
ops-monitoring-bot added a comment.Feb 4 2026, 1:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1023.eqiad.wmnet with OS bullseye executed with errors:

  • ms-fe1023 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-fe1023.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 4 2026, 1:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1023.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 4 2026, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1021.eqiad.wmnet with OS bullseye completed:

  • ms-fe1021 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602041244_jclark_2527779_ms-fe1021.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Feb 4 2026, 1:12 PM
ops-monitoring-bot added a comment.Feb 4 2026, 1:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1024.eqiad.wmnet with OS bullseye completed:

  • ms-fe1024 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602041305_jclark_2528735_ms-fe1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Feb 4 2026, 1:37 PM
ops-monitoring-bot added a comment.Feb 4 2026, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1022.eqiad.wmnet with OS bullseye

gerritbot added a comment.Feb 4 2026, 2:02 PM

Change #1236750 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] installserver: fix preseed config for ms-fe102[1-4]

https://gerrit.wikimedia.org/r/1236750

gerritbot added a project: Patch-For-Review.Feb 4 2026, 2:02 PM
gerritbot added a comment.Feb 4 2026, 2:06 PM

Change #1236750 merged by Elukey:

[operations/puppet@production] installserver: fix preseed config for ms-fe102[1-4]

https://gerrit.wikimedia.org/r/1236750

ops-monitoring-bot added a comment.Feb 4 2026, 2:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1022.eqiad.wmnet with OS bullseye executed with errors:

  • ms-fe1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-fe1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 4 2026, 2:13 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1023.eqiad.wmnet with OS bullseye executed with errors:

  • ms-fe1023 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-fe1023.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 4 2026, 2:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1022.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 4 2026, 2:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ms-fe1023.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Feb 4 2026, 2:31 PM
ops-monitoring-bot added a comment.Feb 4 2026, 2:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1023.eqiad.wmnet with OS bullseye completed:

  • ms-fe1023 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602041434_jclark_2543945_ms-fe1023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Feb 4 2026, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ms-fe1022.eqiad.wmnet with OS bullseye completed:

  • ms-fe1022 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602041436_jclark_2543935_ms-fe1022.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Feb 4 2026, 2:59 PM
Jclark-ctr updated the task description. (Show Details)
MatthewVernon added a project: SRE-swift-storage.Mon, Feb 9, 11:54 AM
gerritbot added a comment.Mon, Feb 16, 11:25 AM

Change #1239643 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: add 4 new eqiad frontends ms-fe102[1-4]

https://gerrit.wikimedia.org/r/1239643

gerritbot added a project: Patch-For-Review.Mon, Feb 16, 11:25 AM
gerritbot added a comment.Mon, Feb 16, 11:32 AM

Change #1239645 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] installserver: new ms-fe nodes are UEFI booted

https://gerrit.wikimedia.org/r/1239645

gerritbot added a comment.Mon, Feb 16, 1:10 PM

Change #1239643 merged by MVernon:

[operations/puppet@production] swift: add 4 new eqiad frontends ms-fe102[1-4]

https://gerrit.wikimedia.org/r/1239643

gerritbot added a comment.Mon, Feb 16, 2:38 PM

Change #1239645 merged by MVernon:

[operations/puppet@production] installserver: new ms-fe nodes are UEFI booted

https://gerrit.wikimedia.org/r/1239645

Maintenance_bot removed a project: Patch-For-Review.Mon, Feb 16, 3:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits