Page MenuHomePhabricator
Create Task
Maniphest T401448

Q1:rack/setup/install ms-fe10[17-20]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ms-fe10[17-20]

Hostname / Racking / Installation Details

Hostnames: ms-fe1017, ms-fe1018. ms-fe1019, ms-fe1020
Racking Proposal: If there's room to put 1 in each of rows A-D (but avoiding A7/B2/C4/D4) that's great, otherwise they can go into E+F avoiding racks in there that already have ms-fe1* in
Networking Setup: # of Connections: 1 - Speed: 10G. - VLAN: Private
OS Distro: bullseye
Boot Method: Legacy BIOS
Sub-team Technical Contact: @MatthewVernon

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-fe1017
  • Receive in system on procurement task T399027 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-fe1018
  • Receive in system on procurement task T399027 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-fe1019
  • Receive in system on procurement task T399027 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-fe1020
  • Receive in system on procurement task T399027 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
swift: add 4 new eqiad frontends ms-fe10[17-20]operations/puppetproduction+13 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT401448 Q1:rack/setup/install ms-fe10[17-20]

Event Timeline

RobH created this task.Aug 7 2025, 9:44 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.
Maintenance_bot added a project: SRE.Aug 7 2025, 10:29 PM
Jclark-ctr claimed this task.Aug 19 2025, 11:33 AM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr updated the task description. (Show Details)Aug 19 2025, 12:15 PM
ops-monitoring-bot added a comment.Aug 19 2025, 1:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-fe1019.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 19 2025, 1:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-fe1018.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 19 2025, 1:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-fe1017.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Aug 19 2025, 1:36 PM
ops-monitoring-bot added a comment.Aug 19 2025, 1:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-fe1020.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 19 2025, 1:56 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-fe1019.eqiad.wmnet with OS bullseye completed:

  • ms-fe1019 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508191338_jclark_2040576_ms-fe1019.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 19 2025, 2:06 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-fe1018.eqiad.wmnet with OS bullseye completed:

  • ms-fe1018 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508191341_jclark_2045106_ms-fe1018.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 19 2025, 2:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-fe1017.eqiad.wmnet with OS bullseye completed:

  • ms-fe1017 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508191348_jclark_2046936_ms-fe1017.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Aug 19 2025, 2:09 PM
ops-monitoring-bot added a comment.Aug 19 2025, 3:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-fe1020.eqiad.wmnet with OS bullseye completed:

  • ms-fe1020 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508191447_jclark_2093684_ms-fe1020.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Aug 19 2025, 3:11 PM
Jclark-ctr updated the task description. (Show Details)
gerritbot added a comment.Aug 26 2025, 1:22 PM

Change #1182142 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: add 4 new eqiad frontends ms-fe10[17-20]

https://gerrit.wikimedia.org/r/1182142

gerritbot added a project: Patch-For-Review.Aug 26 2025, 1:22 PM
gerritbot added a comment.Aug 26 2025, 1:29 PM

Change #1182142 merged by MVernon:

[operations/puppet@production] swift: add 4 new eqiad frontends ms-fe10[17-20]

https://gerrit.wikimedia.org/r/1182142

Maintenance_bot removed a project: Patch-For-Review.Aug 26 2025, 1:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits