Page MenuHomePhabricator
Create Task
Maniphest T407032

Q2:rack/setup/install aqs102[3-7]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of aqs102[3-7]

Hostname / Racking / Installation Details

Hostnames: aqs102[3-7]
Racking Proposal:

New hostReplacesCassandra rack IDPlacement
aqs1023aqs1010rack1Anywhere in rows A or D, (avoiding other aqs hosts if possible)
aqs1024aqs1011rack2Anywhere in rows B or E, (avoiding other aqs hosts if possible)
aqs1025aqs1012rack3Anywhere in rows C or F, (avoiding other aqs hosts if possible)
aqs1026aqs1014rack2Anywhere in rows B or E, (avoiding other aqs hosts if possible)
aqs1027aqs1015rack3Anywhere in rows C or F, (avoiding other aqs hosts if possible)

Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: Bullseye
Boot Method: UEFI
Sub-team Technical Contact: @Eevans

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

aqs1023
  • Receive in system on procurement task T404774 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
aqs1024
  • Receive in system on procurement task T404774 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
aqs1025
  • Receive in system on procurement task T404774 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
aqs1026
  • Receive in system on procurement task T404774 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
aqs1027
  • Receive in system on procurement task T404774 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
sre.hosts.provision: add custom config for the new aqs Supermicrosoperations/cookbooksmaster+1 -0
Provision hosts aqs102[3-7] (refresh of aqs101[0-2,4-5])operations/puppetproduction+9 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT407032 Q2:rack/setup/install aqs102[3-7]

Event Timeline

RobH created this task.Oct 10 2025, 7:30 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to Eevans.Oct 10 2025, 7:32 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

@Eevans,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH unsubscribed.Oct 10 2025, 7:32 PM
gerritbot added a comment.Oct 10 2025, 7:53 PM

Change #1195276 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] Provision hosts aqs102[3-7] (refresh of aqs101[0-2,4-5])

https://gerrit.wikimedia.org/r/1195276

gerritbot added a project: Patch-For-Review.Oct 10 2025, 7:53 PM
gerritbot added a comment.Oct 14 2025, 3:12 PM

Change #1195276 merged by Eevans:

[operations/puppet@production] Provision hosts aqs102[3-7] (refresh of aqs101[0-2,4-5])

https://gerrit.wikimedia.org/r/1195276

Maintenance_bot removed a project: Patch-For-Review.Oct 14 2025, 3:31 PM
Jclark-ctr updated the task description. (Show Details)Dec 10 2025, 12:22 PM
Jclark-ctr claimed this task.Dec 10 2025, 12:30 PM
gerritbot added a comment.Dec 10 2025, 3:24 PM

Change #1217222 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/cookbooks@master] sre.hosts.provision: add custom config for the new aqs Supermicros

https://gerrit.wikimedia.org/r/1217222

gerritbot added a project: Patch-For-Review.Dec 10 2025, 3:24 PM
gerritbot added a comment.Dec 10 2025, 3:30 PM

Change #1217222 merged by Elukey:

[operations/cookbooks@master] sre.hosts.provision: add custom config for the new aqs Supermicros

https://gerrit.wikimedia.org/r/1217222

Maintenance_bot removed a project: Patch-For-Review.Dec 10 2025, 3:31 PM
ops-monitoring-bot added a comment.Dec 10 2025, 8:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host aqs1023.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 10 2025, 8:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host aqs1024.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 10 2025, 8:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host aqs1026.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 10 2025, 8:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host aqs1027.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 10 2025, 8:49 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host aqs1023.eqiad.wmnet with OS bullseye completed:

  • aqs1023 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512102033_jclark_3383037_aqs1023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 10 2025, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host aqs1027.eqiad.wmnet with OS bullseye completed:

  • aqs1027 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512102041_jclark_3385290_aqs1027.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Dec 10 2025, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host aqs1024.eqiad.wmnet with OS bullseye completed:

  • aqs1024 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512102037_jclark_3385277_aqs1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 10 2025, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host aqs1026.eqiad.wmnet with OS bullseye completed:

  • aqs1026 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512102044_jclark_3385285_aqs1026.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 11 2025, 12:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host aqs1025.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 11 2025, 1:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host aqs1025.eqiad.wmnet with OS bullseye completed:

  • aqs1025 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512111244_jclark_3502152_aqs1025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Dec 11 2025, 1:02 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr mentioned this in T407414: aqs1012 is down.
Jhancock.wm mentioned this in T412458: root user not on newest batches of supermicro servers..Dec 11 2025, 9:09 PM
Eevans mentioned this in T412830: Hardware refresh of aqs101[0-2,4-5] w/ aqs102[3-7].Dec 16 2025, 2:41 PM
Jclark-ctr mentioned this in T413704: Degraded RAID on an-worker1168.Jan 6 2026, 12:22 AM
Jclark-ctr mentioned this in T413559: Degraded RAID on aqs1015.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits