Page MenuHomePhabricator
Create Task
Maniphest T418908

Q4:rack/setup/install pc102[1-4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of pc102[1-4]

Hostname / Racking / Installation Details

<Each procurement task also has a Hostnames: pc1021 pc1022 pc1023 pc1024
Racking Proposal: One per row if possible
Networking Setup: # of Connections:1 Speed:1G/10G (we can do both, prefer 10G, but we currently use 1G). - VLAN:Private
OS Distro: Debian Trixie
Boot Method: UEFI
Sub-team Technical Contact: @Marostegui
with racking details, with the heading of "Hostname / Racking / Installation Details", this section will need to be copied and pasted here.>

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

pc1021
  • Receive in system on procurement task T417070 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
pc1022
  • Receive in system on procurement task T417070 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
pc1023
  • Receive in system on procurement task T417070 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
pc1024
  • Receive in system on procurement task T417070 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site.pp: Add pc102[1-4]operations/puppetproduction+5 -0
installserver: Install pc102[1-4]operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Mar 3 2026, 6:51 PM
RobH assigned this task to Marostegui.
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH mentioned this in Unknown Object (Task).Mar 3 2026, 6:51 PM
RobH added a parent task: Unknown Object (Task).
gerritbot added a comment.Mar 4 2026, 7:58 AM

Change #1247907 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] installserver: Install pc102[1-4]

https://gerrit.wikimedia.org/r/1247907

gerritbot added a project: Patch-For-Review.Mar 4 2026, 7:58 AM
gerritbot added a comment.Mar 4 2026, 8:02 AM

Change #1247907 merged by Marostegui:

[operations/puppet@production] installserver: Install pc102[1-4]

https://gerrit.wikimedia.org/r/1247907

gerritbot added a comment.Mar 4 2026, 8:17 AM

Change #1247911 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add pc102[1-4]

https://gerrit.wikimedia.org/r/1247911

gerritbot added a comment.Mar 4 2026, 8:19 AM

Change #1247911 merged by Marostegui:

[operations/puppet@production] site.pp: Add pc102[1-4]

https://gerrit.wikimedia.org/r/1247911

Marostegui removed Marostegui as the assignee of this task.Mar 4 2026, 8:20 AM

Patches are ready

Marostegui updated the task description. (Show Details)Mar 4 2026, 8:21 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 4 2026, 8:30 AM
VRiley-WMF claimed this task.Apr 17 2026, 8:54 PM
Marostegui added a comment.Apr 20 2026, 5:09 AM

@VRiley-WMF please note these hosts require RAID 10 (just saying cause there were some config confusion in codfw and they ended with RAID 0 instead).

VRiley-WMF added a comment.Apr 20 2026, 10:56 AM

Understood, thank you for the heads up! @Marostegui

VRiley-WMF updated the task description. (Show Details)Apr 20 2026, 11:01 AM
VRiley-WMF updated the task description. (Show Details)Apr 20 2026, 1:59 PM
VRiley-WMF updated the task description. (Show Details)May 6 2026, 8:31 PM
ops-monitoring-bot added a comment.May 6 2026, 10:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1021.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 6 2026, 11:38 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1021.eqiad.wmnet with OS trixie executed with errors:

  • pc1021 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console pc1021.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.May 6 2026, 11:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1021.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 7 2026, 1:01 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1021.eqiad.wmnet with OS trixie executed with errors:

  • pc1021 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console pc1021.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.May 7 2026, 7:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1022.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 7 2026, 8:02 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1022.eqiad.wmnet with OS trixie executed with errors:

  • pc1022 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console pc1022.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.May 7 2026, 8:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1021.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 7 2026, 8:53 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1021.eqiad.wmnet with OS trixie completed:

  • pc1021 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605072034_vriley_1214325_pc1021.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.May 7 2026, 10:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1022.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 7 2026, 11:25 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1022.eqiad.wmnet with OS trixie completed:

  • pc1022 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605072309_vriley_1234532_pc1022.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)May 8 2026, 12:57 AM
ops-monitoring-bot added a comment.May 8 2026, 12:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1023.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 8 2026, 1:52 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1023.eqiad.wmnet with OS trixie completed:

  • pc1023 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605080115_vriley_1249412_pc1023.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.May 8 2026, 2:32 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host pc1024.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.May 8 2026, 3:04 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host pc1024.eqiad.wmnet with OS trixie completed:

  • pc1024 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605080249_vriley_1261031_pc1024.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF closed this task as Resolved.May 8 2026, 3:05 AM
VRiley-WMF updated the task description. (Show Details)

@Marostegui these are setup and ready to go!

Marostegui added a comment.May 8 2026, 4:58 AM

Looks good - thanks a lot!

ops-monitoring-bot added a comment.May 8 2026, 8:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host db1265.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 8 2026, 9:42 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host db1265.eqiad.wmnet with OS bookworm completed:

  • db1265 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605082124_vriley_1843772_db1265.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.May 8 2026, 10:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host db1266.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 8 2026, 11:22 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host db1266.eqiad.wmnet with OS bookworm completed:

  • db1266 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605082254_vriley_1855753_db1266.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits