Page MenuHomePhabricator
Create Task
Maniphest T406795

Q2:rack/setup/install logging-sd200[567]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of logging-sd200[567]

Hostname / Racking / Installation Details

Hostnames: logging-sd200[567]
Racking Proposal: Should not share a rack with other logging-sd host(s)
Networking Setup: # of Connections:1/2 - Speed:1G/10G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: @colewhite

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

logging-sd2005
  • Receive in system on procurement task T404785 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
logging-sd2006
  • Receive in system on procurement task T404785 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
logging-sd2007
  • Receive in system on procurement task T404785 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
installserver: fix partmain typooperations/puppetproduction+1 -1
site: initial setup for new logging-sd hostsoperations/puppetproduction+33 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT406795 Q2:rack/setup/install logging-sd200[567]

Event Timeline

RobH created this task.Oct 8 2025, 11:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2025, 11:30 PM
RobH assigned this task to colewhite.Oct 8 2025, 11:30 PM
RobH mentioned this in Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@colewhite,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH added a parent task: Unknown Object (Task).Oct 8 2025, 11:33 PM
colewhite edited projects, added Observability-Logging; removed observability.Oct 15 2025, 2:18 PM
gerritbot added a comment.Oct 27 2025, 8:14 PM

Change #1199062 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] site: add logging-sd hosts insetup

https://gerrit.wikimedia.org/r/1199062

gerritbot added a project: Patch-For-Review.Oct 27 2025, 8:14 PM
gerritbot added a comment.Oct 28 2025, 9:37 PM

Change #1199062 merged by Cwhite:

[operations/puppet@production] site: initial setup for new logging-sd hosts

https://gerrit.wikimedia.org/r/1199062

Maintenance_bot removed a project: Patch-For-Review.Oct 28 2025, 10:31 PM
gerritbot added a comment.Nov 27 2025, 2:58 PM

Change #1212164 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] installserver: fix partmain typo

https://gerrit.wikimedia.org/r/1212164

gerritbot added a project: Patch-For-Review.Nov 27 2025, 2:58 PM
gerritbot added a comment.Nov 27 2025, 3:24 PM

Change #1212164 merged by Filippo Giunchedi:

[operations/puppet@production] installserver: fix partmain typo

https://gerrit.wikimedia.org/r/1212164

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2025, 3:31 PM
Jhancock.wm claimed this task.Dec 9 2025, 9:19 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Dec 10 2025, 12:01 AM
ops-monitoring-bot added a comment.Dec 10 2025, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host logging-sd2006.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 10 2025, 5:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host logging-sd2005.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 10 2025, 5:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host logging-sd2007.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 10 2025, 5:18 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host logging-sd2006.codfw.wmnet with OS bookworm completed:

  • logging-sd2006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512101659_jhancock_3295799_logging-sd2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 10 2025, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host logging-sd2007.codfw.wmnet with OS bookworm completed:

  • logging-sd2007 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512101717_jhancock_3318763_logging-sd2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 10 2025, 6:27 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host logging-sd2005.codfw.wmnet with OS bookworm completed:

  • logging-sd2005 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512101748_jhancock_3318276_logging-sd2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Jhancock.wm closed this task as Resolved.Dec 10 2025, 6:28 PM
Jhancock.wm updated the task description. (Show Details)

@colewhite these are complete

Jhancock.wm mentioned this in T412458: root user not on newest batches of supermicro servers..Dec 11 2025, 9:09 PM
colewhite mentioned this in T413414: Put logging-sd[12]00[567] into service.Dec 22 2025, 11:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits