Page MenuHomePhabricator
Create Task
Maniphest T405966

Q2:rack/setup/install ganeti-jumbo100[1-3]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ganeti-jumbo100[1-3]

Hostname / Racking / Installation Details

Hostnames: ganeti-jumbo100[1-3]
Racking Proposal: Different rack per host
Networking Setup: # of Connections:1/2 - Speed:1G/10G. - VLAN:Private/Public/Other(Specify) : 1 10G/private network
OS Distro: Trixie
Boot Method: UEFI
Sub-team Technical Contact: EU hours: Ben Tullis (@BTullis, btullis on IRC) US hours: Brian King (@bking, inflatador on IRC)

Per host setup checklist

ganeti-jumbo1001
  • Receive in system on procurement task T404778 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti-jumbo1002
  • Receive in system on procurement task T404778 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ganeti-jumbo1003
  • Receive in system on procurement task T404778 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
ganeti-jumbo: Add hosts and partman recipeoperations/puppetproduction+114 -0
site.pp: Add ganeti-jumbo hostsoperations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT410314 ☂️ Stand up ganeti-jumbo clusters ☂️
Unknown Object (Task)
 ResolvedJclark-ctrT405966 Q2:rack/setup/install ganeti-jumbo100[1-3]

Event Timeline

RobH created this task.Sep 29 2025, 7:30 PM
RobH assigned this task to bking.
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.

@bking,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH mentioned this in Unknown Object (Task).Sep 29 2025, 7:31 PM
RobH added a parent task: Unknown Object (Task).
Maintenance_bot added a project: SRE.Sep 29 2025, 8:29 PM
Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Sep 30 2025, 1:00 PM
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE.
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
Gehel added a project: Essential-Work.Sep 30 2025, 1:03 PM
Gehel edited projects, added Data-Platform-SRE (2025.10.17 - 2025.11.07); removed Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:50 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.Oct 17 2025, 8:51 AM
Gehel moved this task from Backlog - operations to Quick Wins on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.Oct 17 2025, 1:51 PM
gerritbot added a comment.Oct 17 2025, 3:35 PM

Change #1196935 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] site.pp: Add ganeti-jumbo hosts

https://gerrit.wikimedia.org/r/1196935

gerritbot added a project: Patch-For-Review.Oct 17 2025, 3:35 PM
gerritbot added a comment.Oct 17 2025, 5:44 PM

Change #1196952 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] ganeti-jumbo: Add hosts and partman recipe

https://gerrit.wikimedia.org/r/1196952

gerritbot added a comment.Oct 17 2025, 5:48 PM

Change #1196935 abandoned by Bking:

[operations/puppet@production] site.pp: Add ganeti-jumbo hosts

Reason:

superseded by 1196952

https://gerrit.wikimedia.org/r/1196935

gerritbot added a comment.Oct 20 2025, 5:08 PM

Change #1196952 merged by Bking:

[operations/puppet@production] ganeti-jumbo: Add hosts and partman recipe

https://gerrit.wikimedia.org/r/1196952

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2025, 5:31 PM
bking changed the task status from Open to In Progress.Oct 27 2025, 1:56 PM
bking moved this task from Quick Wins to In Progress on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.
bking changed the task status from In Progress to Stalled.Oct 27 2025, 1:59 PM
bking mentioned this in T405964: Q2:rack/setup/install ganeti-jumbo200[1-3].
bking moved this task from In Progress to Blocked/Waiting on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.

Hello DC Ops,

I've added the hosts and partman recipe to Puppet as requested. Please note that the partman recipe is untested, so if the reimage fails more than once, please ping me in IRC (inflatador) and/or assign this task to me. The same goes for T405964, as the hosts in that ticket are identical to these.

Thanks for your help!

Gehel edited projects, added Data-Platform-SRE (2025.11.07 - 2025.11.28); removed Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:30 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
bking mentioned this in T409769: Hardware requirements for WDQS backend migration..Nov 14 2025, 6:56 PM
bking mentioned this in T410317: Stand up ganeti-jumbo cluster in EQIAD.Nov 17 2025, 7:44 PM
bking added a parent task: T410314: ☂️ Stand up ganeti-jumbo clusters ☂️.Nov 17 2025, 7:46 PM
bking removed bking as the assignee of this task.Nov 20 2025, 2:47 PM
Jclark-ctr claimed this task.Dec 11 2025, 12:50 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr updated the task description. (Show Details)Dec 11 2025, 3:05 PM
ops-monitoring-bot added a comment.Dec 11 2025, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ganeti-jumbo1001.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 11 2025, 7:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ganeti-jumbo1002.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 11 2025, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ganeti-jumbo1001.eqiad.wmnet with OS trixie completed:

  • ganeti-jumbo1001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512112009_jclark_3604593_ganeti-jumbo1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm mentioned this in T412458: root user not on newest batches of supermicro servers..Dec 11 2025, 9:09 PM
ops-monitoring-bot added a comment.Dec 11 2025, 9:14 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ganeti-jumbo1002.eqiad.wmnet with OS trixie executed with errors:

  • ganeti-jumbo1002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ganeti-jumbo1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 12 2025, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ganeti-jumbo1002.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 12 2025, 2:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host ganeti-jumbo1003.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Dec 12 2025, 2:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ganeti-jumbo1003.eqiad.wmnet with OS trixie completed:

  • ganeti-jumbo1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512121418_jclark_4024268_ganeti-jumbo1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 12 2025, 2:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host ganeti-jumbo1002.eqiad.wmnet with OS trixie completed:

  • ganeti-jumbo1002 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512121404_jclark_4010144_ganeti-jumbo1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Jclark-ctr closed this task as Resolved.Dec 12 2025, 2:52 PM
Jclark-ctr updated the task description. (Show Details)
Gehel moved this task from Blocked/Waiting to Done on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.Dec 19 2025, 9:22 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.Dec 19 2025, 9:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits