Page MenuHomePhabricator
Create Task
Maniphest T389634

Q4:rack/setup/install thanos-fe200[5-7]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of thanos-fe200[5-7]

Hostname / Racking / Installation Details

Hostnames: thanos-fe200[5-7]
Racking Proposal: Not in the same racks as each other, nor in C7 (which has thanos-fe2004); it's OK if they're in a rack that has thanos-fe200[1-3] in as, as they're replacing those nodes.
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private :
OS Distro: Bullseye
Sub-team Technical Contact: @MatthewVernon

Per host setup checklist

thanos-fe2005
  • Receive in system on procurement task T389500 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
thanos-fe2006
  • Receive in system on procurement task T389500 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
thanos-fe2007
  • Receive in system on procurement task T389500 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Thanos: add new thanos-fe200[5-7] nodesoperations/puppetproduction+12 -0
site/install: prep for new apus and thanos nodesoperations/puppetproduction+11 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT389634 Q4:rack/setup/install thanos-fe200[5-7]

Event Timeline

RobH created this task.Mar 21 2025, 4:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 21 2025, 4:03 PM
RobH assigned this task to MatthewVernon.Mar 21 2025, 4:03 PM
RobH mentioned this in Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.

Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving the new servers. This is due to the majority of DC Ops not having root/merge puppet rights.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-sites will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

Maintenance_bot added a project: SRE.Mar 21 2025, 4:29 PM
gerritbot added a comment.Mar 21 2025, 4:43 PM

Change #1130151 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] site/install: prep for new apus and thanosnodes

https://gerrit.wikimedia.org/r/1130151

gerritbot added a project: Patch-For-Review.Mar 21 2025, 4:43 PM
MatthewVernon removed MatthewVernon as the assignee of this task.Mar 21 2025, 4:48 PM

Puppet work done, unassigning myself.

gerritbot added a comment.Mar 24 2025, 11:25 AM

Change #1130151 merged by MVernon:

[operations/puppet@production] site/install: prep for new apus and thanos nodes

https://gerrit.wikimedia.org/r/1130151

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2025, 11:31 AM
Jhancock.wm mentioned this in T390578: Q4:rack/setup/install apus-fe2003.Mar 31 2025, 4:02 PM
Jhancock.wm claimed this task.Mar 31 2025, 4:52 PM
Jhancock.wm updated the task description. (Show Details)
MatthewVernon added a project: SRE-swift-storage.Apr 1 2025, 9:15 AM
ops-monitoring-bot added a comment.Apr 1 2025, 7:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host thanos-fe2005.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 1 2025, 7:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host thanos-fe2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 1 2025, 7:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host thanos-fe2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 1 2025, 8:29 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host thanos-fe2005.codfw.wmnet with OS bullseye completed:

  • thanos-fe2005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504011958_jhancock_1779940_thanos-fe2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 1 2025, 8:29 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host thanos-fe2007.codfw.wmnet with OS bullseye completed:

  • thanos-fe2007 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504012001_jhancock_1781048_thanos-fe2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Apr 1 2025, 8:30 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host thanos-fe2006.codfw.wmnet with OS bullseye completed:

  • thanos-fe2006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504012011_jhancock_1780225_thanos-fe2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Apr 1 2025, 8:31 PM
Jhancock.wm updated the task description. (Show Details)

@MatthewVernon all yours!

gerritbot added a comment.Apr 4 2025, 1:07 PM

Change #1134221 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] Thanos: add new thanos-fe200[5-7] nodes

https://gerrit.wikimedia.org/r/1134221

gerritbot added a project: Patch-For-Review.Apr 4 2025, 1:07 PM
gerritbot added a comment.Apr 8 2025, 10:16 AM

Change #1134221 merged by MVernon:

[operations/puppet@production] Thanos: add new thanos-fe200[5-7] nodes

https://gerrit.wikimedia.org/r/1134221

ops-monitoring-bot added a comment.Apr 8 2025, 10:26 AM

Host rebooted by mvernon@cumin2002 with reason: reboot before bringing into service

Maintenance_bot removed a project: Patch-For-Review.Apr 8 2025, 10:30 AM
ops-monitoring-bot added a comment.Apr 8 2025, 10:32 AM

Host rebooted by mvernon@cumin2002 with reason: reboot before bringing into service

ops-monitoring-bot added a comment.Apr 8 2025, 10:41 AM

Host rebooted by mvernon@cumin2002 with reason: reboot before bringing into service

Stashbot added a comment.Apr 8 2025, 11:00 AM

Mentioned in SAL (#wikimedia-operations) [2025-04-08T11:00:47Z] <Emperor> pool thanos-fe200[5-7] T389634

MatthewVernon mentioned this in T391352: Q4 Thanos hardware refresh.Apr 8 2025, 2:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits