Page MenuHomePhabricator
Create Task
Maniphest T400485

Q1:rack/setup/install deploy2003
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of deploy2003

Hostname / Racking / Installation Details

Hostnames: deploy2003
Racking Proposal: Any rack is fine
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: Bookworm
Boot Method: Legacy BIOS
Sub-team Technical Contact: @Clement_Goubert @jasmine_

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

deploy2003
  • Receive in system on procurement task T399022 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site.pp: Fix insetup::serviceopsoperations/puppetproduction+2 -2
deploy2003: Add to site.ppoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT400485 Q1:rack/setup/install deploy2003

Event Timeline

RobH created this task.Jul 25 2025, 3:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 25 2025, 3:15 PM
RobH assigned this task to Clement_Goubert.Jul 25 2025, 3:15 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@Clement_Goubert,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

gerritbot added a comment.Jul 25 2025, 3:22 PM

Change #1172660 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] deploy2003: Add to site.pp

https://gerrit.wikimedia.org/r/1172660

gerritbot added a project: Patch-For-Review.Jul 25 2025, 3:22 PM
Maintenance_bot added a project: SRE.Jul 25 2025, 3:29 PM
gerritbot added a comment.Jul 28 2025, 8:41 AM

Change #1172660 merged by Clément Goubert:

[operations/puppet@production] deploy2003: Add to site.pp

https://gerrit.wikimedia.org/r/1172660

Clement_Goubert removed Clement_Goubert as the assignee of this task.Jul 28 2025, 8:42 AM
Clement_Goubert added a subscriber: RobH.
In T400485#11035012, @RobH wrote:

@Clement_Goubert,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

Done :)

Clement_Goubert updated the task description. (Show Details)Jul 28 2025, 8:42 AM
Clement_Goubert added a subscriber: jasmine_.
Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 9:31 AM
ops-monitoring-bot added a comment.Aug 7 2025, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm

Jhancock.wm claimed this task.Aug 7 2025, 3:48 PM
Jhancock.wm updated the task description. (Show Details)
ops-monitoring-bot added a comment.Aug 7 2025, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm executed with errors:

  • deploy2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console deploy2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 25 2025, 4:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 25 2025, 6:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm executed with errors:

  • deploy2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console deploy2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 25 2025, 7:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm

Jhancock.wm added a subscriber: Papaul.Aug 25 2025, 8:01 PM

@Papaul this one is going to fail again. looks like there might be a missmatch between hardware and the site.pp or preseed. I'm not sure which, but they both exist in those files.

[32/50, retrying in 96.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title deploy2003 not found yet

Jhancock.wm updated the task description. (Show Details)Aug 25 2025, 8:01 PM
ops-monitoring-bot added a comment.Aug 25 2025, 8:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm executed with errors:

  • deploy2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console deploy2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Papaul added a comment.Aug 25 2025, 9:07 PM

@Jhancock.wm no entry on the wrong puppet server for this server. Please check site.pp. Thanks

Jhancock.wm added a comment.Aug 26 2025, 12:33 PM

it's on site.pp and the pressed

Jhancock.wm added a comment.Aug 29 2025, 2:20 PM

@Clement_Goubert i think their might be a mismatch regarding the site.pp on this one. I am not sure exactly what it is. we had a similar issue in T400195. Could you check it for me when you have a chance? thanks!

gerritbot added a comment.Aug 29 2025, 2:39 PM

Change #1183124 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] site.pp: Fix insetup::serviceops

https://gerrit.wikimedia.org/r/1183124

gerritbot added a project: Patch-For-Review.Aug 29 2025, 2:39 PM

Change #1183124 merged by Clément Goubert:

[operations/puppet@production] site.pp: Fix insetup::serviceops

https://gerrit.wikimedia.org/r/1183124

Clement_Goubert added a comment.Aug 29 2025, 2:42 PM
In T400485#11132084, @Jhancock.wm wrote:

@Clement_Goubert i think their might be a mismatch regarding the site.pp on this one. I am not sure exactly what it is. we had a similar issue in T400195. Could you check it for me when you have a chance? thanks!

Just merged CR should fix it

Maintenance_bot removed a project: Patch-For-Review.Aug 29 2025, 3:31 PM
ops-monitoring-bot added a comment.Aug 29 2025, 4:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host deploy2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2025, 4:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host deploy2003.codfw.wmnet with OS bookworm executed with errors:

  • deploy2003 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console deploy2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
RobH unsubscribed.Aug 29 2025, 5:13 PM
ops-monitoring-bot added a comment.Sep 2 2025, 1:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host deploy2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 2 2025, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host deploy2003.codfw.wmnet with OS bookworm completed:

  • deploy2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509021336_jhancock_2078355_deploy2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Sep 2 2025, 1:56 PM
Jhancock.wm updated the task description. (Show Details)

@Clement_Goubert @jasmine_ this is complete and ready for y'all!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits