Page MenuHomePhabricator
Create Task
Maniphest T400213

Q1:rack/setup/install db224[5-8]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db224[5-8]

Hostname / Racking / Installation Details

Hostnames: db2245.codfw.wmnet db2246.codfw.wmnet db2247.codfw.wmnet db2248.codfw.wmnet
Racking Proposal: Scattered across rows
Networking Setup: # of Connections:1 - Speed:1G. - VLAN:Private
OS Distro: Bookworm
Boot Method: Legacy BIOS
Sub-team Technical Contact: @Marostegui

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

db2245
  • Receive in system on procurement task T398507 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db2246
  • Receive in system on procurement task T398507 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db2247
  • Receive in system on procurement task T398507 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db2248 *this is the 1CPU server*
  • Receive in system on procurement task T398507 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site.pp: Add reference taskoperations/puppetproduction+1 -1
mariadb: Add db224[5-8]operations/puppetproduction+14 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT400213 Q1:rack/setup/install db224[5-8]
 Resolved• MarosteguiT406551 Productionize db224[5-8]

Event Timeline

RobH created this task.Jul 22 2025, 8:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2025, 8:17 PM
RobH assigned this task to • Marostegui.Jul 22 2025, 8:18 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH unsubscribed.Jul 22 2025, 8:19 PM
Maintenance_bot added a project: SRE.Jul 22 2025, 8:29 PM
gerritbot added a comment.Jul 24 2025, 1:19 PM

Change #1172319 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Add db224[5-8]

https://gerrit.wikimedia.org/r/1172319

gerritbot added a project: Patch-For-Review.Jul 24 2025, 1:19 PM
gerritbot added a comment.Jul 24 2025, 1:22 PM

Change #1172319 merged by Marostegui:

[operations/puppet@production] mariadb: Add db224[5-8]

https://gerrit.wikimedia.org/r/1172319

• Marostegui added a comment.Jul 24 2025, 1:23 PM

Patches are done

• Marostegui updated the task description. (Show Details)Jul 24 2025, 1:23 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 24 2025, 1:31 PM
gerritbot added a comment.Jul 29 2025, 6:28 AM

Change #1173622 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add reference task

https://gerrit.wikimedia.org/r/1173622

gerritbot added a project: Patch-For-Review.Jul 29 2025, 6:28 AM

Change #1173622 merged by Marostegui:

[operations/puppet@production] site.pp: Add reference task

https://gerrit.wikimedia.org/r/1173622

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2025, 6:30 AM
Jhancock.wm claimed this task.Aug 6 2025, 12:58 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm added a comment.Aug 6 2025, 1:01 PM

db2245 didn't pass provision. will investigate.

ops-monitoring-bot added a comment.Aug 6 2025, 6:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2245.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 6 2025, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2246.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 6 2025, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2247.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 6 2025, 8:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2245.codfw.wmnet with OS bookworm executed with errors:

  • db2245 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console db2245.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 6 2025, 8:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2246.codfw.wmnet with OS bookworm executed with errors:

  • db2246 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console db2246.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 6 2025, 8:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2247.codfw.wmnet with OS bookworm executed with errors:

  • db2247 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console db2247.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Jhancock.wm added a subscriber: Papaul.Aug 6 2025, 8:30 PM

The curse is strong today. @Papaul 2245-2247 did the wrong puppet thing. I did ask John and Valerie about it while you were out and neither of them are getting the error. I'm starting to think it's a me thing. Could you delete them for now? and we can try to figure out why later.

Jhancock.wm updated the task description. (Show Details)Aug 6 2025, 8:30 PM
Papaul added a comment.Aug 7 2025, 12:45 AM

@Jhancock.wm done

ops-monitoring-bot added a comment.Aug 7 2025, 12:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2245.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 7 2025, 12:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2246.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 7 2025, 12:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2247.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 7 2025, 12:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2245.codfw.wmnet with OS bookworm completed:

  • db2245 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508071228_jhancock_990110_db2245.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 7 2025, 12:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2246.codfw.wmnet with OS bookworm completed:

  • db2246 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508071232_jhancock_990150_db2246.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 7 2025, 12:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2247.codfw.wmnet with OS bookworm completed:

  • db2247 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508071237_jhancock_990173_db2247.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 7 2025, 1:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host db2248.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 7 2025, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host db2248.codfw.wmnet with OS bookworm completed:

  • db2248 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508071330_jhancock_1002319_db2248.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Aug 7 2025, 1:50 PM
Jhancock.wm updated the task description. (Show Details)

@Marostegui these are complete

• Marostegui closed subtask T406551: Productionize db224[5-8] as Resolved.Oct 21 2025, 11:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits