Page MenuHomePhabricator
Create Task
Maniphest T380083

Q2:rack/setup/install db125[0-4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db125[0-4]

Hostname / Racking / Installation Details

Hostnames: db1250 db1251 db1252 db1253 db1254
Racking Proposal: Let's do one per row.
Networking Setup: # of Connections:1/2 - Speed:1G. - VLAN:Private/Public/Other(Specify) : Reminder no IPV6 DNS resolution for these hosts
OS Distro: Bookworm (default unless otherwise specified)
Sub-team Technical Contact: @Marostegui

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

db1250
  • Receive in system on procurement task T379371 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db1251
  • Receive in system on procurement task T379371 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db1252
  • Receive in system on procurement task T379371 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db1253
  • Receive in system on procurement task T379371 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
db1254
  • Receive in system on procurement task T379371 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mariadb: Set db125[0-4] insetupoperations/puppetproduction+18 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT380083 Q2:rack/setup/install db125[0-4]
 ResolvedFCeratto-WMFT385141 Productionize db125[0-4]
 ResolvedMarosteguiT388024 Switch m1 master db1164 -> db1250

Event Timeline

RobH created this task.Nov 15 2024, 5:45 PM
RobH assigned this task to Marostegui.
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.

Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving the new servers. This is due to the majority of DC Ops not having root/merge puppet rights.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-sites will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH added a parent task: Unknown Object (Task).Nov 15 2024, 5:47 PM
RobH mentioned this in Unknown Object (Task).
Maintenance_bot added a project: SRE.Nov 15 2024, 6:29 PM
gerritbot added a comment.Dec 3 2024, 12:27 PM

Change #1100096 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Set db125[0-4] insetup

https://gerrit.wikimedia.org/r/1100096

gerritbot added a project: Patch-For-Review.Dec 3 2024, 12:27 PM
gerritbot added a comment.Dec 3 2024, 12:30 PM

Change #1100096 merged by Marostegui:

[operations/puppet@production] mariadb: Set db125[0-4] insetup

https://gerrit.wikimedia.org/r/1100096

Marostegui removed Marostegui as the assignee of this task.Dec 3 2024, 12:30 PM
Marostegui added a subscriber: RobH.
In T380083#10327817, @RobH wrote:

Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team receiving the new servers. This is due to the majority of DC Ops not having root/merge puppet rights.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-sites will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

Done

Marostegui updated the task description. (Show Details)Dec 3 2024, 12:31 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 3 2024, 12:31 PM
RobH unsubscribed.Dec 3 2024, 1:28 PM
VRiley-WMF updated the task description. (Show Details)Jan 29 2025, 3:01 PM
VRiley-WMF updated the task description. (Show Details)Jan 29 2025, 9:38 PM
ops-monitoring-bot added a comment.Jan 29 2025, 9:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host db1250.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 29 2025, 9:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host db1251.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 30 2025, 6:35 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host db1250.eqiad.wmnet with OS bookworm completed:

  • db1250 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501292202_vriley_3474746_db1250.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 30 2025, 6:35 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host db1251.eqiad.wmnet with OS bookworm completed:

  • db1251 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501292210_vriley_3475673_db1251.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
VRiley-WMF updated the task description. (Show Details)Jan 30 2025, 6:36 AM
Marostegui changed the status of subtask T385141: Productionize db125[0-4] from Open to Stalled.Jan 30 2025, 8:01 AM
VRiley-WMF subscribed.Jan 30 2025, 3:36 PM

db1250
A5
U29
CableID 1953
Port 32

db1251
B5
U10
CableID 3795
Port 11

db1252
C3
U28
CableID 4021
Port 15

db1253
D1
U30
CableID 5171
Port 32

db1254
F5
U36
CableID 5-02595
Port 15

Papaul updated the task description. (Show Details)Jan 30 2025, 9:04 PM
ops-monitoring-bot added a comment.Jan 30 2025, 11:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host db1252.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 31 2025, 12:05 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host db1252.eqiad.wmnet with OS bookworm completed:

  • db1252 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501302347_pt1979_3716977_db1252.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 31 2025, 12:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host db1253.eqiad.wmnet with OS bookworm

Papaul updated the task description. (Show Details)Jan 31 2025, 12:09 AM
Papaul updated the task description. (Show Details)Jan 31 2025, 12:32 AM
ops-monitoring-bot added a comment.Jan 31 2025, 12:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1002 for host db1254.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 31 2025, 12:42 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host db1253.eqiad.wmnet with OS bookworm completed:

  • db1253 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501310025_pt1979_3724784_db1253.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Jan 31 2025, 1:12 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1002 for host db1254.eqiad.wmnet with OS bookworm completed:

  • db1254 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501310053_pt1979_3728849_db1254.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul closed this task as Resolved.Jan 31 2025, 1:13 AM
Papaul claimed this task.
Papaul updated the task description. (Show Details)
Papaul subscribed.

@Marostegui this is complete

Marostegui added a comment.Jan 31 2025, 6:30 AM

Thank you!

Marostegui changed the status of subtask T385141: Productionize db125[0-4] from Stalled to Open.Jan 31 2025, 6:30 AM
FCeratto-WMF closed subtask T385141: Productionize db125[0-4] as Resolved.Mar 7 2025, 7:17 AM
FCeratto-WMF reopened subtask T385141: Productionize db125[0-4] as In Progress.Mar 10 2025, 9:16 AM
FCeratto-WMF closed subtask T385141: Productionize db125[0-4] as Resolved.Mar 11 2025, 1:03 PM
Ladsgroup reopened subtask T385141: Productionize db125[0-4] as Open.Jun 12 2025, 11:16 PM
FCeratto-WMF closed subtask T385141: Productionize db125[0-4] as Resolved.Jun 25 2025, 10:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits