Page MenuHomePhabricator
Create Task
Maniphest T414727

Q3:rack/setup/install backup20[16-20]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of backup20[16-20]

Hostname / Racking / Installation Details

Hostnames: backup20[16-20] (backup2015 being used for expansion order)
Racking Proposal: As distributed among them as reasonable Use https://fault-tolerance.toolforge.org/map to optimize this placement.
Raid Setup: The 2 SSDs as 2 standalone disks (or 2 different virtual disks RAID 0, it is the same). Leave the HDs unconfigured, they will be setup post-install by the data persistence team. Ideally, the SSDs as the first virtual disks, for the partman recipe to work as intended.
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: trixie
Boot Method: UEFI.
Sub-team Technical Contact: Jaime Crespo

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

backup2016
  • Receive in system on procurement task T412434 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
backup2017
  • Receive in system on procurement task T412434 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
backup2018
  • Receive in system on procurement task T412434 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
backup2019
  • Receive in system on procurement task T412434 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
backup2020
  • Receive in system on procurement task T412434 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
backups: Fix error on domain for backup hostsoperations/puppetproduction+2 -2
backup: Set up backup1015-backup1020 & backup2015-backup2020operations/puppetproduction+17 -0
install: Reimage with format backup1015-backup1020 & backup2015-backup2020operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT414727 Q3:rack/setup/install backup20[16-20]

Event Timeline

RobH created this task.Jan 15 2026, 8:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 15 2026, 8:31 PM
RobH mentioned this in Unknown Object (Task).Jan 15 2026, 8:31 PM
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to jcrespo.Jan 15 2026, 8:36 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

Jaime,

I had to split up the expansion and refresh budget lines for backup this quarter, so this racking task (and its parent order task) only covers the line item for refreshing backup hosts in this site (not expansion, expansion racking task was already filed via T414724.

I've taken a guess at the next available backup hostname, but this needs your update in the task description on rack preference plus a double check of the info I've provided.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

gerritbot added a comment.Jan 16 2026, 12:54 PM

Change #1227773 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] install: Reimage with format backup1015-backup1020 & backup2015-backup2020

https://gerrit.wikimedia.org/r/1227773

gerritbot added a project: Patch-For-Review.Jan 16 2026, 12:55 PM
gerritbot added a comment.Jan 16 2026, 12:58 PM

Change #1227773 merged by Jcrespo:

[operations/puppet@production] install: Reimage with format backup1015-backup1020 & backup2015-backup2020

https://gerrit.wikimedia.org/r/1227773

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2026, 1:31 PM
jcrespo updated the task description. (Show Details)Jan 16 2026, 1:32 PM
gerritbot added a comment.Jan 16 2026, 2:00 PM

Change #1227792 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backup: Set up backup1015-backup1020 & backup2015-backup2020

https://gerrit.wikimedia.org/r/1227792

gerritbot added a project: Patch-For-Review.Jan 16 2026, 2:00 PM
gerritbot added a comment.Jan 19 2026, 8:49 AM

Change #1227792 merged by Jcrespo:

[operations/puppet@production] backup: Set up backup1015-backup1020 & backup2015-backup2020

https://gerrit.wikimedia.org/r/1227792

jcrespo updated the task description. (Show Details)Jan 19 2026, 9:24 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 19 2026, 9:30 AM
jcrespo updated the task description. (Show Details)Feb 3 2026, 5:15 PM
jcrespo removed jcrespo as the assignee of this task.Feb 3 2026, 5:17 PM
jcrespo subscribed.
jcrespo updated the task description. (Show Details)Feb 4 2026, 1:58 PM
ops-monitoring-bot added a comment.Tue, Feb 17, 5:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2016.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Tue, Feb 17, 5:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2017.codfw.wmnet with OS trixie

Jhancock.wm updated the task description. (Show Details)Tue, Feb 17, 6:15 PM
Jhancock.wm subscribed.

@jcrespo i need an edit to the site.pp file. the backup20XX servers have eqiad in the name. they should be codfw. Thank you!

ops-monitoring-bot added a comment.Tue, Feb 17, 6:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2016.codfw.wmnet with OS trixie executed with errors:

  • backup2016 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console backup2016.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Feb 17, 6:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2017.codfw.wmnet with OS trixie executed with errors:

  • backup2017 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console backup2017.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Wed, Feb 18, 7:59 AM

Change #1240203 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] backups: Fix error on domain for backup hosts

https://gerrit.wikimedia.org/r/1240203

gerritbot added a project: Patch-For-Review.Wed, Feb 18, 7:59 AM
gerritbot added a comment.Wed, Feb 18, 8:02 AM

Change #1240203 merged by Jcrespo:

[operations/puppet@production] backups: Fix error on domain for backup hosts

https://gerrit.wikimedia.org/r/1240203

jcrespo added a comment.Wed, Feb 18, 8:03 AM
In T414727#11624536, @Jhancock.wm wrote:

@jcrespo i need an edit to the site.pp file. the backup20XX servers have eqiad in the name. they should be codfw. Thank you!

Done.

Maintenance_bot removed a project: Patch-For-Review.Wed, Feb 18, 8:31 AM
ops-monitoring-bot added a comment.Wed, Feb 18, 7:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2016.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Feb 18, 8:19 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2016.codfw.wmnet with OS trixie completed:

  • backup2016 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602181953_jhancock_2702651_backup2016.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Wed, Feb 18, 8:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2017.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Feb 18, 8:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2018.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Feb 18, 9:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2017.codfw.wmnet with OS trixie completed:

  • backup2017 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602182054_jhancock_2737703_backup2017.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Wed, Feb 18, 9:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2018.codfw.wmnet with OS trixie completed:

  • backup2018 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602182058_jhancock_2738025_backup2018.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Wed, Feb 18, 10:13 PM
ops-monitoring-bot added a comment.Thu, Feb 19, 4:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2019.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, Feb 19, 4:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host backup2020.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, Feb 19, 5:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2019.codfw.wmnet with OS trixie completed:

  • backup2019 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602191714_jhancock_3358214_backup2019.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Thu, Feb 19, 5:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host backup2020.codfw.wmnet with OS trixie completed:

  • backup2020 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602191718_jhancock_3358939_backup2020.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Thu, Feb 19, 5:43 PM
Jhancock.wm claimed this task.
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm added a comment.Thu, Feb 19, 5:51 PM

@jcrespo these are ready for you

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits