Page MenuHomePhabricator
Create Task
Maniphest T400637

Q1:rack/setup/install maps201[1-4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of maps201[1-4]

Hostname / Racking / Installation Details

Hostnames: maps201[1-4]
Racking Proposal: Ideally we can have one in each rack, which ones in particular doesn't matter
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: Bookworm
Boot Method: UEFI
Sub-team Technical Contact: Moritz and/or Luca

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

maps2011
  • Receive in system on procurement task T398372 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
maps2012
  • Receive in system on procurement task T398372 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
maps2013
  • Receive in system on procurement task T398372 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
maps2014
  • Receive in system on procurement task T398372 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
fix partman/standard-efi.cfg for maps[1|2]01-4Jclark-ctroperations/puppetproduction+2 -2
Add maps201[1-4] to site.pp and preseed configMuehlenhoffoperations/puppetproduction+7 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT400637 Q1:rack/setup/install maps201[1-4]

Event Timeline

RobH created this task.Jul 28 2025, 6:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2025, 6:32 PM
RobH mentioned this in Unknown Object (Task).Jul 28 2025, 6:32 PM
RobH updated the task description. (Show Details)
RobH added a parent task: Unknown Object (Task).
RobH updated the task description. (Show Details)Jul 28 2025, 6:34 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH assigned this task to • joanna_borun.Jul 28 2025, 6:36 PM

Joanna,

As the order approval is still pending on order task T398372, I've gone ahead and filed the racking task here and now it needs to be updated by someone within ServiceOps to reality check the hostnames I populated, provide racking details, and provide a technical point of contact.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH unsubscribed.Jul 28 2025, 6:36 PM
RobH mentioned this in Unknown Object (Task).Jul 28 2025, 6:47 PM
Maintenance_bot added a project: SRE.Jul 28 2025, 7:29 PM
wiki_willy reassigned this task from • joanna_borun to MoritzMuehlenhoff.Aug 11 2025, 6:22 PM
wiki_willy added subscribers: MoritzMuehlenhoff, wiki_willy.

Hi @MoritzMuehlenhoff - are you able to help confirm the racking details and update site.pp on this one? Thanks, Willy

MoritzMuehlenhoff updated the task description. (Show Details)Aug 12 2025, 9:54 AM
gerritbot added a comment.Aug 12 2025, 9:57 AM

Change #1177959 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add maps201[1-4] to site.pp and preseed config

https://gerrit.wikimedia.org/r/1177959

gerritbot added a project: Patch-For-Review.Aug 12 2025, 9:57 AM
gerritbot added a comment.Aug 12 2025, 10:06 AM

Change #1177959 merged by Muehlenhoff:

[operations/puppet@production] Add maps201[1-4] to site.pp and preseed config

https://gerrit.wikimedia.org/r/1177959

MoritzMuehlenhoff removed MoritzMuehlenhoff as the assignee of this task.Aug 12 2025, 10:28 AM
In T400637#11075140, @wiki_willy wrote:

Hi @MoritzMuehlenhoff - are you able to help confirm the racking details and update site.pp on this one? Thanks, Willy

Sure thing, I've filled in the necessary details and merged the Puppet changes.

Maintenance_bot removed a project: Patch-For-Review.Aug 12 2025, 10:31 AM
wiki_willy added a comment.Aug 12 2025, 4:37 PM

Thanks @MoritzMuehlenhoff!

Jhancock.wm claimed this task.Aug 27 2025, 5:28 PM
Jhancock.wm updated the task description. (Show Details)
ops-monitoring-bot added a comment.Aug 27 2025, 6:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 27 2025, 6:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2012.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 27 2025, 6:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2013.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 27 2025, 6:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2014.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 27 2025, 6:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm executed with errors:

  • maps2011 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console maps2011.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Jhancock.wm updated the task description. (Show Details)Aug 27 2025, 8:36 PM

ran into an issue where the installer thinks that there is a raid when the config does not call for one. need to go back and compare and check for issues.

ops-monitoring-bot added a comment.Aug 28 2025, 10:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host maps2011.codfw.wmnet with OS bookworm

Jclark-ctr subscribed.Aug 28 2025, 10:30 PM

@Jhancock.wm Hey Jen, I was looking at Puppet and here’s what I noticed:

Currently, it has:

'maps201[1-4]':

  • partman/standard.cfg
  • partman/raid10-4dev-efi.cfg

I believe it should instead use partman/standard-efi.cfg for UEFI booting, like the other servers with partman/raid10-4dev-efi.cfg:

'wdqs1025':

  • partman/standard-efi.cfg
gerritbot added a comment.Aug 28 2025, 10:38 PM

Change #1182947 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] fix partman/standard-efi.cfg for maps[1|2]01-4

https://gerrit.wikimedia.org/r/1182947

gerritbot added a project: Patch-For-Review.Aug 28 2025, 10:38 PM
gerritbot added a comment.Aug 28 2025, 10:42 PM

Change #1182947 merged by RobH:

[operations/puppet@production] fix partman/standard-efi.cfg for maps[1|2]01-4

https://gerrit.wikimedia.org/r/1182947

ops-monitoring-bot added a comment.Aug 28 2025, 11:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host maps2012.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Aug 28 2025, 11:30 PM
Jclark-ctr added a subscriber: • elukey.Aug 29 2025, 12:59 PM

@Jhancock.wm These servers are still failing to image. @elukey @MoritzMuehlenhoff , could the older maps* entries in preseed.yaml be causing the issue?

'maps101[1-4]':
  - partman/standard-efi.cfg
  - partman/raid10-4dev-efi.cfg
'maps201[1-4]':
  - partman/standard-efi.cfg
  - partman/raid10-4dev-efi.cfg
'maps*':
  - partman/standard.cfg
  - partman/hwraid-1dev.cfg
MoritzMuehlenhoff added a comment.Aug 29 2025, 1:26 PM

Can you please retry? I fixed a problem with the codfw install server setup earlier the (Euro) day, so it might simply work now.

MoritzMuehlenhoff added a comment.Aug 29 2025, 2:11 PM
In T400637#11131765, @Jclark-ctr wrote:

@Jhancock.wm These servers are still failing to image. @elukey @MoritzMuehlenhoff , could the older maps* entries in preseed.yaml be causing the issue?

The order looks alright, these are match first, maps* is only selected for any server not named [12]001[1234]. Does it reach the Debian installer and then fail there? Maybe kick off a reimage and then we can investigate the still-running system?

ops-monitoring-bot added a comment.Aug 29 2025, 2:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm

Jclark-ctr added a comment.EditedAug 29 2025, 2:38 PM

Mine would not even reach Debian installer yesterday

ops-monitoring-bot added a comment.Aug 29 2025, 2:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host maps2012.codfw.wmnet with OS bookworm

Jclark-ctr added a comment.Aug 29 2025, 2:46 PM

2012 looks to be working now

ops-monitoring-bot added a comment.Aug 29 2025, 3:21 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host maps2012.codfw.wmnet with OS bookworm completed:

  • maps2012 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508291503_jclark_1056769_maps2012.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 29 2025, 3:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host maps2013.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2025, 3:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host maps2014.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2025, 3:54 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host maps2011.codfw.wmnet with OS bookworm executed with errors:

  • maps2011 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console maps2011.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 29 2025, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host maps2011.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 29 2025, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host maps2013.codfw.wmnet with OS bookworm completed:

  • maps2013 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508291608_jhancock_1159904_maps2013.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 29 2025, 4:31 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host maps2014.codfw.wmnet with OS bookworm completed:

  • maps2014 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508291612_jhancock_1160852_maps2014.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 29 2025, 4:45 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host maps2011.codfw.wmnet with OS bookworm completed:

  • maps2011 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508291627_jhancock_1190807_maps2011.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Aug 29 2025, 4:46 PM
Jhancock.wm updated the task description. (Show Details)

@MoritzMuehlenhoff looks like we got everything cleared up. all yours!

MoritzMuehlenhoff added a comment.Aug 29 2025, 5:55 PM

Thanks!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits