Page MenuHomePhabricator
Create Task
Maniphest T416254

Q3:rack/setup/install bast1004
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of bast1004

Hostname / Racking / Installation Details

This section should list the racking restrictions for these hosts. If they shouldn't share a rack/row with one another, or any existing hosts. This section should also list the other details listed below.

Hostnames: bast1004.wikimedia.org
Racking Proposal: Any rack is fine
Networking Setup: Public IP, these don't really need 10G (but if since have a 10G NIC and if we have a free 10G port, it won't hurt of course)
OS Distro: Trixie
Boot Method: UEFI.
Sub-team Technical Contact: Moritz or if not around anyone else from SRE IF

bast1004
  • Receive in system on procurement task T412563 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Other Assignee
Andrew
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Make bast1004 a bastionoperations/puppetproduction+2 -0
Configure UEFI partman config for bast1004operations/puppetproduction+3 -0
Add site.pp and preseed entry for bast1004operations/puppetproduction+7 -0
Customize query in gerrit

Event Timeline

Jhancock.wm created this task.Feb 2 2026, 9:55 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptFeb 2 2026, 9:55 PM
Jhancock.wm mentioned this in Unknown Object (Task).Feb 2 2026, 9:55 PM
Jclark-ctr moved this task from Backlog to Racking Tasks on the ops-eqiad board.Feb 2 2026, 9:55 PM
Maintenance_bot added a project: SRE.Feb 2 2026, 10:29 PM
Jclark-ctr claimed this task.Feb 3 2026, 11:15 AM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr reassigned this task from Jclark-ctr to Andrew.Feb 3 2026, 11:51 AM
Jclark-ctr updated Other Assignee, added: Jclark-ctr.
Jclark-ctr added subscribers: Andrew, Jclark-ctr.

@Andrew would you be able to help with adding server to Site.pp and updating preseed.yaml for efi booting?

Jclark-ctr claimed this task.Feb 3 2026, 8:46 PM
Jclark-ctr updated Other Assignee, added: Andrew; removed: Jclark-ctr.
gerritbot added a comment.Feb 4 2026, 3:13 PM

Change #1236765 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Add site.pp entry for bast1004

https://gerrit.wikimedia.org/r/1236765

gerritbot added a project: Patch-For-Review.Feb 4 2026, 3:13 PM
MoritzMuehlenhoff updated the task description. (Show Details)Feb 4 2026, 3:27 PM
gerritbot added a comment.Feb 4 2026, 3:28 PM

Change #1236767 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Configure UEFI partman config for bast1004

https://gerrit.wikimedia.org/r/1236767

ops-monitoring-bot added a comment.Feb 4 2026, 3:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host bast1004.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Feb 4 2026, 3:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host bast1004.eqiad.wmnet with OS trixie executed with errors:

  • bast1004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console bast1004.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
MoritzMuehlenhoff updated the task description. (Show Details)Feb 4 2026, 3:39 PM
gerritbot added a comment.Feb 4 2026, 3:53 PM

Change #1236765 merged by Andrew Bogott:

[operations/puppet@production] Add site.pp and preseed entry for bast1004

https://gerrit.wikimedia.org/r/1236765

gerritbot added a comment.Feb 4 2026, 3:56 PM

Change #1236767 abandoned by Muehlenhoff:

[operations/puppet@production] Configure UEFI partman config for bast1004

Reason:

Duplicated

https://gerrit.wikimedia.org/r/1236767

ops-monitoring-bot added a comment.Feb 4 2026, 4:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host bast1004.wikimedia.org with OS trixie

Jclark-ctr updated the task description. (Show Details)Feb 4 2026, 4:18 PM
ops-monitoring-bot added a comment.Feb 4 2026, 4:18 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host bast1004.wikimedia.org with OS trixie executed with errors:

  • bast1004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console bast1004.wikimedia.org" to get a root shell, but depending on the failure this may not work.
Maintenance_bot removed a project: Patch-For-Review.Feb 4 2026, 4:30 PM
ops-monitoring-bot added a comment.Feb 4 2026, 5:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host bast1004.wikimedia.org with OS trixie

ops-monitoring-bot added a comment.Feb 4 2026, 5:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host bast1004.wikimedia.org with OS trixie completed:

  • bast1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602041724_jclark_2648072_bast1004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Jclark-ctr closed this task as Resolved.Feb 4 2026, 7:22 PM
Jclark-ctr updated the task description. (Show Details)
gerritbot added a comment.Mon, Feb 9, 2:22 PM

Change #1237916 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make bast1004 a bastion

https://gerrit.wikimedia.org/r/1237916

gerritbot added a project: Patch-For-Review.Mon, Feb 9, 2:22 PM
gerritbot added a comment.Mon, Feb 9, 4:23 PM

Change #1237916 merged by Muehlenhoff:

[operations/puppet@production] Make bast1004 a bastion

https://gerrit.wikimedia.org/r/1237916

Maintenance_bot removed a project: Patch-For-Review.Mon, Feb 9, 4:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits