Page MenuHomePhabricator
Create Task
Maniphest T412229

Q2:rack/setup/install mwlog2003
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of mwlog2003

Hostname / Racking / Installation Details

Hostnames: mwlog2003
Racking Proposal: no preference
Networking Setup: # of Connections:1 - Speed:10G (only 1G needed but 10G fine). - VLAN:Private
OS Distro: Bookworm
Boot Method: BIOS
Sub-team Technical Contact: herron

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

mwlog2003:
  • Receive in system on procurement task T404792 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove mwlog2003 so it can use the default partmanoperations/puppetproduction+0 -3
Add mwlog2003 to preseed.yaml for inital installoperations/puppetproduction+3 -0
assign mwlog[12]003 insetup roleoperations/puppetproduction+5 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT412229 Q2:rack/setup/install mwlog2003

Event Timeline

RobH created this task.Dec 10 2025, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 10 2025, 3:17 PM
RobH mentioned this in Unknown Object (Task).Dec 10 2025, 3:18 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH updated the task description. (Show Details)
RobH unsubscribed.
Maintenance_bot added a project: SRE.Dec 10 2025, 3:29 PM
herron edited projects, added SRE Observability; removed observability.Dec 17 2025, 2:47 PM
herron moved this task from Inbox to FY2025/2026-Q2 on the SRE Observability board.
herron edited projects, added SRE Observability (FY2025/2026-Q2); removed SRE Observability.
Jhancock.wm subscribed.Jan 12 2026, 7:33 PM

@herron two things

  • do you mind if i rack this in the new expansion cage at codfw?
  • can you add this one and the mwlog1003 in T412230 to site.pp and preseed.yaml?

thank you!

hnowlan moved this task from FY2025/2026-Q2 to FY2025/2026-Q3 on the SRE Observability board.Jan 13 2026, 12:16 PM
hnowlan edited projects, added SRE Observability (FY2025/2026-Q3); removed SRE Observability (FY2025/2026-Q2).
gerritbot added a comment.Jan 14 2026, 12:38 AM

Change #1226396 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] assign mwlog[12]003 insetup role

https://gerrit.wikimedia.org/r/1226396

gerritbot added a project: Patch-For-Review.Jan 14 2026, 12:38 AM
gerritbot added a comment.Jan 14 2026, 12:43 AM

Change #1226396 merged by Herron:

[operations/puppet@production] assign mwlog[12]003 insetup role

https://gerrit.wikimedia.org/r/1226396

herron added a comment.Jan 14 2026, 12:44 AM
In T412229#11513890, @Jhancock.wm wrote:

@herron two things

  • do you mind if i rack this in the new expansion cage at codfw?

Sure, sounds good!

  • can you add this one and the mwlog1003 in T412230 to site.pp and preseed.yaml?

Done!

Maintenance_bot removed a project: Patch-For-Review.Jan 14 2026, 1:40 AM
Jhancock.wm claimed this task.Jan 16 2026, 4:26 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Fri, Feb 13, 3:48 PM
gerritbot added a comment.Fri, Feb 13, 4:14 PM

Change #1239371 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add mwlog2003 to preseed.yaml for inital install

https://gerrit.wikimedia.org/r/1239371

gerritbot added a project: Patch-For-Review.Fri, Feb 13, 4:14 PM
gerritbot added a comment.Fri, Feb 13, 4:16 PM

Change #1239371 merged by Papaul:

[operations/puppet@production] Add mwlog2003 to preseed.yaml for inital install

https://gerrit.wikimedia.org/r/1239371

Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 13, 4:30 PM
ops-monitoring-bot added a comment.Fri, Feb 13, 8:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mwlog2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Fri, Feb 13, 9:39 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mwlog2003.codfw.wmnet with OS bookworm executed with errors:

  • mwlog2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console mwlog2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Feb 13, 9:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mwlog2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Fri, Feb 13, 10:30 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mwlog2003.codfw.wmnet with OS bookworm completed:

  • mwlog2003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202602132203_pt1979_3380529_mwlog2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul updated the task description. (Show Details)Fri, Feb 13, 10:44 PM
gerritbot added a comment.Fri, Feb 13, 10:48 PM

Change #1239435 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Remove mwlog2003 so it can use the default partman

https://gerrit.wikimedia.org/r/1239435

gerritbot added a project: Patch-For-Review.Fri, Feb 13, 10:48 PM
gerritbot added a comment.Fri, Feb 13, 10:51 PM

Change #1239435 merged by Papaul:

[operations/puppet@production] Remove mwlog2003 so it can use the default partman

https://gerrit.wikimedia.org/r/1239435

Papaul closed this task as Resolved.Fri, Feb 13, 10:54 PM
Papaul subscribed.

@herron this complete and ready for you. thanks

Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 13, 11:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits