Page MenuHomePhabricator
Create Task
Maniphest T354896

Q3:rack/setup/install cloudcontrol2009-dev.codfw.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudcontrol2006-dev.codfw.wmnet

Hostname / Racking / Installation Details

Hostnames: cloudcontrol2006-dev.codfw.wmnet
Racking Proposal: Same rack as cloudcontrol2001-dev; codfw B1
Networking Setup: One connection. Vlan settings are complicated, best to ask cathal
Partitioning/Raid: Simple mirrored raid (hardware if the host has a hw raid controller) partman/standard.cfg partman/raid1-2dev.cfg
OS Distro: Bookworm
Sub-team Technical Contact: Andrew

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudcontrol2009-dev.codfw.wmnet
  • Receive in system on procurement task T354206 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

SubjectRepoBranchLines +/-
Fix entry for cloudcontrol2009-dev in preseed.yamloperations/puppetproduction+1 -1
Add cloudcontrol2009 to site.pp and preseed.yamloperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT354896 Q3:rack/setup/install cloudcontrol2009-dev.codfw.wmnet

Event Timeline

RobH created this task.Jan 11 2024, 6:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2024, 6:52 PM
RobH mentioned this in Unknown Object (Task).Jan 11 2024, 6:53 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jan 11 2024, 7:29 PM
Jhancock.wm updated the task description. (Show Details)Fri, Apr 12, 5:17 PM
Jhancock.wm updated the task description. (Show Details)Mon, Apr 15, 4:35 PM
Jhancock.wm claimed this task.Mon, Apr 15, 4:40 PM
Jhancock.wm added subscribers: cmooney, Jhancock.wm.

@cmooney what is the vlan for this server?

racked in :
B1-U42
port: 41

Jhancock.wm renamed this task from Q3:rack/setup/install cloudcontrol2006-dev.codfw.wmnet to Q3:rack/setup/install cloudcontrol2009-dev.codfw.wmnet.Mon, Apr 15, 4:51 PM
Papaul subscribed.Mon, Apr 15, 5:07 PM

@Jhancock.wm cloud-hosts1-b1-codfw (2118)

Papaul claimed this task.Tue, Apr 16, 1:08 PM
Papaul updated the task description. (Show Details)Tue, Apr 16, 5:01 PM
Papaul updated the task description. (Show Details)Tue, Apr 16, 6:15 PM
gerritbot added a comment.Tue, Apr 16, 8:06 PM

Change #1020343 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add cloudcontrol2009 to site.pp and preseed.yaml

https://gerrit.wikimedia.org/r/1020343

gerritbot added a project: Patch-For-Review.Tue, Apr 16, 8:06 PM
gerritbot added a comment.Tue, Apr 16, 8:16 PM

Change #1020343 merged by Papaul:

[operations/puppet@production] Add cloudcontrol2009 to site.pp and preseed.yaml

https://gerrit.wikimedia.org/r/1020343

ops-monitoring-bot added a comment.Tue, Apr 16, 8:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Apr 16, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm executed with errors:

  • cloudcontrol2009-dev (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" cloudcontrol2009-dev.codfw.wmnet to get a root shellbut depending on the failure this may not work.
gerritbot added a comment.Tue, Apr 16, 10:15 PM

Change #1020378 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Fix entry for cloudcontrol2009-dev in preseed.yaml

https://gerrit.wikimedia.org/r/1020378

Papaul updated the task description. (Show Details)Tue, Apr 16, 10:15 PM
gerritbot added a comment.Tue, Apr 16, 10:18 PM

Change #1020378 merged by Papaul:

[operations/puppet@production] Fix entry for cloudcontrol2009-dev in preseed.yaml

https://gerrit.wikimedia.org/r/1020378

ops-monitoring-bot added a comment.Tue, Apr 16, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Apr 16, 11:06 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm completed:

  • cloudcontrol2009-dev (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202404162246_pt1979_1752718_cloudcontrol2009-dev.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul closed this task as Resolved.Wed, Apr 17, 12:21 AM
Papaul updated the task description. (Show Details)

Complete

Maintenance_bot removed a project: Patch-For-Review.Fri, Apr 26, 6:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL