Page MenuHomePhabricator
Create Task
Maniphest T313959

Q1:rack/setup/install kafka-logging200[45]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of kafka-logging200[45]

Hostname / Racking / Installation Details

Hostnames: kafka-logging200[45]
Racking Proposal: One host in row D please, and the other in next available location
Networking Setup: # of Connections:1, Speed:10G. Vlan:Private AAAA records:N
Partitioning/Raid: HW Raid: Y, Partman recipe and/or desired Raid Level: raid50, using same config as other kafka-logging hosts
OS Distro: Bullseye (default unless otherwise specified)
Sub-team Technical Contact: Who should our on-sites contact with any questions involving system racking and setup? @herron

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

kafka-logging2004:
  • - receive in system on procurement task T311872 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kafka-logging2005:
  • - receive in system on procurement task T311872 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add kafka-logging200[45] to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT313959 Q1:rack/setup/install kafka-logging200[45]

Event Timeline

RobH created this task.Jul 27 2022, 5:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 27 2022, 5:40 PM
RobH mentioned this in Unknown Object (Task).Jul 27 2022, 5:40 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
RobH mentioned this in T313960: Q1:rack/setup/install kafka-logging100[45].Jul 27 2022, 5:44 PM
lmata added a project: SRE Observability.Aug 1 2022, 7:52 PM
Papaul updated the task description. (Show Details)Aug 17 2022, 1:28 AM
gerritbot added a comment.Aug 17 2022, 5:02 PM

Change 824243 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add kafka-logging200[45] to site.pp

https://gerrit.wikimedia.org/r/824243

gerritbot added a project: Patch-For-Review.Aug 17 2022, 5:02 PM
gerritbot added a comment.Aug 17 2022, 5:04 PM

Change 824243 merged by Papaul:

[operations/puppet@production] Add kafka-logging200[45] to site.pp

https://gerrit.wikimedia.org/r/824243

Maintenance_bot removed a project: Patch-For-Review.Aug 17 2022, 5:30 PM
ops-monitoring-bot added a comment.Aug 17 2022, 8:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-logging2005.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Aug 17 2022, 8:40 PM
ops-monitoring-bot added a comment.Aug 17 2022, 9:07 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-logging2005.codfw.wmnet with OS bullseye completed:

  • kafka-logging2005 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208172028_pt1979_407327_kafka-logging2005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Aug 17 2022, 9:22 PM
Papaul updated the task description. (Show Details)

@herron all yours

herron mentioned this in T326419: Expand kafka-logging using hosts kafka-logging[12]00[45].Jan 6 2023, 3:43 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL