Page MenuHomePhabricator
Create Task
Maniphest T317892

Q2:rack/setup/install puppetdb1003
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of puppetdb1003

Hostname / Racking / Installation Details

Hostnames: puppetdb1003.eqiad.wmnet
Racking Proposal: Doesn't matter
Networking Setup: private VLAN, 1G, no additional addresses needed
Partitioning/Raid: partman/raid1-2dev.cfg
OS Distro: Bullseye
Sub-team Technical Contact: JohnB and Moritz

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

puppetdb1003:
  • - receive in system on procurement task T317322 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add puppetdb1003 and dbprov1004 to site.pp and netboox.cfgoperations/puppetproduction+10 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT317892 Q2:rack/setup/install puppetdb1003

Event Timeline

RobH created this task.Sep 15 2022, 3:18 PM
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in Unknown Object (Task).
RobH renamed this task from Q1:rack/setup/install X to Q2:rack/setup/install puppetdb1003.Sep 15 2022, 3:23 PM
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH edited subscribers, added: jbond, MoritzMuehlenhoff; removed: RobH.
• Cmjohnson assigned this task to Jclark-ctr.Sep 28 2022, 5:19 PM
Jclark-ctr added a comment.Nov 2 2022, 6:41 PM

puppetdb1003 B1 U39 cableID 23000044 port39

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Nov 2 2022, 6:41 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr subscribed.
wiki_willy moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Nov 22 2022, 7:53 PM
wiki_willy added subscribers: Papaul, wiki_willy.Nov 22 2022, 8:01 PM

Hi @MoritzMuehlenhoff - thanks for the heads up on IRC. @Papaul will be taking a look at the host, to wrap up the installation by the end of the week. Thanks, Willy

Papaul updated the task description. (Show Details)Nov 22 2022, 8:58 PM
Papaul updated the task description. (Show Details)Nov 22 2022, 10:17 PM
gerritbot added a comment.Nov 22 2022, 10:30 PM

Change 859625 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add puppetdb1003 and dbprov1004 to site.pp and netboox.cfg

https://gerrit.wikimedia.org/r/859625

gerritbot added a project: Patch-For-Review.Nov 22 2022, 10:30 PM
gerritbot added a comment.Nov 22 2022, 10:32 PM

Change 859625 merged by Papaul:

[operations/puppet@production] Add puppetdb1003 and dbprov1004 to site.pp and netboox.cfg

https://gerrit.wikimedia.org/r/859625

ops-monitoring-bot added a comment.Nov 22 2022, 10:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host puppetdb1003.eqiad.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Nov 22 2022, 10:36 PM
ops-monitoring-bot added a comment.Nov 22 2022, 11:06 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host puppetdb1003.eqiad.wmnet with OS bullseye completed:

  • puppetdb1003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211222236_pt1979_164934_puppetdb1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul closed this task as Resolved.Nov 22 2022, 11:11 PM
Papaul updated the task description. (Show Details)

@MoritzMuehlenhoff this complete

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2022, 11:30 PM
ayounsi reopened this task as Open.Dec 19 2022, 2:28 PM
ayounsi reassigned this task from • Cmjohnson to Papaul.
ayounsi added subscribers: • Cmjohnson, ayounsi.

The host is alerting with puppetdb1003 (WMF10625) Primary IPv6 missing DNS name in https://netbox.wikimedia.org/extras/reports/network.Network/

Papaul added a comment.Jan 11 2023, 8:37 PM

Fixed

Papaul closed this task as Resolved.Jan 12 2023, 9:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL