Page MenuHomePhabricator
Create Task
Maniphest T306849

Q4:(Need By: TBD) rack/setup/install db2175.codfw.wmnet - db2182.codfw.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db2175.codfw.wmnet - db2182.codfw.wmnet

Hostname / Racking / Installation Details

Hostnames: db2175.codfw.wmnet - db2182.codfw.wmnet
Racking Proposal: Two hosts per rack
Networking Setup: 1G private VLAN - no IPv6
Partitioning/Raid: RAID10 256k stripe, writeback cache, no read ahead
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

db2175:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2176:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2177:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2178:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2179:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2180:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2181:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db2182:
  • - receive in system on procurement task T303436 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add new db nodes to site.ppoperations/puppetproduction+5 -0
install_server: Allow reimage db2153 - db2182operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Apr 25 2022, 10:48 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 25 2022, 10:48 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.Apr 25 2022, 10:48 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH updated the task description. (Show Details)Apr 26 2022, 12:01 AM
RobH unsubscribed.
LSobanski moved this task from Triage to Blocked on the DBA board.Apr 26 2022, 8:26 AM
gerritbot added a comment.Jun 28 2022, 7:21 AM

Change 809091 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Allow reimage db2153 - db2182

https://gerrit.wikimedia.org/r/809091

gerritbot added a project: Patch-For-Review.Jun 28 2022, 7:21 AM
gerritbot added a comment.Jun 28 2022, 7:50 AM

Change 809091 merged by Marostegui:

[operations/puppet@production] install_server: Allow reimage db2153 - db2182

https://gerrit.wikimedia.org/r/809091

Maintenance_bot removed a project: Patch-For-Review.Jun 28 2022, 8:30 AM
Papaul updated the task description. (Show Details)Jul 7 2022, 5:28 PM
ops-monitoring-bot added a comment.Jul 7 2022, 8:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2176.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2022, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2177.codfw.wmnet with OS bullseye

gerritbot added a comment.Jul 7 2022, 9:24 PM

Change 812083 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new db nodes to site.pp

https://gerrit.wikimedia.org/r/812083

gerritbot added a project: Patch-For-Review.Jul 7 2022, 9:24 PM

Change 812083 merged by Papaul:

[operations/puppet@production] Add new db nodes to site.pp

https://gerrit.wikimedia.org/r/812083

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2022, 9:31 PM
ops-monitoring-bot added a comment.Jul 7 2022, 9:49 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2177.codfw.wmnet with OS bullseye executed with errors:

  • db2177 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 7 2022, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2176.codfw.wmnet with OS bullseye executed with errors:

  • db2176 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 7 2022, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2176.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2022, 9:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2177.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2022, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2176.codfw.wmnet with OS bullseye completed:

  • db2176 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207072155_pt1979_3545361_db2176.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 7 2022, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2178.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2022, 10:34 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2177.codfw.wmnet with OS bullseye completed:

  • db2177 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207072156_pt1979_3546529_db2177.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 7 2022, 10:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2179.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2022, 11:03 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2178.codfw.wmnet with OS bullseye completed:

  • db2178 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207072225_pt1979_3551025_db2178.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 7 2022, 11:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2180.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Jul 7 2022, 11:15 PM
ops-monitoring-bot added a comment.Jul 7 2022, 11:16 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2179.codfw.wmnet with OS bullseye completed:

  • db2179 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207072239_pt1979_3553594_db2179.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 7 2022, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2181.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2022, 11:43 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2180.codfw.wmnet with OS bullseye completed:

  • db2180 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207072306_pt1979_3559759_db2180.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 8 2022, 12:02 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2181.codfw.wmnet with OS bullseye completed:

  • db2181 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207072326_pt1979_3565103_db2181.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 8 2022, 1:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2182.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 8 2022, 1:49 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2182.codfw.wmnet with OS bullseye completed:

  • db2182 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207080112_pt1979_3583108_db2182.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul updated the task description. (Show Details)Jul 8 2022, 2:02 AM
Papaul added subscribers: Marostegui, Volans.

@Marostegui all the servers are ready but not db2175 @Volans is running some tests on it. Once he is done I will install the OS on it.

Thanks

Volans added a comment.Jul 11 2022, 8:46 AM

@Papaul I've finished my tests on db2175, it's all yours! Thanks for the help. I've sent patches to Gerrit to fix the issue and once merged/released we can test them on a new host.

ops-monitoring-bot added a comment.Jul 11 2022, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db2175.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 11 2022, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db2175.codfw.wmnet with OS bullseye completed:

  • db2175 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207111519_pt1979_203747_db2175.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Jul 11 2022, 4:02 PM
Papaul updated the task description. (Show Details)

@Marostegui All your's

Marostegui added a comment.Jul 12 2022, 5:23 AM

Thanks!

Marostegui closed subtask T311494: Productionize db2175.codfw.wmnet - db2182.codfw.wmnet as Resolved.Aug 22 2022, 7:54 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits