Page MenuHomePhabricator
Create Task
Maniphest T306848

Q4:(Need By: TBD) rack/setup/install db1196.eqiad.wmnet - db1203.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db1196.eqiad.wmnet - db1203.eqiad.wmnet

Hostname / Racking / Installation Details

Hostnames: db1196.eqiad.wmnet - db1203.eqiad.wmnet
Racking Proposal: One host per racks A-D, two hosts per racks E-F
Networking Setup: 1G private VLAN - no IPv6
Partitioning/Raid: RAID10 256k stripe, writeback cache, no read ahead
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

db1196:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)

[]x - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.

  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1197:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1198:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1199:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1200:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1201:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1202:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1203:
  • - receive in system on procurement task T303435 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
site.pp: Add db1196-db1203 to insetupoperations/puppetproduction+6 -0
db1196-db1203: Disable notificationsoperations/puppetproduction+16 -0
install_server: Allow reimage of db1196-db1203operations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Apr 25 2022, 10:45 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.Apr 25 2022, 11:00 PM
RobH updated the task description. (Show Details)Apr 26 2022, 12:01 AM
LSobanski moved this task from Triage to Blocked on the DBA board.Apr 26 2022, 8:25 AM
gerritbot added a comment.Aug 24 2022, 12:46 PM

Change 826288 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Allow reimage of db1196-db1209

https://gerrit.wikimedia.org/r/826288

gerritbot added a project: Patch-For-Review.Aug 24 2022, 12:46 PM
• Marostegui added subscribers: RobH, • Marostegui.EditedAug 24 2022, 12:48 PM

@RobH I just realised the task has more hosts than the ones we actually ordered, at T303435 we say 8 hosts, but this task has racking plans for 14. Not sure which one is right.

gerritbot added a comment.Aug 24 2022, 12:50 PM

Change 826288 merged by Marostegui:

[operations/puppet@production] install_server: Allow reimage of db1196-db1203

https://gerrit.wikimedia.org/r/826288

gerritbot added a comment.Aug 24 2022, 12:53 PM

Change 826290 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1196-db1203: Disable notifications

https://gerrit.wikimedia.org/r/826290

gerritbot added a comment.Aug 24 2022, 12:54 PM

Change 826290 merged by Marostegui:

[operations/puppet@production] db1196-db1203: Disable notifications

https://gerrit.wikimedia.org/r/826290

Maintenance_bot removed a project: Patch-For-Review.Aug 24 2022, 1:30 PM
Jclark-ctr updated the task description. (Show Details)Aug 24 2022, 11:08 PM
Jclark-ctr added a comment.Aug 24 2022, 11:13 PM

@Marostegui Can you confirm racking? i am unsure if you meant 1 per row or per rack. Did you want row diversity 1 host per row A-D and 2 each E,F I read racking instructions as 2 per rack.

db1196 e2
db1197 e2
db1198. e3
db1199 e3
db1120 f2
db1121 f2
db1122 f3
db1123 f3

• Marostegui added a comment.Aug 25 2022, 4:23 AM

That's ok from my side

• Marostegui added a comment.Aug 25 2022, 4:25 AM

Please note that the last hostnames should be:
db1201
db1202
db1203

• Marostegui mentioned this in T316342: Productionize db1196-db1203.Aug 26 2022, 11:16 AM
gerritbot added a comment.Aug 30 2022, 6:22 AM

Change 827859 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] site.pp: Add db1196-db1203 to insetup

https://gerrit.wikimedia.org/r/827859

gerritbot added a project: Patch-For-Review.Aug 30 2022, 6:22 AM

Change 827859 merged by Marostegui:

[operations/puppet@production] site.pp: Add db1196-db1203 to insetup

https://gerrit.wikimedia.org/r/827859

• Marostegui added a subscriber: • Cmjohnson.Aug 30 2022, 6:24 AM

@Cmjohnson @Jclark-ctr I have added the insetup role for these hosts and the partitioning schema in puppet for you. Hope that's ok!

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2022, 6:30 AM
RobH updated the task description. (Show Details)Aug 30 2022, 3:59 PM
RobH unsubscribed.
Jclark-ctr added a comment.Sep 1 2022, 5:51 PM

db1196 E2 U36 Port41 CableId 23000071
db1197 E2 U37 Port42 CableId 23000035
db1198 E3 U36 Port41 CableId 23000038
db1199 E3 U37 Port42 CableId 23000043
db1200 F2 U36 Port41 CableId 23000055
db1201 F2 U37 Port42 CableId 23000072
db1202 F3 U38 Port41 CableId 23000070
db1203 F3 U39 Port42 CableId 23000059

Jclark-ctr reassigned this task from Jclark-ctr to Papaul.Sep 1 2022, 5:52 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr subscribed.
Papaul updated the task description. (Show Details)Sep 2 2022, 1:13 AM
ops-monitoring-bot added a comment.Sep 2 2022, 3:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db1197.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 2 2022, 3:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1198.eqiad.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Sep 2 2022, 3:34 PM
ops-monitoring-bot added a comment.Sep 2 2022, 3:49 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db1197.eqiad.wmnet with OS bullseye completed:

  • db1197 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021518_pt1979_4094718_db1197.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 2 2022, 3:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db1199.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 2 2022, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1198.eqiad.wmnet with OS bullseye completed:

  • db1198 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021533_pt1979_880858_db1198.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 2 2022, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1200.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 2 2022, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db1199.eqiad.wmnet with OS bullseye completed:

  • db1199 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021557_pt1979_4103580_db1199.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 2 2022, 4:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db1201.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 2 2022, 4:39 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1200.eqiad.wmnet with OS bullseye completed:

  • db1200 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021607_pt1979_887717_db1200.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 2 2022, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1202.eqiad.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Sep 2 2022, 4:42 PM
ops-monitoring-bot added a comment.Sep 2 2022, 5:00 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db1201.eqiad.wmnet with OS bullseye completed:

  • db1201 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021631_pt1979_4110493_db1201.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 2 2022, 5:11 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1202.eqiad.wmnet with OS bullseye completed:

  • db1202 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021640_pt1979_894640_db1202.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Sep 2 2022, 5:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host db1203.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 2 2022, 5:58 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host db1203.eqiad.wmnet with OS bullseye completed:

  • db1203 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021729_pt1979_4121193_db1203.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Papaul closed this task as Resolved.Sep 2 2022, 6:09 PM
Papaul updated the task description. (Show Details)

@Marostegui @Jclark-ctr this is complete

• Marostegui added a comment.Sep 6 2022, 5:11 AM

Thanks Papaul and John! I can reach all the hosts just fine

• Marostegui closed subtask T316342: Productionize db1196-db1203 as Resolved.Sep 8 2022, 7:02 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits