Page MenuHomePhabricator
Create Task
Maniphest T299470

(Need By: TBD) rack/setup/install kubernetes20[19|2(012)]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of kubernetes20[19|2(012)]

Hostname / Racking / Installation Details

Hostnames: kubernetes20[19|2(012)]
Racking Proposal: 1 per row
Networking/Subnet/VLAN/IP: single 1g connection, private1 vlan
Partitioning/Raid: echo partman/standard.cfg partman/raid1-2dev.cfg partman/custom/kubernetes-node-overlay.cfg ;; \
OS Distro: Bullseye

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

kubernetes2019 A5 U7 ge-5/0/6
  • - receive in system on procurement task T297157 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2020 B6 U31 ge-6/0/30
  • - receive in system on procurement task T297157 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2021 C3 U39 ge-3/0/38
  • - receive in system on procurement task T297157 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2022 D3 U13 ge-3/0/
  • - receive in system on procurement task T297157 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add kubernetes2019, kubernetes202[0-2] to site.pp and netbooxoperations/puppetproduction+3 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT299470 (Need By: TBD) rack/setup/install kubernetes20[19|2(012)]
 ResolvedJMeybohmT302208 setup/install kubernetes20[1(89)|2(012)]
 ResolvedRequestPapaulT303045 decommission kubernetes200[1-4]

Event Timeline

RobH created this task.Jan 18 2022, 11:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2022, 11:56 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.Jan 18 2022, 11:57 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
Papaul updated the task description. (Show Details)Feb 15 2022, 8:43 PM
JMeybohm subscribed.Feb 17 2022, 9:21 AM
Papaul updated the task description. (Show Details)Feb 17 2022, 9:14 PM
Papaul updated the task description. (Show Details)Feb 18 2022, 1:05 AM
elukey updated the task description. (Show Details)Feb 18 2022, 6:59 AM
elukey subscribed.

@Papaul we are moving all the kubernetes hosts to Bullseye, I modified the task's description with this info + partitioning. Thanks!

Papaul added a comment.Feb 18 2022, 11:36 PM

Thanks

gerritbot added a comment.Feb 18 2022, 11:41 PM

Change 763863 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add kubernetes2019, kubernetes202[0-2] to site.pp and netboox

https://gerrit.wikimedia.org/r/763863

gerritbot added a project: Patch-For-Review.Feb 18 2022, 11:41 PM

Change 763863 merged by Papaul:

[operations/puppet@production] Add kubernetes2019, kubernetes202[0-2] to site.pp and netboox

https://gerrit.wikimedia.org/r/763863

ops-monitoring-bot added a comment.Feb 18 2022, 11:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2019.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2022, 12:10 AM
ops-monitoring-bot added a comment.Feb 19 2022, 12:19 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2019.codfw.wmnet with OS bullseye completed:

  • kubernetes2019 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202182347_pt1979_4117672_kubernetes2019.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 19 2022, 12:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2020.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 19 2022, 12:58 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2020.codfw.wmnet with OS bullseye completed:

  • kubernetes2020 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202190027_pt1979_4123333_kubernetes2020.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 19 2022, 1:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2021.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 19 2022, 1:34 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2021.codfw.wmnet with OS bullseye completed:

  • kubernetes2021 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202190101_pt1979_4128423_kubernetes2021.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 19 2022, 1:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2022.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 19 2022, 2:11 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2022.codfw.wmnet with OS bullseye completed:

  • kubernetes2022 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202190140_pt1979_4135614_kubernetes2022.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Feb 19 2022, 2:12 AM
Papaul updated the task description. (Show Details)

complete

JMeybohm added a subtask: T302208: setup/install kubernetes20[1(89)|2(012)].Feb 21 2022, 5:02 PM
JMeybohm closed subtask T302208: setup/install kubernetes20[1(89)|2(012)] as Resolved.Mar 4 2022, 1:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits