Page MenuHomePhabricator
Create Task
Maniphest T313873

Q1:rack/setup/install kubernetes102[34]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of kubernetes102[34]

Hostname / Racking / Installation Details

Hostnames: kubernetes1023, kubernetes1024
Racking Proposal: Row C or row D preferably.
Networking Setup: 1G, private
Partitioning/Raid: partman/standard.cfg partman/raid1-2dev.cfg partman/custom/kubernetes-node-overlay.cfg
OS Distro: Bullseye
Sub-team Technical Contact: @akosiaris

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

kubernetes1023:
  • - receive in system on procurement task T311853 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes1024:
  • - receive in system on procurement task T311853 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
updateing site.pp for kubernetes servers to change role to insetup::serviceopsoperations/puppetproduction+1 -1
Adding kubernetes1023/1024 to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 Resolved• CmjohnsonT313873 Q1:rack/setup/install kubernetes102[34]
 ResolvedjijikiT313874 kubernetes102[34] implemetation tracking
 ResolvedakosiarisT331126 Update wikikube eqiad to k8s 1.23

Event Timeline

RobH created this task.Jul 26 2022, 11:04 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.Jul 26 2022, 11:07 PM
• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Jul 27 2022, 4:27 PM
Jclark-ctr updated the task description. (Show Details)Aug 17 2022, 6:44 PM
Jclark-ctr added a comment.Aug 26 2022, 3:29 PM

@akosiaris Can you verify host names? kubernetes102[01] Already in use Racking task T290202

akosiaris renamed this task from Q1:rack/setup/install kubernetes102[01] to Q1:rack/setup/install kubernetes102[34].Aug 29 2022, 1:38 PM
akosiaris updated the task description. (Show Details)
In T313873#8189120, @Jclark-ctr wrote:

@akosiaris Can you verify host names? kubernetes102[01] Already in use Racking task T290202

Indeed. My mistake. Updated the task to say kubernetes102[34]. Thanks!

Jclark-ctr added a comment.Aug 29 2022, 8:38 PM

kubernetes1023 c6 u42 port 36 cableid 23000039
kubernetes1024 d8 u25 port 40 cableid 101760

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Aug 29 2022, 8:39 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr subscribed.
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:17 PM
jijiki moved this task from 🙈🙉🙊Backlog to 🛠 Upgrades and Hardware on the serviceops board.Sep 28 2022, 4:30 PM
• Cmjohnson reassigned this task from • Cmjohnson to Jclark-ctr.Sep 28 2022, 4:42 PM
• Cmjohnson subscribed.

@Jclark-ctr can you verify the port for kubernetes1023, looks like something is already in c6/port 36

ops-monitoring-bot added a comment.Sep 28 2022, 5:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 28 2022, 5:16 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1024 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson added a comment.Sep 28 2022, 5:17 PM

Also, @Jclark-ctr please check the network cables are in the correct port. 1024 is giving me a cable failure

PXE-E61: Media test failure, check cable

• Cmjohnson updated the task description. (Show Details)Sep 28 2022, 5:17 PM
Jclark-ctr added a comment.Sep 29 2022, 2:13 PM

kubernetes1023 c6 u42 port 42 cableid 23000039

reseated cable on 1024 it has light now

Jclark-ctr reassigned this task from Jclark-ctr to • Cmjohnson.Sep 29 2022, 2:13 PM
wiki_willy moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Nov 22 2022, 7:53 PM
ops-monitoring-bot added a comment.Dec 6 2022, 9:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye

gerritbot added a comment.Dec 6 2022, 10:50 PM

Change 865189 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] Adding kubernetes1023/1024 to site.pp

https://gerrit.wikimedia.org/r/865189

gerritbot added a project: Patch-For-Review.Dec 6 2022, 10:50 PM

Change 865189 merged by Cmjohnson:

[operations/puppet@production] Adding kubernetes1023/1024 to site.pp

https://gerrit.wikimedia.org/r/865189

ops-monitoring-bot added a comment.Dec 6 2022, 11:04 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1024 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 6 2022, 11:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1023.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 6 2022, 11:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Dec 6 2022, 11:30 PM
ops-monitoring-bot added a comment.Dec 7 2022, 12:18 AM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1023.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1023 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 7 2022, 12:21 AM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye executed with errors:

  • kubernetes1024 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Dec 7 2022, 4:49 PM

Change 865632 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] updateing site.pp for kubernetes servers to change role to insetup::serviceops

https://gerrit.wikimedia.org/r/865632

gerritbot added a project: Patch-For-Review.Dec 7 2022, 4:49 PM
gerritbot added a comment.Dec 7 2022, 4:51 PM

Change 865632 merged by Cmjohnson:

[operations/puppet@production] updateing site.pp for kubernetes servers to change role to insetup::serviceops

https://gerrit.wikimedia.org/r/865632

ops-monitoring-bot added a comment.Dec 7 2022, 4:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 7 2022, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host kubernetes1023.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Dec 7 2022, 5:31 PM
ops-monitoring-bot added a comment.Dec 7 2022, 6:45 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1024.eqiad.wmnet with OS bullseye completed:

  • kubernetes1024 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212071657_cmjohnson_3925466_kubernetes1024.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Dec 7 2022, 6:45 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host kubernetes1023.eqiad.wmnet with OS bullseye completed:

  • kubernetes1023 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212071658_cmjohnson_3925669_kubernetes1023.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
• Cmjohnson closed this task as Resolved.Dec 7 2022, 8:47 PM
• Cmjohnson updated the task description. (Show Details)

completed

akosiaris closed subtask T313874: kubernetes102[34] implemetation tracking as Resolved.Mar 8 2023, 11:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL