Page MenuHomePhabricator
Create Task
Maniphest T342534

Q1:rack/setup/install kubernetes20[25-53]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of kubernetes20[25-53]

Hostname / Racking / Installation Details

Hostnames: kubernetes2025-kubernetes2054
Racking Proposal: Split across rack rows. 7 in rows A and B, 8 in rows C and D. Also split per rack, preferably 1 or 2 nodes per rack please. No exact U positions requirements from our side
Networking Setup: # of Connections: 1 , Speed: 1G . Vlan: Private. AAAA records: Y, Additional IP records: No
Partitioning/Raid: HW Raid: N, Partman recipe: echo partman/standard.cfg partman/raid1-2dev.cfg partman/custom/kubernetes-node-overlay.cfg ;;
OS Distro: Bullseye (default unless otherwise specified)
Sub-team Technical Contact: @akosiaris

Per host setup checklist

kubernetes2025:

Rack: A8
U: 31
Port: 30

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2026:

Rack: A8
U: 30
Port: 29

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2027:

Rack: A6
U: 5
Port: 4

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2028:

Rack: A6
U: 4
Port: 3

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2029:

Rack: B3
U: 5
Port: 4

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2030:

Rack: B3
U: 4
Port: 3

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2031:

Rack: B5
U: 27
Port: 26

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2032:

Rack: B5
U: 10
Port: 9

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2033:

Rack: B6
U: 43
Port: 42

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2034:

Rack: B6
U: 19
Port: 9

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2035:

Rack: B8
U: 30
Port: 29

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2036:

Rack: C1
U: 6
Port: 5

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2037:

Rack: C1
U: 5
Port: 4

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2038:

Rack: C3
U: 42
Port: 41

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2039:

Rack: C3
U: 41
Port: 40

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2040:

Rack: C5
U: 19
Port: 18

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2041:

Rack: C5
U: 10
Port: 9

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2042:

Rack: C6
U: 3
Port: 2

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2043:

Rack: C6
U: 2
Port: 1

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2044:

Rack: D1
U: 12
Port: 11

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2045:

Rack: D1
U: 11
Port: 10

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2046:

Rack: D3
U: 36
Port: 35

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2047:

Rack: D3
U: 35
Port: 34

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2048:

Rack: D5
U: 29
Port: 28

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2049:

Rack: D5
U: 15
Port: 14

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2050:

Rack: D6
U: 28
Port: 27

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2051:

Rack: D6
U: 9
Port: 8

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2052:

Rack: D8
U: 1
Port: 0

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
kubernetes2053:

Rack: D8
U: 20
Port: 19

  • - receive in system on procurement task 341265 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Fix typo on kubernetes203[0-9] missing s at the endoperations/puppetproduction+1 -1
site.pp: Split wikikube workers per DCoperations/puppetproduction+5 -1
Add new kubernetes node to netboot.cfgoperations/puppetproduction+1 -1
Add new kubernetes node to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 ResolvedPapaulT342534 Q1:rack/setup/install kubernetes20[25-53]
 ResolvedJoeT345709 Setup kubernetes20[25-53]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Sep 1 2023, 4:23 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kubernetes2036.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2036 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309011622_jhancock_2968942_kubernetes2036.out
    • The reimage failed, see the cookbook logs for the details
Papaul claimed this task.Sep 1 2023, 4:29 PM
Papaul added a subscriber: Jhancock.wm.
Jhancock.wm updated the task description. (Show Details)Sep 1 2023, 5:22 PM
Jhancock.wm added a comment.Sep 1 2023, 5:24 PM

got 25-39 provisioned without errors.
25-39 had bios and idrac firmware upgraded without errors.
did not downgrade NIC. got about mismatched component. But version matches 40-54.
attempted to and failed to do OS on 25, 36, 37, 38, and 39

ops-monitoring-bot added a comment.Sep 1 2023, 5:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 6:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2026.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 6:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 6:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2028.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 6:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2029.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 6:48 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2027 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309011842_pt1979_3093259_kubernetes2027.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 6:53 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2028.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2028 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309011846_pt1979_3097461_kubernetes2028.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 6:57 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2029.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2029 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309011851_pt1979_3107487_kubernetes2029.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 7:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 7:11 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2027 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309011910_pt1979_3147576_kubernetes2027.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 7:20 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2025 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 7:23 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2026.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2026 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 7:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 8:49 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2025 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 1 2023, 10:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 1 2023, 10:45 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2027 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309012245_pt1979_3371946_kubernetes2027.out
    • The reimage failed, see the cookbook logs for the details
Papaul added subscribers: jbond, Volans.Sep 1 2023, 10:59 PM

@akosiaris kubernetes2025 to 2029 was failing durin os install because of puppet, i login to 2025 console and manually did the puppet run i got the error

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /etc/puppet/modules/profile/manifests/kubernetes/node.pp, line: 87, column: 15) on node kubernetes2025.codfw.wmnet
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

I then looked at the site.pp file we had at line 1633:

1633 node /^kubernetes[12]0(0[5-9]|1[0-9]|2[0-6]|5[5-8])\.(codfw|eqiad)\./ {
1634     role(kubernetes::worker)
1635 }

and at line 1646 i had added

1646 # codfw new kubernetes nodes T342534
1647 node /^kubernetes20(2[5-9]|3[0-9]|4[0-9]|5[0-4])\.codfw\./ {
1648     role(insetup::serviceops)
1649 }

so 2025 and 2026 nodes had 2 roles, insetup and kubernetes::worker roles

@jbond @Volans on 2027 - 2029
puppet is failing with

----- OUTPUT of 'run-puppet-agent --quiet' -----                                                                      
Error: Could not prepare for execution: The puppet agent command does not take parameters                             
================                                                                                                      
PASS |                                                                                |   0% (0/1) [00:01<?, ?hosts/s]
FAIL |████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00,  1.79s/hosts]
100.0% (1/1) of nodes failed to execute command 'run-puppet-agent --quiet': cumin2002.codfw.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'run-puppet-agent --quiet'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.
Volans added a comment.Sep 4 2023, 10:12 AM
In T342534#9137952, @Papaul wrote:

@jbond @Volans on 2027 - 2029
puppet is failing with

----- OUTPUT of 'run-puppet-agent --quiet' -----                                                                      
Error: Could not prepare for execution: The puppet agent command does not take parameters                             
================                                                                                                      
PASS |                                                                                |   0% (0/1) [00:01<?, ?hosts/s]
FAIL |████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00,  1.79s/hosts]
100.0% (1/1) of nodes failed to execute command 'run-puppet-agent --quiet': cumin2002.codfw.wmnet
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'run-puppet-agent --quiet'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

@Papaul that's not a cookbook failure, is the run-puppet-agent command failing, see T345548

gerritbot added a comment.Sep 4 2023, 10:45 AM

Change 954615 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] site.pp: Split wikikube workers per DC

https://gerrit.wikimedia.org/r/954615

gerritbot added a project: Patch-For-Review.Sep 4 2023, 10:45 AM
JMeybohm subscribed.Sep 4 2023, 11:03 AM
In T342534#9137952, @Papaul wrote:

[...]
so 2025 and 2026 nodes had 2 roles, insetup and kubernetes::worker roles

That was overlooked when we added some nodes to eqiad only, sorry! Should be fixed now.

gerritbot added a comment.Sep 4 2023, 11:03 AM

Change 954615 merged by JMeybohm:

[operations/puppet@production] site.pp: Split wikikube workers per DC

https://gerrit.wikimedia.org/r/954615

Stashbot added a comment.Sep 4 2023, 11:08 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-04T11:08:05Z] <jbond@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: " - jbond@cumin1001 - T342534"

Stashbot added a comment.Sep 4 2023, 11:08 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-04T11:08:56Z] <jbond@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: " - jbond@cumin1001 - T342534"

Maintenance_bot removed a project: Patch-For-Review.Sep 4 2023, 11:10 AM
jbond added a comment.Sep 4 2023, 11:11 AM

FYi i just checked kubernetes2025 (via install-console), kubernetes2027 and kubernetes2029 and puppet seems to be running well now

Volans mentioned this in T268369: how to deal with cumin alias alerts.Sep 5 2023, 11:49 AM
ops-monitoring-bot added a comment.Sep 5 2023, 1:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 1:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2026.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 1:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye

Papaul added a comment.Sep 5 2023, 1:10 PM

@JMeybohm @jbond @Volans thanks.

ops-monitoring-bot added a comment.Sep 5 2023, 1:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2028.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 1:20 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2025.codfw.wmnet with OS bullseye completed:

  • kubernetes2025 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051309_pt1979_406299_kubernetes2025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 1:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2029.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Sep 5 2023, 1:21 PM
ops-monitoring-bot added a comment.Sep 5 2023, 1:22 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2027.codfw.wmnet with OS bullseye completed:

  • kubernetes2027 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051311_pt1979_409146_kubernetes2027.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul updated the task description. (Show Details)Sep 5 2023, 1:22 PM
ops-monitoring-bot added a comment.Sep 5 2023, 1:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2028.codfw.wmnet with OS bullseye completed:

  • kubernetes2028 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051315_pt1979_414821_kubernetes2028.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul updated the task description. (Show Details)Sep 5 2023, 1:25 PM
ops-monitoring-bot added a comment.Sep 5 2023, 1:36 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2026.codfw.wmnet with OS bullseye completed:

  • kubernetes2026 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051318_pt1979_407637_kubernetes2026.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 1:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 1:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 2:13 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2029.codfw.wmnet with OS bullseye completed:

  • kubernetes2029 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051343_pt1979_427286_kubernetes2029.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2030 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
Papaul updated the task description. (Show Details)Sep 5 2023, 2:17 PM
ops-monitoring-bot added a comment.Sep 5 2023, 2:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2031 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 5 2023, 2:33 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2032 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Sep 5 2023, 3:05 PM

Change 954960 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Fix typo on kubernetes203[0-9] missing s at the end

https://gerrit.wikimedia.org/r/954960

gerritbot added a project: Patch-For-Review.Sep 5 2023, 3:05 PM

Change 954960 merged by Papaul:

[operations/puppet@production] Fix typo on kubernetes203[0-9] missing s at the end

https://gerrit.wikimedia.org/r/954960

Maintenance_bot removed a project: Patch-For-Review.Sep 5 2023, 3:11 PM
ops-monitoring-bot added a comment.Sep 5 2023, 3:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 3:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 3:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 4:06 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2032.codfw.wmnet with OS bullseye completed:

  • kubernetes2032 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051547_pt1979_562067_kubernetes2032.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2031.codfw.wmnet with OS bullseye completed:

  • kubernetes2031 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051544_pt1979_560266_kubernetes2031.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Papaul updated the task description. (Show Details)Sep 5 2023, 4:07 PM
ops-monitoring-bot added a comment.Sep 5 2023, 4:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2034.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 4:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2035.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 4:28 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2030.codfw.wmnet with OS bullseye completed:

  • kubernetes2030 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051611_pt1979_551885_kubernetes2030.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 4:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2036.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Sep 5 2023, 4:30 PM
ops-monitoring-bot added a comment.Sep 5 2023, 4:38 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2033.codfw.wmnet with OS bullseye completed:

  • kubernetes2033 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051621_pt1979_605199_kubernetes2033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 4:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 5:02 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2034.codfw.wmnet with OS bullseye completed:

  • kubernetes2034 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051633_pt1979_616087_kubernetes2034.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 5:03 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2035.codfw.wmnet with OS bullseye completed:

  • kubernetes2035 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051635_pt1979_616729_kubernetes2035.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Papaul updated the task description. (Show Details)Sep 5 2023, 5:03 PM
ops-monitoring-bot added a comment.Sep 5 2023, 5:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 5:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2038 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 5 2023, 5:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 5:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye executed with errors:

  • kubernetes2038 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 5 2023, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2023, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2036.codfw.wmnet with OS bullseye completed:

  • kubernetes2036 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051649_pt1979_641713_kubernetes2036.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Sep 5 2023, 5:09 PM
Papaul renamed this task from Q1:rack/setup/install kubernetes20[25-54] to Q1:rack/setup/install kubernetes20[25-53].Sep 5 2023, 5:16 PM
Papaul mentioned this in T345650: Q1:rack/setup/install kubernetes2054.
Papaul updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 5 2023, 5:21 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2037.codfw.wmnet with OS bullseye completed:

  • kubernetes2037 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051705_pt1979_652510_kubernetes2037.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Sep 5 2023, 5:55 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2038.codfw.wmnet with OS bullseye completed:

  • kubernetes2038 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051734_pt1979_688561_kubernetes2038.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 5 2023, 5:56 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2039.codfw.wmnet with OS bullseye completed:

  • kubernetes2039 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202309051736_pt1979_688949_kubernetes2039.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Papaul updated the task description. (Show Details)Sep 5 2023, 5:57 PM
Papaul updated the task description. (Show Details)
Papaul added a comment.Sep 5 2023, 6:11 PM

This is all done but i have to fix DNS mgmt issue with 2029 and 2030 during the provision of those servers the serial number were swapped. Whne you login to 2029 mgmt it is sending you to 2030.

Papaul closed this task as Resolved.Sep 6 2023, 12:45 AM

mgmt DNS for k8s2029 and 2030 fixed.
@akosiaris all yours the last node will be track @T345650

JMeybohm closed subtask T345709: Setup kubernetes20[25-53] as Resolved.Sep 25 2023, 11:58 AM
ops-monitoring-bot added a comment.Oct 6 2023, 10:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2054.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 6 2023, 11:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2054.codfw.wmnet with OS bullseye completed:

  • kubernetes2054 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310062250_pt1979_3421612_kubernetes2054.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
jbond unsubscribed.Oct 9 2023, 10:35 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits