Page MenuHomePhabricator
Create Task
Maniphest T326362

Q3:rack/setup/install mw2420-mw2451
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of mw2420-mw2451

Hostname / Racking / Installation Details

Hostnames: mw2420-mw2451
Racking Proposal: Spread as evenly as possible across all racks rows. So, around 12 to 13 boxes per rack row (dcops pick as to which racks get more)
Networking Setup: # of Connections:1, Speed:1G. Vlan:Private AAAA records:N, Additional IP records (Cassandra)? No
Partitioning/Raid: HW Raid: N, Partman recipe: partman/custom/mw-raid1-lvm.cfg
OS Distro: Buster
Sub-team Technical Contact: @akosiaris

Per host setup checklist

mw2420: Rack A5 - port 11
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2421: Rack A5 - port 17
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2422: Rack A5 - port 36
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2423: Rack A5 - port 40
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2424: Rack A6 - port 40
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2425: Rack A6 - port41
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2426: Rack A6 - port 42
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2427: Rack A6 - port 43
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2428: Rack B6 - Port 19
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2429: Rack B6 - Port 33
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2430: Rack B6 - port 40
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2431: Rack B6 -port 41
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2432: Rack B8 - port 27
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2433: Rack B8 - port 06
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2434: Rack B8 - port 15
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2435: Rack B8 - port 16
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2436: Rack C1 - U35 : Port 34
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2437: Rack C1 - U36 : Port 35
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2438: Rack C1 - U37 : Port 36
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2439: Rack C1 - U 38 : Port 37
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2440: Rack C5 - U 15 : Port 14
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2441: Rack C5 - U16 : Port 15
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2442: Rack C5 - U17 : Port 16
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2443: Rack C5 - U18 : Port 17
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2444: Rack D5 - U 18 : Port 17
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2445: Rack D5 - U 19 : Port 18
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2446: Rack D5 - U 20 : Port 19
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2447: Rack D5 - U 21 : Port 20
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2448: Rack D6 - U 17 : Port 16
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2449: Rack D6 - U 18 : Port 17
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2450: Rack D6 - U19 : Port 18
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
mw2451: Rack D6 - U20 : Port 19
  • - receive in system on procurement task T325215 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - fix H745 issue by setting each OS disk to 1 disk raid0 (will be fixed in future orders)
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add new mw node to site.ppoperations/puppetproduction+1 -1
Add new mw node to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Feb 9 2023, 12:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2425.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 1:22 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2425.codfw.wmnet with OS buster completed:

  • mw2425 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302090040_pt1979_864298_mw2425.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 1:22 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2424.codfw.wmnet with OS buster completed:

  • mw2424 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302090024_pt1979_860532_mw2424.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 1:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2426.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 1:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2427.codfw.wmnet with OS buster

Papaul updated the task description. (Show Details)Feb 9 2023, 1:36 AM
ops-monitoring-bot added a comment.Feb 9 2023, 2:10 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2426.codfw.wmnet with OS buster completed:

  • mw2426 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302090123_pt1979_874483_mw2426.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 2:11 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2427.codfw.wmnet with OS buster completed:

  • mw2427 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302090127_pt1979_874506_mw2427.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 2:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2428.codfw.wmnet with OS buster

Papaul updated the task description. (Show Details)Feb 9 2023, 2:11 AM
ops-monitoring-bot added a comment.Feb 9 2023, 2:56 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2428.codfw.wmnet with OS buster completed:

  • mw2428 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302090211_pt1979_887384_mw2428.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul updated the task description. (Show Details)Feb 9 2023, 3:12 AM
ops-monitoring-bot added a comment.Feb 9 2023, 1:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2429.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2430.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 2:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 2:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2432.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2429.codfw.wmnet with OS buster completed:

  • mw2429 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091354_pt1979_1017630_mw2429.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2430.codfw.wmnet with OS buster completed:

  • mw2430 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091357_pt1979_1020114_mw2430.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 2:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2433.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 2:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2434.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 3:08 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2432.codfw.wmnet with OS buster completed:

  • mw2432 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091421_pt1979_1023170_mw2432.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 3:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2435.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 3:10 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster executed with errors:

  • mw2431 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 3:31 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2434.codfw.wmnet with OS buster completed:

  • mw2434 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091446_pt1979_1031368_mw2434.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 3:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 3:39 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster executed with errors:

  • mw2431 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 3:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 3:42 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2433.codfw.wmnet with OS buster executed with errors:

  • mw2433 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2435.codfw.wmnet with OS buster executed with errors:

  • mw2435 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 4:37 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster executed with errors:

  • mw2431 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 5:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2433.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 6:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2435.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 6:32 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2433.codfw.wmnet with OS buster completed:

  • mw2433 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091751_pt1979_1069646_mw2433.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 9 2023, 6:33 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2431.codfw.wmnet with OS buster completed:

  • mw2431 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091743_pt1979_1068697_mw2431.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul updated the task description. (Show Details)Feb 9 2023, 6:34 PM
ops-monitoring-bot added a comment.Feb 9 2023, 6:49 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2435.codfw.wmnet with OS buster completed:

  • mw2435 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091808_pt1979_1073984_mw2435.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul updated the task description. (Show Details)Feb 9 2023, 6:55 PM
Jhancock.wm updated the task description. (Show Details)Feb 9 2023, 7:36 PM
Papaul updated the task description. (Show Details)Feb 10 2023, 1:33 AM
Papaul updated the task description. (Show Details)Feb 10 2023, 2:17 AM
Papaul updated the task description. (Show Details)Feb 10 2023, 3:08 AM
Papaul updated the task description. (Show Details)Feb 10 2023, 4:53 PM
Papaul updated the task description. (Show Details)Feb 11 2023, 12:13 AM
Papaul updated the task description. (Show Details)Feb 11 2023, 1:41 AM
Papaul updated the task description. (Show Details)Feb 13 2023, 4:29 PM
gerritbot added a comment.Feb 13 2023, 10:10 PM

Change 888806 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add new mw node to site.pp

https://gerrit.wikimedia.org/r/888806

gerritbot added a project: Patch-For-Review.Feb 13 2023, 10:10 PM

Change 888806 merged by Papaul:

[operations/puppet@production] Add new mw node to site.pp

https://gerrit.wikimedia.org/r/888806

ops-monitoring-bot added a comment.Feb 13 2023, 10:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2436.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 13 2023, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2437.codfw.wmnet with OS buster

Maintenance_bot removed a project: Patch-For-Review.Feb 13 2023, 10:30 PM
ops-monitoring-bot added a comment.Feb 13 2023, 11:18 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2437.codfw.wmnet with OS buster completed:

  • mw2437 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302132225_pt1979_2187776_mw2437.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 13 2023, 11:18 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2436.codfw.wmnet with OS buster completed:

  • mw2436 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302132213_pt1979_2186210_mw2436.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 13 2023, 11:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2438.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 13 2023, 11:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2439.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 12:22 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2439.codfw.wmnet with OS buster completed:

  • mw2439 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302132336_pt1979_2202475_mw2439.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 12:22 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2438.codfw.wmnet with OS buster completed:

  • mw2438 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302132335_pt1979_2202333_mw2438.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 12:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 12:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 12:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 12:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 1:35 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster executed with errors:

  • mw2440 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 14 2023, 1:37 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster executed with errors:

  • mw2441 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 14 2023, 1:39 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster executed with errors:

  • mw2442 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 14 2023, 1:42 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster executed with errors:

  • mw2443 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 14 2023, 1:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 2:01 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster executed with errors:

  • mw2440 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Papaul added a comment.Feb 14 2023, 2:04 AM

@Jhancock.wm can you please take a look at mw244[0-3], it looks like you connected the network cable to NIC 2 and not NIC 1. Thank you

Papaul updated the task description. (Show Details)Feb 14 2023, 2:07 AM
ops-monitoring-bot added a comment.Feb 14 2023, 2:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2444.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 2:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2445.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 2:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2446.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 2:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2447.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 2:55 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2444.codfw.wmnet with OS buster completed:

  • mw2444 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302140211_pt1979_2234204_mw2444.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 3:04 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2446.codfw.wmnet with OS buster completed:

  • mw2446 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302140219_pt1979_2235051_mw2446.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 3:04 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2445.codfw.wmnet with OS buster completed:

  • mw2445 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302140218_pt1979_2234919_mw2445.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 3:29 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2447.codfw.wmnet with OS buster completed:

  • mw2447 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302140243_pt1979_2241522_mw2447.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 2:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2448.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2449.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 3:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2450.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2451.codfw.wmnet with OS buster

Papaul updated the task description. (Show Details)Feb 14 2023, 3:08 PM
ops-monitoring-bot added a comment.Feb 14 2023, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2449.codfw.wmnet with OS buster completed:

  • mw2449 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302141501_pt1979_2434017_mw2449.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2450.codfw.wmnet with OS buster completed:

  • mw2450 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302141504_pt1979_2434354_mw2450.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2448.codfw.wmnet with OS buster completed:

  • mw2448 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302141459_pt1979_2433732_mw2448.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2451.codfw.wmnet with OS buster completed:

  • mw2451 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302141507_pt1979_2434847_mw2451.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 9:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 9:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2440.codfw.wmnet with OS buster completed:

  • mw2440 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302142119_pt1979_2515470_mw2440.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2441.codfw.wmnet with OS buster completed:

  • mw2441 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302142120_pt1979_2515695_mw2441.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 10:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 10:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 14 2023, 11:09 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2443.codfw.wmnet with OS buster completed:

  • mw2443 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302142226_pt1979_2531390_mw2443.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Feb 14 2023, 11:11 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host mw2442.codfw.wmnet with OS buster completed:

  • mw2442 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302142225_pt1979_2531317_mw2442.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul closed this task as Resolved.Feb 14 2023, 11:59 PM
Papaul updated the task description. (Show Details)

@akosiaris this is complete

Clement_Goubert closed subtask T326363: mw2420-mw2451 service implementation tracking as Resolved.Mar 27 2023, 11:48 AM
JMeybohm mentioned this in T358489: mw2420-mw2451 do have unnecessary raid controllers (configured).Feb 26 2024, 1:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits