Page MenuHomePhabricator
Create Task
Maniphest T327295

Q3:rack/setup/install an-worker11[49-56]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of an-worker11[49-56]

Hostname / Racking / Installation Details

Hostnames: an-worker11[49-56]
Racking Proposal: Spread anywhere across rows A-F, trying to avoid having too many other hadoop workers (analytics1* or an-worker1*) in a since rack.
Networking Setup: # of Connections:1 Speed:10G. Vlan:Analytics AAAA records:Y, Additional IP records N.
Partitioning/Raid: HW Raid: Y (only the O/S disks), Partman recipe and/or desired Raid Level: partman/custom/analytics-flex.cfg
OS Distro: Bullseye
Sub-team Technical Contact: @BTullis

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

an-worker1149:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1150:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1151:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1152:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1153:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1154:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1155:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1156:
  • - receive in system on procurement task T325206 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Add an-worker11[49-56] to site.pp and netboot.cfgoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT327295 Q3:rack/setup/install an-worker11[49-56]

Event Timeline

RobH created this task.Jan 18 2023, 3:35 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.
• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jan 31 2023, 4:10 PM
• EChetty moved this task from Backlog to Shared Data Infra on the Data-Engineering-Planning board.Jan 31 2023, 4:12 PM
• EChetty added a project: Shared-Data-Infrastructure.
Jclark-ctr updated the task description. (Show Details)Feb 17 2023, 2:49 PM
Jclark-ctr added a comment.EditedMar 2 2023, 3:24 PM

an-worker1149 A7 U1 port 1 CableId 4899
an-worker1150 B7 U38 port 34 CableId 5013
an-worker1151 C7 U32 port 10 CableId 2965
an-worker1152 D7 U28 port 27 CableId 20220032
an-worker1153 E1 U15 port 15 CableId 5070
an-worker1154 E3 U15 port 15 CableId 5076
an-worker1155 F1 U15 port 15 CableId 20220023
an-worker1155 F3 U15 port 15 CableId20220224

Jclark-ctr updated the task description. (Show Details)Mar 2 2023, 7:13 PM
Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Mar 2 2023, 7:21 PM
Papaul mentioned this in T332523: Eqiad: Backlog HW failure-racking tasks - Decommision and remote work tasks .Mar 19 2023, 8:38 PM
Jclark-ctr added a comment.Mar 30 2023, 11:07 PM

@BTullis what HW raid to not in task

BTullis added a comment.Mar 31 2023, 12:15 AM

Hi @Jclark-ctr

Apologies for any omission on my part.

For these servers we use RAID1 for the OS, based on the two risks in the rear-mounted flex bay.

All of the remaining 12 disks are left unconfigured.

We set them up as single volume RAID0 volumes using this cookbook: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/hadoop/init-hadoop-workers.py

Hope that helps.

Jclark-ctr updated the task description. (Show Details)Mar 31 2023, 12:19 AM
Jclark-ctr updated the task description. (Show Details)Mar 31 2023, 10:39 AM
Jclark-ctr updated the task description. (Show Details)Mar 31 2023, 1:51 PM
Jclark-ctr updated the task description. (Show Details)Mar 31 2023, 2:03 PM
Jclark-ctr updated the task description. (Show Details)Mar 31 2023, 7:27 PM
Jclark-ctr added a comment.May 22 2023, 8:40 PM

Configured raid on an-worker11(49,51,52,53,54,55,56)

Unable to login to management on 50.
need to verify psu1 on 49

Jclark-ctr updated the task description. (Show Details)May 23 2023, 3:32 PM
Jclark-ctr added a comment.May 23 2023, 7:14 PM

Reset idrac. still unable to login to an-worker1150
Fixed psu1 on 49

Jclark-ctr updated the task description. (Show Details)May 24 2023, 6:34 PM
BTullis added a project: Data-Platform-SRE.Jun 9 2023, 11:56 AM
Jhancock.wm claimed this task.Jun 12 2023, 5:03 PM
Jhancock.wm added subscribers: Jclark-ctr, Jhancock.wm.
This comment was removed by Jhancock.wm.
Jhancock.wm updated the task description. (Show Details)Jun 15 2023, 9:03 PM
gerritbot added a comment.Jun 15 2023, 10:57 PM

Change 930724 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add an-worker11[49-56] to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/930724

gerritbot added a project: Patch-For-Review.Jun 15 2023, 10:57 PM
gerritbot added a comment.Jun 15 2023, 11:03 PM

Change 930724 merged by Papaul:

[operations/puppet@production] Add an-worker11[49-56] to site.pp and netboot.cfg

https://gerrit.wikimedia.org/r/930724

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2023, 11:10 PM
ops-monitoring-bot added a comment.Jun 16 2023, 12:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1149.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 16 2023, 12:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1149.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1149 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 16 2023, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1156.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 16 2023, 7:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1156.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1156 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
JArguello-WMF removed a project: Shared-Data-Infrastructure.Jun 29 2023, 1:45 PM
JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 4:14 PM
ops-monitoring-bot added a comment.Jun 30 2023, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1149.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 30 2023, 3:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1149.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1149 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 7 2023, 8:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1156.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 7 2023, 10:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1156.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1156 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Papaul subscribed.Jul 12 2023, 1:40 PM

@Jhancock.wm do we have any update on this?

Jhancock.wm added a comment.Jul 13 2023, 7:32 PM

got the firmware for all the servers updated for everything (bios,idrac,network,sas) last night. finding time and connectivity to make sure all the bios settings are correct and raids are configured correctly.

ops-monitoring-bot added a comment.Jul 14 2023, 6:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1153.eqiad.wmnet with OS bullseye

Jhancock.wm added a comment.EditedJul 14 2023, 7:38 PM

@Jclark-ctr I need your assistance next time you're onsite at Eqiad. These servers do not have a network connection on the 1st port of the NIC card. Can you please change or connect as needed? TY!
an-worker1150 in B7 at U38
an-worker1151 in C7 at U32
an-worker1154 in E3 at U15

edit: an-worker1152 in D7 at U28

@Papaul
I'm having a problem on an-worker1149 where the boot order will not update after applying changes and rebooting. The network boot stays on Embedded rather than changing to Integrated. I've ensured the firmware version of the NIC is 21.85. It's not one of the servers with no connection I tagged John on, so it's not likely physical. Please lmk if you know what's causing this. thanks!

Papaul added a comment.Jul 17 2023, 1:28 PM

@Jhancock.wm I will check and let you know

Jhancock.wm added a comment.EditedJul 17 2023, 2:13 PM

@Jclark-ctr sorry missed one. edited my previous comment with the addition to keep all the info together.

Jhancock.wm added a comment.Jul 20 2023, 6:11 PM

@Papaul finished double checking that I got everything like we discussed. All the firmware is up to date and the NIC issues have been solved. Can you please assist me with by performing this step on all 8 servers..

  • operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::data_engineering.

Thanks.

Papaul added a comment.Jul 20 2023, 8:50 PM

@Jhancock.wm that step was already done on june15 see link below. so you should be good to proceed with the OS install.
Thanks
https://gerrit.wikimedia.org/r/c/operations/puppet/+/930724/

ops-monitoring-bot added a comment.Jul 20 2023, 9:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1156.eqiad.wmnet with OS bullseye

Jhancock.wm updated the task description. (Show Details)Jul 20 2023, 9:41 PM
ops-monitoring-bot added a comment.Jul 20 2023, 9:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1156.eqiad.wmnet with OS bullseye completed:

  • an-worker1156 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307202138_jhancock_1553686_an-worker1156.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Jhancock.wm updated the task description. (Show Details)Jul 20 2023, 9:56 PM
ops-monitoring-bot added a comment.Jul 20 2023, 10:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1155.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 20 2023, 10:48 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1155.eqiad.wmnet with OS bullseye completed:

  • an-worker1155 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307202232_jhancock_1611525_an-worker1155.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Jhancock.wm updated the task description. (Show Details)Jul 20 2023, 10:50 PM
ops-monitoring-bot added a comment.Jul 20 2023, 10:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1154.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 20 2023, 11:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1154.eqiad.wmnet with OS bullseye completed:

  • an-worker1154 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307202316_jhancock_1668076_an-worker1154.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Jhancock.wm updated the task description. (Show Details)Jul 20 2023, 11:38 PM
ops-monitoring-bot added a comment.Jul 20 2023, 11:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1153.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2023, 12:25 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1153.eqiad.wmnet with OS bullseye completed:

  • an-worker1153 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307210003_jhancock_1719874_an-worker1153.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Jhancock.wm updated the task description. (Show Details)Jul 21 2023, 12:26 AM

finished 53-56. should have time to finish the last 4 tomorrow afternoon

Papaul added a comment.Jul 21 2023, 1:33 AM

Thank you

ops-monitoring-bot added a comment.Jul 21 2023, 5:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1152.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2023, 6:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1152.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2023, 7:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1152.eqiad.wmnet with OS bullseye completed:

  • an-worker1152 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307211848_jhancock_2890713_an-worker1152.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Jhancock.wm updated the task description. (Show Details)Jul 21 2023, 7:06 PM
ops-monitoring-bot added a comment.Jul 21 2023, 7:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1151.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2023, 7:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1151.eqiad.wmnet with OS bullseye completed:

  • an-worker1151 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307211924_jhancock_2927613_an-worker1151.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jul 21 2023, 7:41 PM
ops-monitoring-bot added a comment.Jul 21 2023, 7:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1150.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2023, 8:19 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1150.eqiad.wmnet with OS bullseye completed:

  • an-worker1150 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307212002_jhancock_2966140_an-worker1150.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Jul 21 2023, 8:20 PM
ops-monitoring-bot added a comment.Jul 21 2023, 8:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host an-worker1149.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2023, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host an-worker1149.eqiad.wmnet with OS bullseye completed:

  • an-worker1149 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202307212038_jhancock_3007509_an-worker1149.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Jul 21 2023, 9:01 PM
Jhancock.wm updated the task description. (Show Details)

@BTullis finally finished. thanks for your patience.

BTullis mentioned this in T343762: Bring Hadoop workers an-worker11[49-56] into service.Aug 7 2023, 9:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits