Page MenuHomePhabricator
Create Task
Maniphest T306928

Q3:(Need By: TBD) rack/setup/install db1185.eqiad.wmnet - db1195.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of db1185.eqiad.wmnet - db1195.eqiad.wmnet

Hostname / Racking / Installation Details

Hostnames: db1185.eqiad.wmnet - db1195.eqiad.wmnet
Racking Proposal: If possible spread across rows and racks: 3 on row A, 3 on row B, 3 on row C, 2 on row D
Networking/Subnet/VLAN/IP: 1G private VLAN - no IPv6 assignment
Partitioning/Raid: RAID10 256k stripe, writeback cache, no read ahead
OS Distro: Bullseye

Per host setup checklist

db1185:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1186:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1187:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1188:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1189:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1190:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1191:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1192:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1193:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1194:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
db1195:
  • - receive in system on procurement task T297648 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
install_server: Allow reimage db1185-db1195operations/puppetproduction+1 -1
Adding new db servers to site.ppoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Jul 21 2022, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host db1195.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1189.eqiad.wmnet with OS bullseye executed with errors:

  • db1189 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1193.eqiad.wmnet with OS bullseye executed with errors:

  • db1193 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1190.eqiad.wmnet with OS bullseye executed with errors:

  • db1190 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1186.eqiad.wmnet with OS bullseye executed with errors:

  • db1186 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1188.eqiad.wmnet with OS bullseye executed with errors:

  • db1188 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1194.eqiad.wmnet with OS bullseye executed with errors:

  • db1194 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1185.eqiad.wmnet with OS bullseye executed with errors:

  • db1185 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1191.eqiad.wmnet with OS bullseye executed with errors:

  • db1191 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1187.eqiad.wmnet with OS bullseye executed with errors:

  • db1187 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1195.eqiad.wmnet with OS bullseye executed with errors:

  • db1195 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 21 2022, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1192.eqiad.wmnet with OS bullseye executed with errors:

  • db1192 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson added a comment.Jul 21 2022, 2:25 PM

@Jclark-ctr These all failed because the cable is not connected, can you please verify that you connected the 1G ports to the switch. There is also a possibility that the script is looking for the 10G port.

• Cmjohnson updated the task description. (Show Details)Jul 21 2022, 2:26 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 21 2022, 2:30 PM
Marostegui changed the status of subtask T313569: Productionize db1185.eqiad.wmnet - db1195.eqiad.wmnet from Open to Stalled.Jul 22 2022, 5:17 AM
Jclark-ctr added a subscriber: cmooney.Jul 27 2022, 10:35 PM

@cmooney curious if this is a switch config that needs to change these are racked in 10g racks but only use 1g ports

db1190 E1 35 port:40 2013339101767
db1191 E2 35 port:40 2013339101930
db1192 E3 35 port:40 2013339101779
db1193 F1 35 port:40 2013330101775
db1194 F2 35 port:40 2013339101757
db1195 F3 37 port:40 2013339101795

Papaul subscribed.Jul 28 2022, 8:45 PM
papaul@lsw1-e2-eqiad> show interfaces descriptions | match db1191
ge-0/0/40                  db1191 {#2013339101930}
papaul@lsw1-e2-eqiad> show interfaces ge-0/0/40
error: device ge-0/0/40 not found
{master:0}[edit chassis fpc 0 pic 0]
papaul@lsw1-e2-eqiad# show
port 24 {
    speed 1G;
}
master:0}[edit chassis fpc 0 pic 0]
papaul@lsw1-e2-eqiad# set port 40 speed 1G
[edit chassis fpc 0 pic 0 port 40 speed]
  'speed 1G'
    warning: 1g config will be applied to ports 40 to 43

double check that there is nothing connected to ports 40 to 43 before commit

{master:0}[edit chassis fpc 0 pic 0]
papaul@lsw1-e2-eqiad# show
port 24 {
    speed 1G;
}
port 40 {
    speed 1G;
}

After setting the port to 1G on the switch

papaul@lsw1-e2-eqiad# run show interfaces ge-0/0/40 descriptions
Interface       Admin Link Description
ge-0/0/40       up    up   db1191 {#2013339101930}
Papaul added a comment.Jul 29 2022, 12:23 AM

@Cmjohnson For the once in row A,B and C it looks like the OS is already installed on them so you can use the --no-pxe and --new flags to re-run the reimage.

ops-monitoring-bot added a comment.Jul 29 2022, 2:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host db1185.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 29 2022, 2:03 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1185.eqiad.wmnet with OS bullseye executed with errors:

  • db1185 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 29 2022, 2:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host db1186.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 29 2022, 2:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host db1187.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 29 2022, 2:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host db1188.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 29 2022, 2:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host db1189.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 29 2022, 2:23 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1187.eqiad.wmnet with OS bullseye completed:

  • db1187 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207291409_cmjohnson_424458_db1187.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 29 2022, 2:26 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1186.eqiad.wmnet with OS bullseye completed:

  • db1186 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207291406_cmjohnson_424199_db1186.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 29 2022, 2:28 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1188.eqiad.wmnet with OS bullseye completed:

  • db1188 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207291409_cmjohnson_424622_db1188.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Jul 29 2022, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host db1189.eqiad.wmnet with OS bullseye completed:

  • db1189 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207291409_cmjohnson_424670_db1189.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
• Cmjohnson updated the task description. (Show Details)Jul 29 2022, 2:35 PM
Marostegui added a comment.Aug 4 2022, 8:18 AM

Any rough ETA about when these hosts will be ready?
We are also seeing some mgmt alerts for db1186, db1187 and db1188 regarding their DNS:

db1186.mgmt
DNS
View Extra Service Notes
CRITICAL	2022-08-04 07:22:32	5d 17h 40m 22s	3/3	DNS CRITICAL - expected '10.65.3.0' but got '10.65.2.255'	
db1187.mgmt
DNS
View Extra Service Notes
CRITICAL	2022-08-04 07:42:36	5d 17h 23m 36s	3/3	DNS CRITICAL - expected '10.65.3.1' but got '10.65.3.0'	
db1188.mgmt
DNS
View Extra Service Notes
CRITICAL	2022-08-04 07:50:55	5d 17h 18m 43s	3/3	DNS CRITICAL - expected '10.65.2.255' but got '10.65.3.1'
Papaul added a subscriber: ayounsi.Aug 4 2022, 12:50 PM

@ayounsi see @Marostegui comment above.

Thnaks

ayounsi added a comment.EditedAug 4 2022, 2:03 PM

That's something for ops-eqiad, I think there was some confusion during the provisioning of those hosts:

For example, here mgmt DNS is set to .2.255, but there IPMI is configured with .3.0.

Furthermore, according to https://netbox.wikimedia.org/extras/reports/puppetdb.PhysicalHosts/ those same 3 hosts are alerting with:
db1186 (WMF11513) - mismatched serials: 4D[...] (netbox) != 2D[...] (puppetdb)
db1187 (WMF11512) - mismatched serials: 2D[...] (netbox) != JC[...] (puppetdb)
db1188 (WMF11514) - mismatched serials: JC[...] (netbox) != 4D[...] (puppetdb)

Papaul added a comment.Aug 5 2022, 2:31 PM

@Marostegui Chris is on Vacation, I tried to fixed the issues @ayounsi mentioned above. For the install i will take a look at it later.

ops-monitoring-bot added a comment.Aug 5 2022, 2:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1185.eqiad.wmnet with OS bullseye

Ladsgroup subscribed.Aug 5 2022, 2:54 PM

Manuel is also on vacation, I can take a look once you are done. Thanks.

ops-monitoring-bot added a comment.Aug 5 2022, 3:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1190.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 5 2022, 3:10 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1185.eqiad.wmnet with OS bullseye completed:

  • db1185 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051440_pt1979_2158271_db1185.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Aug 5 2022, 3:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1191.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 5 2022, 3:42 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1190.eqiad.wmnet with OS bullseye completed:

  • db1190 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051505_pt1979_2164206_db1190.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ayounsi unsubscribed.Aug 5 2022, 3:45 PM
ops-monitoring-bot added a comment.Aug 5 2022, 3:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1192.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 5 2022, 3:52 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1191.eqiad.wmnet with OS bullseye completed:

  • db1191 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051522_pt1979_2166082_db1191.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Aug 5 2022, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1193.eqiad.wmnet with OS bullseye

Papaul updated the task description. (Show Details)Aug 5 2022, 3:57 PM
ops-monitoring-bot added a comment.Aug 5 2022, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1192.eqiad.wmnet with OS bullseye completed:

  • db1192 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051551_pt1979_2174045_db1192.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Failed to get Netbox script results, try manually: https://netbox.discovery.wmnet/api/extras/job-results/3621179/
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Aug 5 2022, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1192.eqiad.wmnet with OS bullseye executed with errors:

  • db1192 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051551_pt1979_2174045_db1192.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Failed to get Netbox script results, try manually: https://netbox.discovery.wmnet/api/extras/job-results/3621179/
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 5 2022, 4:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1193.eqiad.wmnet with OS bullseye completed:

  • db1193 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051555_pt1979_2176458_db1193.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Aug 5 2022, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1194.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 5 2022, 4:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1195.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 5 2022, 4:54 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1194.eqiad.wmnet with OS bullseye completed:

  • db1194 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051625_pt1979_2184438_db1194.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Aug 5 2022, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1195.eqiad.wmnet with OS bullseye completed:

  • db1195 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208051634_pt1979_2185467_db1195.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Papaul updated the task description. (Show Details)Aug 5 2022, 5:21 PM

@Cmjohnson @Marostegui , this is complete. I created all the 1g interfaces on for the nodes in row E and F and also added those interfaces into Netbox. What left to do here is to fix the mgmt DNS for db1188 i changed it but i couldn't re-start the IDRAC because i lost connectivity to the IDRAC. So please make sure that the IDRAC IP address is 10.65.3.1 and restart the IDRAC.

Thanks.

ayounsi raised the priority of this task from Medium to High.Aug 8 2022, 10:21 AM
Aug  8 06:10:32  mr1-eqiad /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.65.2.255 from d0:8e:79:f4:1a:6a to d0:8e:79:f4:1a:5a
Aug  8 07:15:08  mr1-eqiad /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.65.2.255 from d0:8e:79:f4:1a:5a to d0:8e:79:f4:1a:6a
Aug  8 07:15:09  mr1-eqiad /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 10.65.2.255 from d0:8e:79:f4:1a:6a to d0:8e:79:f4:1a:5a

There are ongoing alerts about duplicate IP for 10.65.2.255 please fix it as soon as possible as this means instability on the management network.

ayounsi added a comment.EditedAug 10 2022, 7:26 AM

@Cmjohnson there is an outstanding diff:

Changes for 1 devices: ['lsw1-f3-eqiad.mgmt.eqiad.wmnet']

[edit interfaces ge-0/0/40]
-   description "db1195 {#2013339101795}";

And indeed there in an interface fully configured, but not connected to any device:
https://netbox.wikimedia.org/dcim/interfaces/26528/

Which has been alerting on the Netbox network report (with others)
https://netbox.wikimedia.org/extras/reports/network.Network/

Could you please have a look and clear those errors?

Of course let me know if there is any issue on the automation/process side so we can improve them as needed.

• Cmjohnson added a comment.Aug 10 2022, 5:32 PM

Sorry about that I thought all the updates merged. As far as connected to ports, all of them are now connected with no port description. I will have that fixed today.

• Cmjohnson added a comment.Aug 16 2022, 4:15 PM

Fixed the db1188 mgmt ip address

Marostegui added a comment.Aug 17 2022, 7:41 AM

Thanks everyone for woking on this! From my side db1187 and db1185 aren't reachable. @Cmjohnson can you take a look? Thanks.

Marostegui added a comment.Aug 22 2022, 9:10 AM

Any ETA for getting db1187 and db1185 online?

Marostegui changed the status of subtask T313569: Productionize db1185.eqiad.wmnet - db1195.eqiad.wmnet from Stalled to Open.Aug 22 2022, 9:11 AM
Papaul added a comment.Aug 22 2022, 7:52 PM

@Marostegui Chris is out on vacation I will take a look later to see.

ops-monitoring-bot added a comment.Aug 22 2022, 8:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1195.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2022, 8:59 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1195.eqiad.wmnet with OS bullseye executed with errors:

  • db1195 (FAIL)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 22 2022, 8:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1185.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2022, 8:59 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1185.eqiad.wmnet with OS bullseye executed with errors:

  • db1185 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 22 2022, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1185.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2022, 9:02 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1185.eqiad.wmnet with OS bullseye executed with errors:

  • db1185 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
Papaul added a comment.EditedAug 22 2022, 11:37 PM

@Jclark-ctr when you back on site can you please check the cable on db1185 looks like the cable is not connected.

papaul@asw2-a-eqiad> show interfaces descriptions | match db1185
ge-1/0/33       up    down db1185 {#23000053}
ops-monitoring-bot added a comment.Aug 22 2022, 11:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1187.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 23 2022, 12:09 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1187.eqiad.wmnet with OS bullseye completed:

  • db1187 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208222339_pt1979_2352531_db1187.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul added a comment.Aug 23 2022, 12:12 AM

@Marostegui I re-imaged db1187 it is back online. I can not access db1185 so i asked @Jclark-ctr to check the cable.

pt1979@cumin1001:~$ ping db1187
PING db1187.eqiad.wmnet (10.64.16.29) 56(84) bytes of data.
64 bytes from db1187.eqiad.wmnet (10.64.16.29): icmp_seq=1 ttl=63 time=0.243 ms
64 bytes from db1187.eqiad.wmnet (10.64.16.29): icmp_seq=2 ttl=63 time=0.290
Marostegui added a comment.Aug 23 2022, 4:48 AM

Thank you Papaul! I can access db1187 just fine now!

Marostegui added a comment.Aug 23 2022, 4:50 AM

@Papaul it seems that db1186 and db1188 are no longer accessible now.

Papaul added a comment.Aug 23 2022, 1:38 PM

@Marostegui can you access those servers using serial console and provide me with the IP addresses.

Marostegui added a comment.Aug 23 2022, 2:15 PM

db1185: 10.64.0.108
This one seems to have link ok so maybe it is not the cable?:

root@db1185:~# mii-tool eno1
eno1: negotiated 1000baseT-FD flow-control, link ok

db1186: 10.64.0.142
db1188: 10.64.16.239

Papaul added a comment.Aug 23 2022, 2:21 PM

in Netbox we have
db1186 using 10.64.0.108
db1188 using 10.64.16.238

10.64.0.142 in Netbox is not assign to any hosts
so there is IP address issues on those servers

Papaul added a comment.Aug 23 2022, 2:24 PM

db1186 is using in Netbox the IP assigned to db1185
``
10.64.0.108/22 Global Active — — db1186.eqiad.wmnet

Papaul added a comment.EditedAug 23 2022, 3:17 PM

@Marostegui db1186 and db1188 are now back online db1185 is still showing that the link is down so i will have @Jclark-ctr check the cable just to make sure.

papaul@asw2-a-eqiad> show interfaces descriptions | match db1185
ge-1/0/33       up    down db1185 {#23000053}
Marostegui added a comment.Aug 24 2022, 5:23 AM

Thank you Papaul, I can access db1186 and db1188 fine!

Jclark-ctr added a comment.Aug 24 2022, 1:50 PM

@Papaul db1185 is fixed loosed cable

Marostegui added a comment.Aug 24 2022, 1:53 PM

I can now access db1185 - thank you @Jclark-ctr.
From my side, we can close this task

Papaul added a comment.Aug 24 2022, 1:53 PM

@Jclark-ctr thanks

ops-monitoring-bot added a comment.Aug 24 2022, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1185.eqiad.wmnet with OS bullseye

Marostegui added a comment.Aug 24 2022, 1:56 PM

I am doing a reimage of db1185 as its puppet cert was revoked and it was in a weird state.

ops-monitoring-bot added a comment.Aug 24 2022, 2:25 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1185.eqiad.wmnet with OS bullseye completed:

  • db1185 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208241355_marostegui_2759693_db1185.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Marostegui closed this task as Resolved.Aug 24 2022, 2:25 PM

db1185 was reimaged successfully. Closing this. Thanks everyone for getting these hosts up!

Marostegui closed subtask T313569: Productionize db1185.eqiad.wmnet - db1195.eqiad.wmnet as Resolved.Aug 26 2022, 11:13 AM
ops-monitoring-bot added a comment.Sep 2 2022, 3:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host db1196.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 2 2022, 3:31 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host db1196.eqiad.wmnet with OS bullseye completed:

  • db1196 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209021500_pt1979_874043_db1196.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits