Page MenuHomePhabricator
Create Task
Maniphest T363209

Q4:rack/setup/install kafka-main200[6789] & kafka-main2010
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of X

Hostname / Racking / Installation Details

Hostnames: kafka-main200[6789] and kafka-main2010
Racking Proposal: Row A (A4 if possible), Row B (B4 if possible), Row C (C7 if possible), Row D (D7 if possible), Row D (D4 if possible)
Networking Setup: 1 connection, 1G
VLAN: Private, AAAA records: Y, Additional IP records (Cassandra)? No
Partitioning/Raid: HW Raid: N, Partman recipe and/or desired Raid Level: raid10-4dev.cfg
OS Distro: Bullseye
Sub-team Technical Contact: @akosiaris

Per host setup checklist

kafka-main2006:
  • Receive in system on procurement task T361362 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
kafka-main2007:
  • Receive in system on procurement task T361362 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
kafka-main2008:
  • Receive in system on procurement task T361362 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
kafka-main2009:
  • Receive in system on procurement task T361362 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
kafka-main2010:
  • Receive in system on procurement task T361362 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Other Assignee
Papaul
SubjectRepoBranchLines +/-
Add back reuse option after testingoperations/puppetproduction+2 -1
The resue option is making re-image to fail so testing without it.operations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 23 2024, 7:19 PM
RobH renamed this task from Q#:rack/setup/install X to Q#:rack/setup/install kafka-main200[6789] & kafka-main2010.Apr 23 2024, 7:21 PM
RobH renamed this task from Q#:rack/setup/install kafka-main200[6789] & kafka-main2010 to Q4:rack/setup/install kafka-main200[6789] & kafka-main2010.
RobH mentioned this in T363210: serviceops kafka-main200[6789] and kafka-main2010 implementation tracking.
RobH added a project: ops-codfw.
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
Maintenance_bot added a project: SRE.Apr 23 2024, 7:29 PM
Jhancock.wm claimed this task.Mon, May 6, 2:50 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Wed, May 8, 2:56 PM
ops-monitoring-bot added a comment.Thu, May 9, 3:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 9, 3:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 9, 3:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 9, 3:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 9, 3:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2010.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 9, 4:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 9, 4:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2008.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2008 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2008.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 9, 4:56 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2010.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2010 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2010.codfw.wmnet to get a root shellbut depending on the failure this may not work.
Jhancock.wm updated the task description. (Show Details)Thu, May 9, 10:33 PM
Jhancock.wm updated Other Assignee, added: Papaul.Mon, May 13, 1:36 PM
Jhancock.wm added a subscriber: Papaul.Mon, May 13, 3:13 PM

@Papaul, This was the last screen I got. The servers all have the OS installed and it failed at the certificate stage. I think it's cause I used python 7 instead of 5. when I attempt to retry with 5, it fails.

@cumin2002:~$ sudo secure-cookbook sre.hosts.reimage  -t T363209 --os bullseye kafka-main2006 -p 5 --new. 
==> ATTENTION: Destructive action for kafka-main2006. Proceed?
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
Starting reimage on kafka-main2006. You can check progress via serial console or by running `install-console kafka-main2006.codfw.wmnet` on any cumin host
==> Host kafka-main2006.codfw.wmnet was found in PuppetDB but --new was set. Are you sure you want to proceed? The --new option will be unset
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
The option --new has been unset
Exception raised while initializing the Cookbook sre.hosts.reimage:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 199, in run
  runner = self.instance.get_runner(args)
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 101, in get_runner
  return ReimageRunner(args, self.spicerack)
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 181, in __init__
  self.new_puppet_server = self._get_puppet_server()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 227, in _get_puppet_server
  has_puppet7 = self.puppet_server.hiera_lookup(self.fqdn, "profile::puppet::agent::force_puppet7")
File "/usr/lib/python3/dist-packages/spicerack/puppet.py", line 599, in hiera_lookup
  result = self.server_host.run_sync(command, is_safe=True, print_output=False, print_progress_bars=False)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 514, in run_sync
  return self._execute(
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 720, in _execute
  raise RemoteExecutionError(ret, "Cumin execution failed", worker.get_results())
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 199, in run
  runner = self.instance.get_runner(args)
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 101, in get_runner
  return ReimageRunner(args, self.spicerack)
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 181, in __init__
  self.new_puppet_server = self._get_puppet_server()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 227, in _get_puppet_server
  has_puppet7 = self.puppet_server.hiera_lookup(self.fqdn, "profile::puppet::agent::force_puppet7")
File "/usr/lib/python3/dist-packages/spicerack/puppet.py", line 599, in hiera_lookup
  result = self.server_host.run_sync(command, is_safe=True, print_output=False, print_progress_bars=False)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 514, in run_sync
  return self._execute(
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 720, in _execute
  raise RemoteExecutionError(ret, "Cumin execution failed", worker.get_results())
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
MoritzMuehlenhoff subscribed.Mon, May 13, 3:16 PM

All insetup roles default to Puppet 7 these days (as does the kafka-main roler itself), so these should be installed with Puppet 7.

MoritzMuehlenhoff added a comment.Mon, May 13, 3:17 PM

I think the reason the installation failed is because there is no entry in site.pp yet.

ops-monitoring-bot added a comment.Wed, May 15, 1:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, May 15, 1:02 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2006.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2006.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, May 15, 1:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, May 15, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2006.codfw.wmnet with OS bullseye completed:

  • kafka-main2006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405151325_jhancock_2409379_kafka-main2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Wed, May 15, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, May 15, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, May 15, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, May 15, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2010.codfw.wmnet with OS bullseye

Jhancock.wm updated the task description. (Show Details)Wed, May 15, 2:28 PM
ops-monitoring-bot added a comment.Wed, May 15, 2:39 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2007.codfw.wmnet with OS bullseye completed:

  • kafka-main2007 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405151422_jhancock_2467812_kafka-main2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Wed, May 15, 2:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2008.codfw.wmnet with OS bullseye completed:

  • kafka-main2008 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405151424_jhancock_2468073_kafka-main2008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Wed, May 15, 2:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2010.codfw.wmnet with OS bullseye completed:

  • kafka-main2010 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405151426_jhancock_2468124_kafka-main2010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)Wed, May 15, 2:49 PM

2009 is on timeout until I can take another crack at it. stuck on this even thought the rest passed.

6/10, retrying in 160.00s] Attempt to run 'spicerack.puppet.PuppetServer.wait_for_csr' raised: The puppet server has no CSR for kafka-main2009.codfw.wmnet
[7/10, retrying in 320.00s] Attempt to run 'spicerack.puppet.PuppetServer.wait_for_csr' raised: The puppet server has no CSR for kafka-main2009.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 15, 3:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 16, 12:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, May 16, 1:12 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, May 17, 3:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, May 17, 4:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
Jhancock.wm reassigned this task from Jhancock.wm to Papaul.Fri, May 17, 6:39 PM
Jhancock.wm subscribed.

@Papaul I'm still having trouble with the same spot as noted before. Can you take a look at it?

Papaul added a comment.Fri, May 17, 7:10 PM

Thank you will do

ops-monitoring-bot added a comment.Fri, May 17, 7:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, May 17, 7:42 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, May 17, 7:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

gerritbot added a comment.Fri, May 17, 7:55 PM

Change #1032857 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] The resue option is making re-image to fail so testing without it.

https://gerrit.wikimedia.org/r/1032857

gerritbot added a project: Patch-For-Review.Fri, May 17, 7:56 PM
gerritbot added a comment.Fri, May 17, 8:00 PM

Change #1032857 merged by Papaul:

[operations/puppet@production] The resue option is making re-image to fail so testing without it.

https://gerrit.wikimedia.org/r/1032857

Maintenance_bot removed a project: Patch-For-Review.Fri, May 17, 8:30 PM
ops-monitoring-bot added a comment.Fri, May 17, 9:10 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
gerritbot added a comment.Fri, May 17, 9:15 PM

Change #1032871 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add back reuse option after testing

https://gerrit.wikimedia.org/r/1032871

gerritbot added a project: Patch-For-Review.Fri, May 17, 9:16 PM
gerritbot added a comment.Fri, May 17, 10:06 PM

Change #1032871 abandoned by Papaul:

[operations/puppet@production] Add back reuse option after testing

Reason:

https://gerrit.wikimedia.org/r/1032871

ops-monitoring-bot added a comment.Fri, May 17, 10:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Fri, May 17, 10:31 PM
ops-monitoring-bot added a comment.Fri, May 17, 11:08 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, May 17, 11:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sat, May 18, 12:04 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kafka-main2009.codfw.wmnet with OS bullseye completed:

  • kafka-main2009 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405172346_pt1979_1649208_kafka-main2009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul added a comment.Sat, May 18, 12:04 AM

@Jhancock.wm thank you for working on this. Like I mentioned to you this morning the reason kafka-main2009 was failing is because it was contacting the wrong puppet server for cert request. (see below) what I did was to delete the cert req on puppetmaster and restart the re-image.

@akosiaris please see my comment above. We have been having this issue for some time now where the node send the cert req to puppetmaster1001 even though when you lunch the cookbook you tell it to use puppet 7. Thanks. This is all yours.

pt1979@puppetmaster1001:~$ sudo puppet cert --list
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
  "kafka-main2009.codfw.wmnet" (SHA256) B5:5F:81:9E:67:18:D4:E1:CF:AA:C7:A7:8B:DE:B1:8B:D3:04:C3:E7:04:64:D1:6A:27:04:23:EF:24:E3:37:46
pt1979@puppetmaster1001:~$ sudo puppet cert clean kafka-main2009.codfw.wmnet
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Removing file Puppet::SSL::CertificateRequest kafka-main2009.codfw.wmnet at '/var/lib/puppet/server/ssl/ca/requests/kafka-main2009.codfw.wmnet.pem'
Papaul closed this task as Resolved.Sat, May 18, 12:04 AM
Papaul updated the task description. (Show Details)
ops-monitoring-bot added a comment.Mon, May 20, 1:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 1:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 1:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 1:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 1:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 2:18 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2008.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2008 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2008.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, May 20, 2:19 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2009 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2009.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, May 20, 2:19 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2010 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2010.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, May 20, 3:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 3:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 3:51 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye executed with errors:

  • kafka-main2010 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" kafka-main2010.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, May 20, 4:21 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2009.codfw.wmnet with OS bullseye completed:

  • kafka-main2009 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405201604_akosiaris_1154020_kafka-main2009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, May 20, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2006.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2007.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 4:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2008.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, May 20, 5:21 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2007.codfw.wmnet with OS bullseye completed:

  • kafka-main2007 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405201703_akosiaris_1163483_kafka-main2007.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Mon, May 20, 5:25 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2010.codfw.wmnet with OS bullseye completed:

  • kafka-main2010 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405201709_akosiaris_1163663_kafka-main2010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, May 20, 5:30 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2008.codfw.wmnet with OS bullseye completed:

  • kafka-main2008 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405201713_akosiaris_1163582_kafka-main2008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, May 20, 5:33 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host kafka-main2006.codfw.wmnet with OS bullseye completed:

  • kafka-main2006 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405201716_akosiaris_1163423_kafka-main2006.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
akosiaris added a comment.Tue, May 21, 9:50 AM

As an update, I had to reimage these servers as I had messed up the original recipe.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL