Page MenuHomePhabricator
Create Task
Maniphest T384966

Q3:rack/setup/install elastic1111-elastic1122, relforge1008-1010
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of elastic1123-elastic1122

Hostname / Racking / Installation Details

Hostnames: elastic1110-elastic1125 (15 total hosts)
Networking Setup: 1 10G
OS Distro: Bullseye
Sub-team Technical Contact: Brian King (IRC: inflatador) Ryan Kemper (IRC: rkemper)
Racking Proposal: Same rows as hosts being replaced works well; otherwise just try to distribute roughly evenly. Here's the existing racks/rows of the to-be-replaced hosts:

elastic1053 <-> A4
elastic1054 <-> A4
elastic1055 <-> B2
elastic1056 <-> B7
elastic1057 <-> C7
elastic1058 <-> C7
elastic1059 <-> D2
elastic1060 <-> D2
elastic1061 <-> D4
elastic1062 <-> D4
elastic1063 <-> D4
elastic1064 <-> D7

And for the relforge hosts:
(there is a gap between relforge1004 and relforge1008 due to some repurposing, see T386357 for details)
relforge1008: any row not shared by the other 2 new relforge hosts (we don't care about 1003 and 1004 as they will be decommissioned soon)
reforge1009: "
relforge1010: "

Per host setup checklist

elastic1111
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1112
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1113
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1114
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1115
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1116
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1117
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1118
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1119
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1120
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1121
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
elastic1122
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
relforge1008 (previously called elastic1123 )
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
relforge1009 (previously called elastic1124)
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
relforge1010 (previously called elastic1125)
  • Receive in system on procurement task T382905 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
elastic/cirrussearch: add cirrussearch11[11-25] to cluster, fix typoBkingoperations/puppetproduction+533 -469
relforge: set partitioning scheme for new hostsBkingoperations/puppetproduction+1 -1
elasticsearch/relforge: rename 3 elastic hosts to relforgeBkingoperations/puppetproduction+8 -1
preseed: Fix syntax for new elastic UEFI nodesMuehlenhoffoperations/puppetproduction+1 -1
elastic: 15 refresh hostsRyan Kemperoperations/puppetproduction+30 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Mar 17 2025, 10:26 AM

Change #1128364 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] preseed: Fix syntax for new elastic UEFI nodes

https://gerrit.wikimedia.org/r/1128364

gerritbot added a project: Patch-For-Review.Mar 17 2025, 10:26 AM
gerritbot added a comment.Mar 17 2025, 10:30 AM

Change #1128364 merged by Muehlenhoff:

[operations/puppet@production] preseed: Fix syntax for new elastic UEFI nodes

https://gerrit.wikimedia.org/r/1128364

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2025, 10:30 AM
Jclark-ctr mentioned this in T386946: Supermicro: Issues tracking task.Mar 18 2025, 11:20 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr added a comment.Mar 18 2025, 11:38 PM

elastic1111 , elastic1111
updated username and password to normal wiki password
Both fail provisioning

elukey subscribed.Mar 19 2025, 10:21 AM

@Jclark-ctr thanks a lot for reporting!

From the logs I see two issues for the hosts:

  1. RuntimeError: JunOS config commit failed - see above - device may need Homer run - this one seems network related and not cookbook-related, and it goes away, lemme know if it is persistent.
  2. spicerack.apiclient.APIClientResponseError: GET https://10.65.3.17/redfish/v1/UpdateService/FirmwareInventory/BMC returned HTTP 401 - This one means that the ADMIN password is wrong, it should be calvin in this case (these hosts have the up-to-date Firmware afaics, so the BMC password shouldn't be needed).

What password have you used for 2)? Since the new hosts should all have calvin as password nowadays I am planning to make it more transparent for the cookbook users: the BMC password will be asked only if calvin doesn't work, I'll try to prioritize the fix this week.

elukey added a comment.Mar 19 2025, 10:22 AM

I've ran the cookbook as cookbook sre.hosts.provision elastic1111 --no-switch --no-users --uefi and it worked nicely, so I think the issues are mostly what it has been outlined above. @Jclark-ctr let's sync on IRC when you are online :)

bking claimed this task.Mar 19 2025, 5:23 PM

@Jclark-ctr

Per IRC mention in #wikimedia-dcops: sorry to steal this one from you, but I just wanted to make sure you didn't do extra work. We need to remove the last 3 elastic hosts in this ticket and use them to backfill relforge (ref T387782). We should be able to get this one back to you by the end of the week with the proper Puppet patches and a separate provisioning ticket for relforge hosts.

Apologies for not doing this earlier, we are a bit swamped with the Opensearch migration ( T370147 ).

ops-monitoring-bot added a comment.Mar 19 2025, 8:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1111.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Mar 19 2025, 8:51 PM
ops-monitoring-bot added a comment.Mar 19 2025, 9:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1112.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Mar 20 2025, 8:04 PM
Gehel edited projects, added: Data-Platform-SRE (2025.03.22 - 2025.04.11); removed: Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:56 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.
ops-monitoring-bot added a comment.Mar 21 2025, 1:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1111.eqiad.wmnet with OS bullseye

bking mentioned this in T380752: Migrate Relforge to Opensearch.Mar 21 2025, 1:45 PM
ops-monitoring-bot added a comment.Mar 21 2025, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1111.eqiad.wmnet with OS bullseye completed:

  • elastic1111 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211345_jclark_1877401_elastic1111.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 1:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1112.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1113.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 2:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1112.eqiad.wmnet with OS bullseye completed:

  • elastic1112 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211413_jclark_1906803_elastic1112.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1114.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1113.eqiad.wmnet with OS bullseye completed:

  • elastic1113 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211416_jclark_1907060_elastic1113.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 2:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1115.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 2:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1116.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 2:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1114.eqiad.wmnet with OS bullseye completed:

  • elastic1114 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211443_jclark_1914330_elastic1114.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 3:02 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1115.eqiad.wmnet with OS bullseye completed:

  • elastic1115 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211446_jclark_1914582_elastic1115.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1117.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1118.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1120.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 3:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1122.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 3:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1116.eqiad.wmnet with OS bullseye completed:

  • elastic1116 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211508_jclark_1927929_elastic1116.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 3:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1118.eqiad.wmnet with OS bullseye completed:

  • elastic1118 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211525_jclark_1956451_elastic1118.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Mar 21 2025, 3:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1121.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 3:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1117.eqiad.wmnet with OS bullseye completed:

  • elastic1117 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211527_jclark_1956436_elastic1117.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Mar 21 2025, 3:42 PM
Jclark-ctr updated the task description. (Show Details)
bking updated Other Assignee, added: Jclark-ctr.Mar 21 2025, 3:46 PM
ops-monitoring-bot added a comment.Mar 21 2025, 3:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1120.eqiad.wmnet with OS bullseye completed:

  • elastic1120 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211530_jclark_1956517_elastic1120.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Mar 21 2025, 3:52 PM
Jclark-ctr updated the task description. (Show Details)
bking renamed this task from Q3:rack/setup/install elastic1111-elastic1125 to Q3:rack/setup/install elastic1111-elastic1122, relforge1008-1010.Mar 21 2025, 3:53 PM
bking updated the task description. (Show Details)
gerritbot added a comment.Mar 21 2025, 3:56 PM

Change #1130145 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch/relforge: rename 3 elastic hosts to relforge

https://gerrit.wikimedia.org/r/1130145

gerritbot added a project: Patch-For-Review.Mar 21 2025, 3:56 PM
ops-monitoring-bot added a comment.Mar 21 2025, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1121.eqiad.wmnet with OS bullseye completed:

  • elastic1121 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211553_jclark_1993743_elastic1121.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Mar 21 2025, 4:09 PM
gerritbot added a comment.Mar 21 2025, 4:10 PM

Change #1130145 merged by Bking:

[operations/puppet@production] elasticsearch/relforge: rename 3 elastic hosts to relforge

https://gerrit.wikimedia.org/r/1130145

gerritbot added a comment.Mar 21 2025, 4:26 PM

Change #1130150 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] relforge: set partitioning scheme for new hosts

https://gerrit.wikimedia.org/r/1130150

ops-monitoring-bot added a comment.Mar 21 2025, 4:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1122.eqiad.wmnet with OS bullseye executed with errors:

  • elastic1122 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console elastic1122.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Mar 21 2025, 4:29 PM

Change #1130150 merged by Bking:

[operations/puppet@production] relforge: set partitioning scheme for new hosts

https://gerrit.wikimedia.org/r/1130150

Maintenance_bot removed a project: Patch-For-Review.Mar 21 2025, 4:31 PM
ops-monitoring-bot added a comment.Mar 21 2025, 4:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1122.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 21 2025, 5:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1122.eqiad.wmnet with OS bullseye completed:

  • elastic1122 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503211646_jclark_2074348_elastic1122.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Mar 21 2025, 5:00 PM
Jclark-ctr updated the task description. (Show Details)Mar 21 2025, 5:07 PM
elukey added a comment.Mar 24 2025, 9:17 AM

@Jclark-ctr elastic1119 should be ready for reimage now (I've set UEFI in provisioning, hope it was the right one). The error is tracked in T386946#10666670, I hope to find a long term solution.

bking updated the task description. (Show Details)Mar 24 2025, 1:58 PM
bking removed bking as the assignee of this task.Mar 24 2025, 2:25 PM
bking updated Other Assignee, removed: Jclark-ctr.
bking mentioned this in Unknown Object (Task).Mar 24 2025, 4:07 PM
ops-monitoring-bot added a comment.Mar 24 2025, 8:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host relforge1008.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2025, 8:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host relforge1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2025, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host relforge1009.eqiad.wmnet with OS bullseye completed:

  • relforge1009 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503242042_jclark_3354071_relforge1009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Mar 24 2025, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host relforge1008.eqiad.wmnet with OS bullseye completed:

  • relforge1008 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503242048_jclark_3352745_relforge1008.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Mar 24 2025, 10:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1119.eqiad.wmnet with OS bullseye

bking assigned this task to Jclark-ctr.Mar 24 2025, 10:18 PM
ops-monitoring-bot added a comment.Mar 24 2025, 11:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host elastic1119.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2025, 11:23 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host elastic1119.eqiad.wmnet with OS bullseye executed with errors:

  • elastic1119 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console elastic1119.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr updated the task description. (Show Details)Mar 24 2025, 11:50 PM
ops-monitoring-bot added a comment.Mar 25 2025, 12:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host elastic1119.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 25 2025, 12:43 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host elastic1119.eqiad.wmnet with OS bullseye completed:

  • elastic1119 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202503250028_jhancock_1950106_elastic1119.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Mar 25 2025, 2:00 PM
Jclark-ctr added a comment.Mar 25 2025, 2:02 PM

@bking these are finished except relforge1010 we are waiting on response from supermicro

bking mentioned this in T387356: Q3:rack/setup/install elastic112[345].Mar 27 2025, 2:21 PM
bking mentioned this in T391118: Bring cirrussearch1111-1122, 1123-25 into production.Apr 4 2025, 3:28 PM
Gehel added a subtask: T391118: Bring cirrussearch1111-1122, 1123-25 into production.Apr 8 2025, 2:54 PM
Jclark-ctr added a comment.Apr 9 2025, 10:16 PM

relforge1010 is still pending RMA with supermicro.

Gehel edited projects, added: Data-Platform-SRE (2025.04.12 - 2025.05.02); removed: Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:13 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.
ops-monitoring-bot added a comment.Apr 25 2025, 6:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host relforge1010.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 25 2025, 7:27 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host relforge1010.eqiad.wmnet with OS bullseye completed:

  • relforge1010 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504251912_jclark_2903243_relforge1010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Apr 25 2025, 7:29 PM
Jclark-ctr updated the task description. (Show Details)

@bking thanks for patience with this one. just wrapped up final server

Gehel moved this task from Blocked/Waiting to Done on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.Apr 29 2025, 7:58 AM
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.May 1 2025, 4:25 PM
gerritbot added a comment.May 5 2025, 2:47 PM

Change #1141914 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic/cirrussearch: add cirrussearch11[11-25] to cluster, fix typo

https://gerrit.wikimedia.org/r/1141914

gerritbot added a project: Patch-For-Review.May 5 2025, 2:47 PM
bking changed the status of subtask T391118: Bring cirrussearch1111-1122, 1123-25 into production from Open to In Progress.May 5 2025, 3:02 PM
gerritbot added a comment.May 5 2025, 4:30 PM

Change #1141914 merged by Bking:

[operations/puppet@production] elastic/cirrussearch: add cirrussearch11[11-25] to cluster, fix typo

https://gerrit.wikimedia.org/r/1141914

Maintenance_bot removed a project: Patch-For-Review.May 5 2025, 5:30 PM
bking closed subtask T391118: Bring cirrussearch1111-1122, 1123-25 into production as Resolved.May 8 2025, 6:04 PM
bking closed subtask T389957: Bring relforge100[89] into production as Resolved.Jun 11 2025, 1:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits