Page MenuHomePhabricator
Create Task
Maniphest T388610

Migrate production Elastic clusters to Opensearch
Open, In Progress, MediumPublic
Actions

Description

As we have successfully migrated our smaller clusters from Elastic -> OpenSearch (see T380752 and T387904), the time has come to migrate our production clusters from Elastic to OpenSearch.

Current status

Progress by number of hosts (updated at end of shift):
overall: 53/116 hosts migrated
codfw: 53/61 hosts migrated

  • by row (non-masters only, we will do masters last):
    • A
    • B
    • C in progress
    • D
    • currently broken:
      • cirrussearch2078 - continually failing PXE, ref T392644
      • cirrussearch2091 - continually failing PXE and has history of hardware issues (ref T391639)

eqiad: 0/55 hosts migrated

Open patches
See the "elastic-2-opensearch" Gerrit topic

Checking progress manually

You can check site.pp in the Puppet repo . See how many hosts are using the old elasticsearch::cirrus role vs. the new cirrus::opensearch role

If you have cumin access, you can also run the following commands: sudo cumin O:elasticsearch::cirrus for the old role, sudo cumin O:cirrus::opensearch for the new role

Creating this ticket to:

  • Create a plan. Documented here
  • Notify stakeholders
  • Migrate each host from Elastic to Opensearch and confirm operation
    • also rename like so: s/elastic/cirrussearch
  • Update docs
  • Roll back any config we temporarily changed for the migration T391350

Details

Other Assignee
RKemper
SubjectRepoBranchLines +/-
cirrussearch: allow any cirrussearch host to join clusteroperations/puppetproduction+273 -1
cirrus: add to-be-renamed mastersoperations/puppetproduction+22 -2
cirrussearch: remove remaining elastic hostsoperations/puppetproduction+0 -12
conftool: Remove mentions to elastic2095operations/puppetproduction+0 -1
conftool: Remove no longer existent elastic hostsoperations/puppetproduction+0 -3
conftool: Remove mentions to elastic2094operations/puppetproduction+0 -1
conftool: Remove mentions to elastic2064operations/puppetproduction+0 -1
cirrus: migrate elastic2061->cirrussearch2061operations/puppetproduction+2 -0
cirrussearch: Change whitespace from 4 to 2operations/puppetproduction+651 -651
cirrussearch: fix typo in regex.yamloperations/puppetproduction+1 -1
cirrussearch: prepare for eqiad migrationoperations/puppetproduction+16 -6
cirrussearch: add more newly-reimaged hosts to conftooloperations/puppetproduction+6 -5
cirrussearch: update conftool with correct pool dataoperations/puppetproduction+4 -5
sre.elasticsearch.rolling-operation: restart ferm after host renameoperations/cookbooksmaster+14 -3
cirrussearch: fix some omega vs psi assignmentsoperations/puppetproduction+4 -4
cirrussearch: Add newly-reimaged hosts to conftooloperations/puppetproduction+12 -10
cirrus: (WIP) support rename elastic->cirrussearchoperations/puppetproduction+28 -2
cirrussearch: fix row B regexoperations/puppetproduction+1 -1
cirrussearch: Enable row B for OpenSearch migration.operations/puppetproduction+8 -8
cirrussearch: Add new master-eligiblesoperations/puppetproduction+4 -0
cirrussearch: Add row D non-master hosts to elasticsearch poolsoperations/puppetproduction+8 -7
cirrussearch: Add row C hostsoperations/puppetproduction+3 -6
cirrussearch: remove no-longer-existing master-eligibles.operations/puppetproduction+0 -11
cirrussearch: Add new master-eligiblesoperations/puppetproduction+1 -0
cirrussearch: add the firewall suffixoperations/puppetproduction+1 -1
cirrussearch: Use new insetup roleoperations/puppetproduction+1 -1
cirrussearch2014: move to insetupoperations/puppetproduction+1 -1
cirrussearch: temporarily add cirrussearch2014 so we can renameoperations/puppetproduction+1 -1
temporarily add cirrussearch2014 as a hostoperations/puppetproduction+6 -1
cirrus: fix some incorrect serversoperations/puppetproduction+5 -4
cirrussearch: update conftool data with new hostnames (row A)operations/puppetproduction+13 -1
cirrus: enable opensearch roles in row Doperations/puppetproduction+1 -5
cirrussearch: remove all references to non-row-A hostsoperations/puppetproduction+40 -40
cirrussearch: add row D host as cirrussearchoperations/puppetproduction+5 -1
cirrussearch: fix rack a7 regexoperations/puppetproduction+1 -1
elastic: remove row A worker hostsoperations/puppetproduction+0 -11
cirrussearch: Add row A hosts to new cirrussearch roleoperations/puppetproduction+5 -7
cirrussearch: Add regex data for cirrussearch hostsoperations/puppetproduction+43 -43
elasticsearch rolling-operation: add arguments for rename & reimage cookbooksoperations/cookbooksmaster+11 -2
cirrus: disable completion indices in codfwoperations/puppetproduction+1 -0
cirrussearch: Add cumin aliasesoperations/puppetproduction+19 -5
cirrussearch: Enable new role with existing aliasoperations/puppetproduction+1 -1
cirrussearch: Add row/rack hieradataoperations/puppetproduction+94 -0
cirrussearch: add puppet 7 hieradata to DC-specific configoperations/puppetproduction+2 -0
cirrussearch: add second canary for OpenSearch migrationoperations/puppetproduction+16 -18
cirrussearch: add second canary for OpenSearch migrationoperations/puppetproduction+13 -16
cirrussearch: add cirrussearch2055 to existing elastic clusteroperations/puppetproduction+3 -3
cirrussearch: use correct mandatory pluginsoperations/puppetproduction+6 -6
cirrussearch: fix s3-related variable pathsoperations/puppetproduction+4 -6
cirrus: fix regex to match new hostoperations/puppetproduction+1 -1
cirrussearch: fix variable scope for row awarenessoperations/puppetproduction+2 -2
cirrussearch: Fix variable paths and reorganizeoperations/puppetproduction+12 -11
cirrussearch: fix hieradata for codfwoperations/puppetproduction+2 -2
cirrussearch: set correct hieradata for opensearch::instancesoperations/puppetproduction+2 -2
cirrussearch: match filename to lookup paths (again)operations/puppetproduction+0 -0
cirrussearch: fix typo in hieradataoperations/puppetproduction+1 -1
cirrussearch: Add host-specific hieradata for Puppet 7operations/puppetproduction+1 -0
cirrussearch: set correct cluster name for LVSoperations/puppetproduction+1 -1
cirrussearch: use correct filename for opensearch configoperations/puppetproduction+0 -0
cirrussearch: don't include ipip LVS profileoperations/puppetproduction+0 -1
elastic/cirrussearch: use correct role nameoperations/puppetproduction+6 -9
cirrus: explicitly route search traffic to codfwoperations/mediawiki-configmaster+3 -3
cirrus: switch search traffic back to multi-DCoperations/mediawiki-configmaster+3 -3
cirrus: test rename of single host elastic2055operations/puppetproduction+101 -6
elastic: enable performance governor for selected hostsoperations/puppetproduction+44 -0
WIP: Begin CODFW transition to OpenSearchoperations/puppetproduction+87 -0
Move cirrus traffic to eqiad for platform upgradeoperations/mediawiki-configmaster+2 -2
elastic: Change first batch of prod elastic hosts to OpenSearchoperations/puppetproduction+6 -6
cirrussearch: use EFI for soon-to-be-reimaged Elastic hostsoperations/puppetproduction+5 -0
elastic/cirrussearch: begin production migrationoperations/puppetproduction+47 -5
cirrus: explicitly route search traffic to eqiadoperations/mediawiki-configmaster+1 -1
cirrus: create symlink from /etc/elasticsearch to /etc/opensearchoperations/puppetproduction+10 -0
Cirrus: Prepare production hosts for OpenSearchoperations/puppetproduction+462 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Wed, Apr 23, 2:52 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2101 to cirrussearch2101 completed:

  • elastic2101 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Wed, Apr 23, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2101.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, Apr 23, 3:29 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2102 to cirrussearch2102 completed:

  • elastic2102 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Wed, Apr 23, 3:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2102.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Wed, Apr 23, 4:10 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2101.codfw.wmnet with OS bullseye completed:

  • cirrussearch2101 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504231547_bking_2035727_cirrussearch2101.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Wed, Apr 23, 4:17 PM

Change #1138406 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: add more newly-reimaged hosts to conftool

https://gerrit.wikimedia.org/r/1138406

ops-monitoring-bot added a comment.Wed, Apr 23, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2102.codfw.wmnet with OS bullseye completed:

  • cirrussearch2102 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504231609_bking_2087982_cirrussearch2102.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Apr 23, 4:51 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2113 to cirrussearch2113 completed:

  • elastic2113 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Wed, Apr 23, 4:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2113.codfw.wmnet with OS bullseye

gerritbot added a comment.Wed, Apr 23, 5:33 PM

Change #1138406 merged by Bking:

[operations/puppet@production] cirrussearch: add more newly-reimaged hosts to conftool

https://gerrit.wikimedia.org/r/1138406

ops-monitoring-bot added a comment.Wed, Apr 23, 6:09 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2113.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2113 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2113.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Wed, Apr 23, 6:55 PM

Change #1138446 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: Add new master-eligibles

https://gerrit.wikimedia.org/r/1138446

gerritbot added a comment.Wed, Apr 23, 6:57 PM

Change #1137069 merged by Bking:

[operations/puppet@production] cirrussearch: prepare for eqiad migration

https://gerrit.wikimedia.org/r/1137069

bking updated the task description. (Show Details)Wed, Apr 23, 6:58 PM
gerritbot added a comment.Wed, Apr 23, 7:06 PM

Change #1138449 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: fix typo in regex.yaml

https://gerrit.wikimedia.org/r/1138449

gerritbot added a comment.Wed, Apr 23, 7:08 PM

Change #1138449 merged by Bking:

[operations/puppet@production] cirrussearch: fix typo in regex.yaml

https://gerrit.wikimedia.org/r/1138449

bking updated the task description. (Show Details)Wed, Apr 23, 8:09 PM
gerritbot added a comment.Wed, Apr 23, 9:38 PM

Change #1138446 merged by Ryan Kemper:

[operations/puppet@production] cirrussearch: Change whitespace from 4 to 2

https://gerrit.wikimedia.org/r/1138446

gerritbot added a comment.Wed, Apr 23, 9:42 PM

Change #1138479 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cirrus: migrate elastic2061->cirrussearch2061

https://gerrit.wikimedia.org/r/1138479

gerritbot added a comment.Wed, Apr 23, 9:48 PM

Change #1138479 merged by Ryan Kemper:

[operations/puppet@production] cirrus: migrate elastic2061->cirrussearch2061

https://gerrit.wikimedia.org/r/1138479

ops-monitoring-bot added a comment.Wed, Apr 23, 9:55 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2061 to cirrussearch2061 completed:

  • elastic2061 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Wed, Apr 23, 9:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2061.codfw.wmnet with OS bullseye

bking updated the task description. (Show Details)Wed, Apr 23, 10:06 PM
gerritbot added a comment.Wed, Apr 23, 10:18 PM

Change #1138489 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] cirrus: add to-be-renamed masters

https://gerrit.wikimedia.org/r/1138489

bking updated the task description. (Show Details)Wed, Apr 23, 10:19 PM
ops-monitoring-bot added a comment.Wed, Apr 23, 10:46 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2061.codfw.wmnet with OS bullseye completed:

  • cirrussearch2061 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504232221_bking_2471203_cirrussearch2061.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Thu, Apr 24, 9:23 AM

Change #1138687 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] conftool: Remove mentions to elastic2064

https://gerrit.wikimedia.org/r/1138687

gerritbot added a comment.Thu, Apr 24, 9:26 AM

Change #1138687 merged by Vgutierrez:

[operations/puppet@production] conftool: Remove mentions to elastic2064

https://gerrit.wikimedia.org/r/1138687

gerritbot added a comment.Thu, Apr 24, 9:34 AM

Change #1138693 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] conftool: Remove mentions to elastic2094

https://gerrit.wikimedia.org/r/1138693

gerritbot added a comment.Thu, Apr 24, 9:36 AM

Change #1138693 merged by Vgutierrez:

[operations/puppet@production] conftool: Remove mentions to elastic2094

https://gerrit.wikimedia.org/r/1138693

gerritbot added a comment.Thu, Apr 24, 9:46 AM

Change #1138695 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] conftool: Remove no longer existent elastic hosts

https://gerrit.wikimedia.org/r/1138695

gerritbot added a comment.Thu, Apr 24, 9:56 AM

Change #1138695 merged by Vgutierrez:

[operations/puppet@production] conftool: Remove no longer existent elastic hosts

https://gerrit.wikimedia.org/r/1138695

gerritbot added a comment.Thu, Apr 24, 10:02 AM

Change #1138704 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] conftool: Remove mentions to elastic2095

https://gerrit.wikimedia.org/r/1138704

gerritbot added a comment.Thu, Apr 24, 10:07 AM

Change #1138704 merged by Vgutierrez:

[operations/puppet@production] conftool: Remove mentions to elastic2095

https://gerrit.wikimedia.org/r/1138704

gerritbot added a comment.Thu, Apr 24, 2:20 PM

Change #1138804 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: remove remaining elastic hosts

https://gerrit.wikimedia.org/r/1138804

gerritbot added a comment.Thu, Apr 24, 2:30 PM

Change #1138804 merged by Bking:

[operations/puppet@production] cirrussearch: remove remaining elastic hosts

https://gerrit.wikimedia.org/r/1138804

gerritbot added a comment.Thu, Apr 24, 4:55 PM

Change #1138489 merged by Bking:

[operations/puppet@production] cirrus: add to-be-renamed masters

https://gerrit.wikimedia.org/r/1138489

ops-monitoring-bot added a comment.Thu, Apr 24, 5:05 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2073 to cirrussearch2073 completed:

  • elastic2073 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Thu, Apr 24, 5:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2073.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Apr 24, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2073.codfw.wmnet with OS bullseye completed:

  • cirrussearch2073 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504241732_bking_3649904_cirrussearch2073.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, Apr 24, 7:28 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2076 to cirrussearch2076 completed:

  • elastic2076 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Thu, Apr 24, 7:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2076.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Apr 24, 8:20 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2076.codfw.wmnet with OS bullseye completed:

  • cirrussearch2076 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504241955_bking_3795261_cirrussearch2076.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, Apr 24, 8:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Apr 24, 9:11 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Thu, Apr 24, 9:34 PM

Change #1138925 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] cirrussearch: allow any cirrussearch host to join cluster

https://gerrit.wikimedia.org/r/1138925

ops-monitoring-bot added a comment.Thu, Apr 24, 9:35 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2080 to cirrussearch2080 completed:

  • elastic2080 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Thu, Apr 24, 9:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2080.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Apr 24, 9:39 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2080.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2080 (FAIL)
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2080.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, Apr 24, 9:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2080.codfw.wmnet with OS bullseye

gerritbot added a comment.Thu, Apr 24, 9:49 PM

Change #1138925 merged by Bking:

[operations/puppet@production] cirrussearch: allow any cirrussearch host to join cluster

https://gerrit.wikimedia.org/r/1138925

ops-monitoring-bot added a comment.Thu, Apr 24, 10:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Apr 24, 10:09 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, Apr 24, 10:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Apr 24, 10:34 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2080.codfw.wmnet with OS bullseye completed:

  • cirrussearch2080 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504242207_bking_3934146_cirrussearch2080.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, Apr 24, 11:32 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Apr 25, 1:49 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2081 to cirrussearch2081 completed:

  • elastic2081 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Fri, Apr 25, 1:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2081.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 2:46 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2081.codfw.wmnet with OS bullseye completed:

  • cirrussearch2081 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504251429_bking_724557_cirrussearch2081.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Fri, Apr 25, 3:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 4:00 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2083 to cirrussearch2083 completed:

  • elastic2083 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Fri, Apr 25, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2083.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 4:45 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2083.codfw.wmnet with OS bullseye completed:

  • cirrussearch2083 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504251626_bking_866677_cirrussearch2083.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Fri, Apr 25, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Apr 25, 5:13 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2084 to cirrussearch2084 completed:

  • elastic2084 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Fri, Apr 25, 5:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2084.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 6:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 6:23 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Apr 25, 6:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 6:57 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2084.codfw.wmnet with OS bullseye completed:

  • cirrussearch2084 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504251831_bking_940120_cirrussearch2084.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Fri, Apr 25, 7:39 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Apr 25, 8:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

bking updated the task description. (Show Details)Fri, Apr 25, 8:41 PM
bking added a subtask: T392644: cirrussearch2078 (R440 Config D, Row/Rack B2) unable to PXE boot.Fri, Apr 25, 8:45 PM
bking updated the task description. (Show Details)
ops-monitoring-bot added a comment.Fri, Apr 25, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2078 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2078.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Apr 25, 8:54 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2086 to cirrussearch2086 completed:

  • elastic2086 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Fri, Apr 25, 8:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2086.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Fri, Apr 25, 8:58 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2086.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2086 (FAIL)
    • Failed to migrate host to the new VLAN, sre.hosts.move-vlan cookbook returned 94
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2086.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Apr 25, 9:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2086.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sat, Apr 26, 3:21 AM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2086.codfw.wmnet with OS bullseye completed:

  • cirrussearch2086 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504260254_bking_1224546_cirrussearch2086.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Apr 28, 1:53 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2092 to cirrussearch2092 completed:

  • elastic2092 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Mon, Apr 28, 1:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2092.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Apr 28, 2:43 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2092.codfw.wmnet with OS bullseye completed:

  • cirrussearch2092 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504281422_bking_917112_cirrussearch2092.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Apr 28, 3:31 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2093 to cirrussearch2093 completed:

  • elastic2093 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Mon, Apr 28, 3:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2093.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Apr 28, 3:59 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2093.codfw.wmnet with OS bullseye executed with errors:

  • cirrussearch2093 (FAIL)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2093.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Apr 28, 6:08 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2100 to cirrussearch2100 completed:

  • elastic2100 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Mon, Apr 28, 6:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2100.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Apr 28, 8:58 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2100.codfw.wmnet with OS bullseye completed:

  • cirrussearch2100 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504282032_bking_1179440_cirrussearch2100.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Apr 29, 1:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye

bking closed subtask T392644: cirrussearch2078 (R440 Config D, Row/Rack B2) unable to PXE boot as Resolved.Tue, Apr 29, 2:05 PM
ops-monitoring-bot added a comment.Tue, Apr 29, 2:35 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2078.codfw.wmnet with OS bullseye completed:

  • cirrussearch2078 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504291416_bking_2400239_cirrussearch2078.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Apr 29, 2:47 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2106 to cirrussearch2106 completed:

  • elastic2106 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Tue, Apr 29, 2:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2106.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Tue, Apr 29, 3:27 PM

Cookbook cookbooks.sre.hosts.rename started by bking@cumin2002 from elastic2108 to cirrussearch2108 completed:

  • elastic2108 (PASS)
    • ✔️ Downtimed host on Icinga/Alertmanager
    • ✔️ Disabled puppet and its timer
    • ✔️ Disabled debmonitor-client timer
    • ✔️ Netbox updated
    • ✔️ BMC Hostname updated
    • ✔️ DNS updated
    • ✔️ Switch description updated
    • ✔️ Removed from DebMonitor
    • ✔️ Removed from Puppet master and PuppetDB
    • Rename completed 👍 - now please run the re-image cookbook on the new name with --new
ops-monitoring-bot added a comment.Tue, Apr 29, 3:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cirrussearch2108.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Tue, Apr 29, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2106.codfw.wmnet with OS bullseye completed:

  • cirrussearch2106 (WARN)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504291526_bking_2453344_cirrussearch2106.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, Apr 29, 4:14 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cirrussearch2108.codfw.wmnet with OS bullseye completed:

  • cirrussearch2108 (PASS)
    • Host successfully migrated to the new VLAN
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202504291553_bking_2491396_cirrussearch2108.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits