As per title, the goal is for role::thanos::frontend to have only swift bits and not thanos bits anymore
Description
Description
Details
Details
Related Changes in Gerrit:
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T341488 Split Thanos components from thanos-fe hosts into titan hosts | |||
| Resolved | fgiunchedi | T346143 Remove thanos components from thanos-fe role and reimage thanos-fe hosts | |||
| Resolved | fgiunchedi | T347964 new_rings.tar.bz2 not found after host reimage | |||
| Resolved | fgiunchedi | T348119 thanos-fe2004 fail to boot into PXE |
Event Timeline
Comment Actions
Change 956886 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] install_server: restore raid1 for thanos-fe
Comment Actions
Change 956888 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] conftool-data: split thanos-fe / titan hosts' services
Comment Actions
Change 956886 merged by Filippo Giunchedi:
[operations/puppet@production] install_server: restore raid1 for thanos-fe
Comment Actions
Change 956904 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] thanos: move rule evaluation to titan hosts
Comment Actions
Change 956905 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] thanos: move thanos-compact to titan host
Comment Actions
Change 956906 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] thanos: remove thanos components from thanos::frontend role
Comment Actions
Change 956888 merged by Filippo Giunchedi:
[operations/puppet@production] conftool-data: split thanos-fe / titan hosts' services
Comment Actions
Change 956904 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: move rule evaluation to titan hosts
Comment Actions
Change 956905 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: move thanos-compact to titan host
Comment Actions
Change 956906 merged by Filippo Giunchedi:
[operations/puppet@production] thanos: remove thanos components from thanos::frontend role
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe1001.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe1001.eqiad.wmnet with OS bullseye completed:
- thanos-fe1001 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe1001.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310030803_filippo_436934_thanos-fe1001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=eqiad,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe1002.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe1002.eqiad.wmnet with OS bullseye completed:
- thanos-fe1002 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe1002.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310030927_filippo_480502_thanos-fe1002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=eqiad,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe1003.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe1003.eqiad.wmnet with OS bullseye completed:
- thanos-fe1003 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe1003.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031035_filippo_514138_thanos-fe1003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=eqiad,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe1004.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe1004.eqiad.wmnet with OS bullseye completed:
- thanos-fe1004 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe1004.eqiad.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031133_filippo_543338_thanos-fe1004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=eqiad,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
- Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2001.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2001.codfw.wmnet with OS bullseye completed:
- thanos-fe2001 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe2001.codfw.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031245_filippo_581031_thanos-fe2001.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=codfw,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2002.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2002.codfw.wmnet with OS bullseye completed:
- thanos-fe2002 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe2002.codfw.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310031349_filippo_616764_thanos-fe2002.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=codfw,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2003.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2003.codfw.wmnet with OS bullseye completed:
- thanos-fe2003 (WARN)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe2003.codfw.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310040756_filippo_1142315_thanos-fe2003.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'dc=codfw,cluster=thanos,service=thanos-swift' set/pooled=yes
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye executed with errors:
- thanos-fe2004 (FAIL)
- Downtimed on Icinga/Alertmanager
- Set pooled=inactive for the following services on confctl:
{"thanos-fe2004.codfw.wmnet": {"weight": 100, "pooled": "yes"}, "tags": "dc=codfw,cluster=thanos,service=thanos-swift"}
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
sudo confctl select 'name=thanos\-fe2004\.codfw\.wmnet,dc=codfw,cluster=thanos,service=thanos\-swift' set/pooled=yes
- The reimage failed, see the cookbook logs for the details
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye executed with errors:
- thanos-fe2004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye executed with errors:
- thanos-fe2004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host thanos-fe2004.codfw.wmnet with OS bullseye completed:
- thanos-fe2004 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202310041117_filippo_1241387_thanos-fe2004.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB