cloudelastic1007-1010 are racked and ready to join the cluster. See Search Platform docs for the procedure (which might be outdated and need review).
AC
Change 974693 had a related patch set uploaded (by Bking; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: bring cloudelastic10[07-10] into svc
Change 974693 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: bring cloudelastic10[07-10] into svc
Mentioned in SAL (#wikimedia-operations) [2023-11-15T22:20:06Z] <ryankemper> T351354 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/974693; running puppet on hosts
Change 974694 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: hosts need racking info
Change 974694 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: hosts need racking info
Change 974696 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: switch new hosts back to insetup
Change 974696 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: switch new hosts back to insetup
Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host cloudelastic1008.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host cloudelastic1008.wikimedia.org with OS bullseye executed with errors:
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl-public"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl-public"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl-public"}
Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye executed with errors:
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl-public"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl-public"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl-public"}
Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1008.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1008.wikimedia.org with OS bullseye completed:
@bking i took a look at cloudelastic1010 as i had thought this was in some broken state from the reimage cookbook. however from the puppet certs i can see its been around since Nov 9 07:30:40 2023 GMT and has had puppet disabled for the last 36 hours.
as a side note you shouldn't need to disable puppet when a server has the in-setup role and its bad to do so.
@jbond Sorry for the confusion, I associated the reimage with the wrong ticket. The output of the last reimage is here . Puppet was disabled because the hosts were previously set to their production role, but due to the PKI errors we put them back to insetup. I should have paid more attention...it seems the reimage never actually wiped the disks, whereas I had assumed it failed on later steps.
As far as what led to this situation, I'll try to recount in the hopes that it could be useful:
This isn't a blocker to our work, so don't feel like you have to dig in too deeply. I've left 1010 up in hopes that it might be useful. If it isn't, ping me and I'll reimage again.
@bking in order for me to investigate further i need either broken host to investigate or a way to replicate the issue.
Change 975824 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] cloudelastic: force Puppet 7 for cloudelastic1010
Change 975824 merged by Bking:
[operations/puppet@production] cloudelastic: force Puppet 7 for cloudelastic1010
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye completed: