cloudelastic1007-1010 are racked and ready to join the cluster. See Search Platform docs for the procedure (which might be outdated and need review).
AC
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | bking | T351354 Service implementation for cloudelastic1007-1010 | |||
| Resolved | bking | T355617 Migrate cloudelastic from public to private IPs | |||
| Resolved | bking | T355720 Change TLS/load balancer configuration for cloudelastic | |||
| Resolved | VRiley-WMF | T356919 Comm Error: Backplane 0 on cloudelastic1008 | |||
| Resolved | bking | T356941 Stale data/failed queries on wikidatawiki index |
Change 974693 had a related patch set uploaded (by Bking; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: bring cloudelastic10[07-10] into svc
Change 974693 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: bring cloudelastic10[07-10] into svc
Mentioned in SAL (#wikimedia-operations) [2023-11-15T22:20:06Z] <ryankemper> T351354 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/974693; running puppet on hosts
Change 974694 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: hosts need racking info
Change 974694 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: hosts need racking info
Change 974696 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: switch new hosts back to insetup
Change 974696 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: switch new hosts back to insetup
Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host cloudelastic1008.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host cloudelastic1008.wikimedia.org with OS bullseye executed with errors:
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl-public"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl-public"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl"}
{"cloudelastic1008.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl-public"}
Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye executed with errors:
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-psi-ssl-public"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-chi-ssl-public"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl"}
{"cloudelastic1007.wikimedia.org": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=cloudelastic,service=cloudelastic-omega-ssl-public"}
Cookbook cookbooks.sre.hosts.reimage was started by jbond@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by jbond@cumin1001 for host cloudelastic1007.wikimedia.org with OS bullseye completed:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1008.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1008.wikimedia.org with OS bullseye completed:
@bking i took a look at cloudelastic1010 as i had thought this was in some broken state from the reimage cookbook. however from the puppet certs i can see its been around since Nov 9 07:30:40 2023 GMT and has had puppet disabled for the last 36 hours.
as a side note you shouldn't need to disable puppet when a server has the in-setup role and its bad to do so.
@jbond Sorry for the confusion, I associated the reimage with the wrong ticket. The output of the last reimage is here . Puppet was disabled because the hosts were previously set to their production role, but due to the PKI errors we put them back to insetup. I should have paid more attention...it seems the reimage never actually wiped the disks, whereas I had assumed it failed on later steps.
As far as what led to this situation, I'll try to recount in the hopes that it could be useful:
This isn't a blocker to our work, so don't feel like you have to dig in too deeply. I've left 1010 up in hopes that it might be useful. If it isn't, ping me and I'll reimage again.
@bking in order for me to investigate further i need either broken host to investigate or a way to replicate the issue.
Change 975824 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] cloudelastic: force Puppet 7 for cloudelastic1010
Change 975824 merged by Bking:
[operations/puppet@production] cloudelastic: force Puppet 7 for cloudelastic1010
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye executed with errors:
Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host cloudelastic1010.wikimedia.org with OS bullseye completed:
Change 991788 had a related patch set uploaded (by Bking; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: bring cloudelastic10[07-10] into svc
Change 991788 merged by Bking:
[operations/puppet@production] cloudelastic: bring cloudelastic10[07-10] into svc
Change 991797 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] cloudelastic: allow new hosts to request TLS certs
Change 991797 merged by Bking:
[operations/puppet@production] cloudelastic: allow new hosts to request TLS certs
Change 991845 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] cloudelastic: cleanup allowed_regexes
Change 991845 merged by Bking:
[operations/puppet@production] cloudelastic: cleanup allowed_regexes
We got a diffscan alert as those servers are running on public IPs and new ports are exposed to the diffscan cloudVM.
After a quick look it seems like those servers already expose their endpoint through LVS (cloudelastic.wikimedia.org), that's why I'm wondering why can't they be in the private vlans ? If they have good reasons, could they be defined somewhere ? if not could they be re-numbered to private IPs ?
See https://wikitech.wikimedia.org/wiki/Wikimedia_network_guidelines#Public_IPs
@ayounsi Thanks for the link. We're in the process of rolling out new hosts and unfortunately, we reused the existing puppet code without much thought about public IPs. What is the urgency on this request and how long do you think it would take to re-ip these servers? If there are any docs on how to do this let us know.
What is the urgency on this request
Without sounding alarming, if they don't need public IPs, it should be done now to prevent having to handle them for the next 5 years. Public IP hosts are quite a pain for the reasons listed on the wiki page.
and how long do you think it would take to re-ip these servers? If there are any docs on how to do this let us know.
That's quite straightforward, outside of re-image scripts, I'd say 15min per servers.
The procedure is on https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs and I can walk you through it no pb.
From timeline and my understanding of traffic flows and service owner it seems like the hosts are more suited in the prod private vlan than cloud-private but happy to discuss it.
Likewise I am happy to assist here if needed. The process is a little crunky, but not too tricky if they are new servers not yet live.
It'd be a real shame to bring live a bunch of new servers on the public vlan, using those IPs up for the next few years when we don't need to.
Change 992538 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] cloudelastic: promote new hosts to master-eligible
Change 992538 merged by Bking:
[operations/puppet@production] cloudelastic: promote new hosts to master-eligible
Forgot to add the Bug: label but https://gerrit.wikimedia.org/r/c/operations/puppet/+/992826 is part of this ticket as well
Change 993038 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] cloudelastic: remove old masters
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:08:57Z] <ryankemper> T351354 Downtimed cloudelastic*; shortly will restart cloudelastic100[1,2,4] one host at a time to make them no longer masters
Change 993038 merged by Ryan Kemper:
[operations/puppet@production] cloudelastic: remove old masters
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:11:25Z] <ryankemper> T351354 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/993038; restarting cloudelastic1001 following puppet run
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:15:50Z] <ryankemper> T351354 Restarting cloudelastic1004 following puppet run
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:25:58Z] <ryankemper> T351354 Restarting cloudelastic1002
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:33:19Z] <ryankemper> T351354 Now restarting new masters to keep configs in sync; restarting cloudelastic1007
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:34:42Z] <ryankemper> T351354 Now restarting new masters to keep configs in sync; restarting cloudelastic1009
Mentioned in SAL (#wikimedia-operations) [2024-01-25T22:40:06Z] <ryankemper> T351354 Restarting cloudelastic1006 (final restart for today)
Old masters are no longer master-eligible. They're still participating in the actual cluster; we're holding off on the physical decom until T355617 is done
cloudelastic10[07-10] are now in service (most work happened in T355617) . Closing....