In T323913: Move thanos-sso away from CNAME discovery.wmnet we moved away from dns-controlled thanos-fe hosts for thanos-web (i.e. thanos.w.o) and to a conftool-controlled service.
The move solved the problem of easily swapping backends (e.g. for maintenance) though it created a nuisance: namely that we currently need to (remember to) have a single thanos-fe host pooled at a time per site.
This is because mod_auth_cas / SSO sessions are not shared between hosts, and backend selection is (for all intended purposes) random. Therefore, if we have more than one backend pooled then clients (in this case ATS, by way of the frontend cdn) could land on different hosts and not have their SSO session stored there.
After a chat/brainstorm with @Muehlenhoff here's my current thinking:
- We need an authenticating proxy in front of thanos anyways since thanos web serving doesn't ship authentication/authorization natively
- We want said proxy to be compatible with our SSO _and_ have a mechanism to share sessions between different hosts
To this end, we have at least a couple of solutions:
- Keep mod_auth_cas and find a way to make it share its sessions, as of March 2023 the module supports filesystem storage only. In other words some form of shared filesystem between all backends would be needed.
- Find an authenticating/authorizing proxy (including an apache module) that can talk SAML or OIDC (or CAS really!) and supports sharing sessions among hosts natively (e.g. with memcached)