This task tracks the deployment of Thanos stateless components, the big win being a query endpoint that can reach out to all prometheus instances and merge/deduplicate results as needed.
Outline of what's needed:
[x] Thanos Debian package
[x] Prometheus instances need to advertise unique `external_labels`
- [x] Need to add labels: `instance` (or `name` or sth like that) plus `replica` (`A` or `B`)
- [x] The labels above need to be filtered out before ingestion by our `global` instance for backwards compatibility
[x] Deploy Thanos sidecar alongside each Prometheus instance (save for `global`)
- [x] Needs two ports for each of http + grpc interfaces, likely as an offset of the instance's port itself
[x] Deploy Thanos query component
- [x] Deploy on thanos-fe2* hosts
- [x] Deploy on thanos-fe1* hosts
- [x] Needs two ports for http+grpc, and labels that are considered for deduplication (`replica` in our case)
- [x] Needs to locate and reach all other Thanos sidecars
- [x] Deploy behind LVS and can be active/active (i.e. discovery DNS records) following https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service.
[x] Configure and test datasource in Grafana