Track the implementation of Jaeger secure access to OpenSearch. Items in no particular order (possibly incomplete list):
- k8s cluster is allowed to talk to OpenSearch API (e.g. firewall rules)
- Jaeger can talk https to OpenSearch (ideally with some kind of mutual authentication)
I (Filippo) have investigated this a little and so far there's a few options (in no particular order) for access:
- jaeger accesses opensearch as a "standard" service. In other words we point jaeger to a load-balanced HTTPS service/api backed by logstash hosts running logging::opensearch::collector (essentially stateless frontends).
- pros: familiar pattern of deployment/traffic, we can depool/pool logstash hosts as we do e.g. for kibana, jager config has sth like "https://logs-opensearch.discovery.wmnet"
- cons: requires more work up front to set up (load balanced service, envoy configs, etc), access control needs to happen at the HTTP level
- jaeger accesses opensearch as a cluster. In this case jaeger talks HTTP to port 9200 directly to all opensearch cluster hosts (currently all hosts running logging::opensearch::data and logging::opensearch::collector), cluster node list happens automatically via sniffing.
- pros: easy to setup (open port 9200 on the logstash cluster to k8x-aux workers)
- cons: no tls out of the box, need to bake some production hostnames in jaeger config
- Jaeger outputs to kafka-logging as a buffer, jaeger-ingester (perhaps deployed within the logging cluster) reads from kafka-logging and persists to opensearch (from @herron)
- Pros: reuses existing/understood logging pipeline architecture & monitoring, allows for queueing and recovery during backend outages/maintenances, helps backend cluster stability by absorbing bursts
- Cons: presumably some setup/maint to be done around packaging, puppetizing/etc, montioring for the jaeger-ingester service