This is an extremely pertinent question. Thanks @bking for creating the ticket. It's got me thinking.

There are also some useful docs here: https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html

I've been working with this file (GDocs) Airflow on Kubernetes - Test and migration plan and I have made a recommendation in that document, regarding logging.

Having reviewed Logging and Monitoring architecture — Airflow Documentation I suggest that we look to enable both an Elasticsearch based backend, and an S3 based central file store.

For the S3 based system we can use our Ceph cluster, as it has advantages for this use case over both the core swift mediaservers and the new APUs Ceph cluster, managed by Data Persistence.

For the elasticsearch hosting, we could either:

• Run a small but dedicated elasticsearch/opensearch under kubernetes
• Use our new multualised (and virtualised) opensearch cluster
• Request that we use an existing opensearch cluster maintained by Observability

I would be tempted to go with option 1 to engine with, at least during this period of testing.

Getting the S3 based logging to work will require us to complete these two tasks:

• T330152: Deploy ceph radosgw processes to data-engineering cluster
• T330153: Configure Anycast load-balancing ceph radosgw services on the data-engineering cluster

We can already start work on the elasticsearch/opensearch part of it now.

For the elasticsearch hosting

@BTullis why not just used existing logstash infra? Or am I misunderstanding something?

In T369634#10069693, @Ottomata wrote:

For the elasticsearch hosting

@BTullis why not just used existing logstash infra? Or am I misunderstanding something?

Yes, we could do. It's an option discussed in the doc.

I just thought that for this period of early testing it would be as easy to spin up an empty, disposable instance.If it looks like the logstash infra will be better long term for the production instances, then we can use that.

I hadn't looked into whether we can create our own arbitrary indices on the logstash infa, but I can do this too.

I hadn't looked into whether we can create our own arbitrary indices on the logstash infa, but I can do this too.

Hm! Do you need this? Would the ECS index be sufficient?

In T369634#10069787, @Ottomata wrote:

I hadn't looked into whether we can create our own arbitrary indices on the logstash infa, but I can do this too.

Hm! Do you need this? Would the ECS index be sufficient?

I don't know yet, but I think we should be able to find out with a bit of experimentation.
I've been looking at the configuration options described here: https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/logging/index.html

They only seems to mention a host:port for elasticsearch, then there is an option to enable JSON logging for tasks. I'm just not sure how easy it would be to integrate the elasticsearch logging provider with our kafka/rsyslog based logstash pipeline.
Anyway, I'll keep an open mind to it and see what we discover during testing.

I have decided to ask a question on the Airfow Slack about whether or not there is any benefit to running both S3 and Elasticsearch based logs.

I have received one response so far to my question, which is from an Airflow contributor.

So there may be no use case for implementing both S3 and Elastic based logging, after all.