In order to test Airflow's Elastic logging capabilities as described in the parent ticket, we need an Opensearch service.
To that end, this ticket will track the work needed to create a Docker image suitable for hosting said Opensearch service. Not in scope: Creating an Opensearch image suitable for production.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Gehel | T327267 Create a DSE Kubernetes cluster with support for persistent storage from Ceph | |||
| Resolved | Gehel | T324660 Install Ceph Cluster for Data Platform Engineering | |||
| Open | None | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | BTullis | T330152 Deploy ceph radosgw processes to data-engineering cluster | |||
| Resolved | BTullis | T330153 Configure Anycast load-balancing ceph radosgw services on the data-engineering cluster | |||
| Resolved | BTullis | T369634 Decide how to do DAG logging on dse-k8s | |||
| Declined | None | T372788 Implement elasticsearch/opensearch based logging for Airflow on dse-k8s | |||
| Resolved | bking | T373034 Build Opensearch2 Docker image for testing |
Not in scope: Creating an Opensearch image suitable for production.
I can appreciate that you don't want to be offering too much at this point, but how would you expect an image suitable for production to differ from the image you are creating here?
Personally, I would prefer it if we didnt use the -test suffix in the repository name: i.e. https://gitlab.wikimedia.org/repos/data-engineering/opensearch-test because this will be burnt into the docker-registry path.
I can appreciate that you don't want to be offering too much at this point, but how would you expect an image suitable for production to differ from the image you are creating here?
Sorry, this is mostly advice to myself to stay focused on an MVP instead of thinking ahead to production. From a practical standpoint, I doubt there would be all that much difference.
Personally, I would prefer it if we didnt use the -test suffix in the repository name: i.e. https://gitlab.wikimedia.org/repos/data-engineering/opensearch-test because this will be burnt into the docker-registry path.
Thanks for the advice...I completely agree and have removed the -test suffix as requested.
bking opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/97
data-engineering: add opensearch2 image to trusted runners
dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/97
data-engineering: add opensearch2 image to trusted runners