The ML team currently uses thanos-swift as part of their module handling workflows.
This was defined in: T294409: Workflow to upload models to Swift
We need to allow the Airflow-ML instance to access thanos-swift and provision the correct tokens for it.
Change #1164991 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] global_config: provision thanos-swift-{eqiad,codfw} external services
Change #1164993 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow-ml: enable task pods to reach out to thanos-swift in both DCs
Change #1164994 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow-ml: define a connection to thanos-swift
Change #1164991 merged by Brouberol:
[operations/puppet@production] global_config: provision thanos-swift-{eqiad,codfw} external services
Change #1164993 merged by Brouberol:
[operations/deployment-charts@master] airflow-ml: enable task pods to reach out to thanos-swift in both DCs
The egress traffic between airflow-ml task pods http://thanos-swift.discovery.wmnet is now open:
airflow@airflow-task-shell-6b8f47cf88-z2w2m:/opt/airflow$ ./usr/bin/is_port_open thanos-swift.discovery.wmnet 443 thanos-swift.discovery.wmnet:443 is open
Let's now make sure that we provision a working connection.
airflow@airflow-scheduler-cd68f6bb6-gzrdp:/opt/airflow$ python3
Python 3.10.15 (main, Jun 11 2025, 12:33:45) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from wmf_airflow_common.clients.s3 import get_s3_client.
>>> s3 = get_s3_client(conn_id='thanos_swift')
[2025-06-30T12:51:55.643+0000] {crypto.py:82} WARNING - empty cryptography key - values will not be stored encrypted.
>>> s3.list_buckets()
{'ResponseMetadata': {'RequestId': 'tx9963ed8d72e043e2b57c2-0068628871', 'HostId': 'tx9963ed8d72e043e2b57c2-0068628871', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'application/xml', 'content-length': '326', 'x-amz-id-2': 'tx9963ed8d72e043e2b57c2-0068628871', 'x-amz-request-id': 'tx9963ed8d72e043e2b57c2-0068628871', 'x-trans-id': 'tx9963ed8d72e043e2b57c2-0068628871', 'x-openstack-request-id': 'tx9963ed8d72e043e2b57c2-0068628871', 'date': 'Mon, 30 Jun 2025 12:52:01 GMT', 'x-envoy-upstream-service-time': '17', 'server': 'envoy'}, 'RetryAttempts': 0}, 'Buckets': [{'Name': 'wmf-ml-models', 'CreationDate': datetime.datetime(2009, 2, 3, 16, 45, 9, tzinfo=tzlocal())}], 'Owner': {'DisplayName': 'mlserve:prod', 'ID': 'mlserve:prod'}}Change #1164994 merged by Brouberol:
[operations/deployment-charts@master] airflow-ml: define a connection to thanos-swift
@gkyziridis This is what you need to have in your DAG task code to get an s3 client to thanos-swift:
from wmf_airflow_common.clients.s3 import get_s3_client. s3 = get_s3_client(conn_id='thanos_swift')