confluent-kafka 2.6.2 started using httpx instead of requests for the SchemaRegistryClient (https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/index.html?#schemaregistryclient), causing the REQUESTS_CA_BUNDLE environment variable to have no effect, ultimately causing SSL certificate validation errors (see https://wikimedia.slack.com/archives/C02291Z9YQY/p1738097562314939).
We create this task to make sure we don't pin confluent-kafka to 2.6.0 indefinitely.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Fix typos in changelog | repos/data-engineering/airflow-dags!1069 | brouberol | origin/T386092-package | main | |
| Upgrade confluent-kafka to 2.8.0 now that we found how to avoid SSL validation errors | repos/data-engineering/airflow-dags!1068 | brouberol | T386092-package | main | |
| test_k8s: define a DAG emitting dataset lineage | repos/data-engineering/airflow-dags!1064 | brouberol | T386092 | main |
According to https://docs.confluent.io/platform/current/clients/confluent-kafka-python/html/_modules/confluent_kafka/schema_registry/schema_registry_client.html#SchemaRegistryClient, we should be able to specify ssl.cert.location: /etc/ssl/certs/ca-certificates.crt in the datahub_kafka_jumbo.extra_dejson.connection dict.
brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/48
Drop confluent-kafka version constraint
Specifically, in the datahub_kafka_jumbo.extra_dejson.connection.schema_registry_config dict: https://github.com/datahub-project/datahub/blob/20409fd702d2d4454de9170f77bffecbfba8b527/metadata-ingestion/src/datahub/configuration/kafka.py#L15C5-L15C27
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/48
Drop confluent-kafka version constraint
Change #1118812 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] Airflow: upgrade to confluent-kafka 2.8.0 and specify path to CA cert bundle
Once I redeployed airflow-test-k8s with the following diff:
datahub_kafka_jumbo:
conn_type: datahub_kafka
host: kafka-jumbo-eqiad.external-services.svc.cluster.local:9092
extra_dejson:
connection:
schema_registry_url: https://datahub-gms.discovery.wmnet:30443/schema-registry/api/
+ schema_registry_config:
+ ssl.ca.location: /etc/ssl/certs/ca-certificates.crtthe dag was able to execute successfully.
brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1064
test_k8s: define a DAG emitting dataset lineage
Change #1118812 merged by Brouberol:
[operations/deployment-charts@master] Airflow: upgrade to confluent-kafka 2.8.0 and specify path to CA cert bundle
Change #1119043 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] airflow-analytics: specify the path to the SSL CA certificate bundle
brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1068
Upgrade confluent-kafka to 2.8.0 now that we found how to avoid SSL validation errors
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1068
Upgrade confluent-kafka to 2.8.0 now that we found how to avoid SSL validation errors
brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1069
Fix typos in changelog
Change #1119043 merged by Brouberol:
[operations/puppet@production] airflow-analytics: specify the path to the SSL CA certificate bundle
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1069
Fix typos in changelog
Change #1119185 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] airflow-analytics: fix typo in config
I don't know why but it seems like the kafka sink doesn't recognize schema_registry_config and it's currently failing:
Exception: 1 validation error for KafkaSinkConfig schema_registry_config extra fields not permitted (type=value_error.extra)
Change #1119185 merged by Brouberol:
[operations/puppet@production] airflow-analytics: fix typo in config
Change #1119456 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] airflow-analytics: upgarde the airflow deb package to get a new confluent-kafka version
Change #1119456 merged by Brouberol:
[operations/puppet@production] airflow-analytics: upgrade airflow to get a new confluent-kafka version
The datahub emitter operator task ran successfully even with confluent-kafka 2.8.0, with the config change.
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1064
test_k8s: define a DAG emitting dataset lineage