Page MenuHomePhabricator
Create Task
Maniphest T372286

Test the integration of the airflow chart with the cloudnative-pg-cluster chart
Closed, ResolvedPublic
Actions

Description

We'd like to get a PG cluster deployed as a dependency of the airflow chart. We'd need the dependency to be resolved by helmfile and applied/installed before airflow itself. This way, we'd get the PG cluster deployed, and then airflow.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cloudnative-pg-cluster: stop printing variable in resource manifestoperations/deployment-chartsmaster+3 -3
airflow-test-k8s: Move pooler.imageTag to a value file common to all PG clusters in dse-k8s-eqiadoperations/deployment-chartsmaster+4 -2
airflow: disable sqlalchemy pooling when deployed against a cloudnative clusteroperations/deployment-chartsmaster+12 -1
airflow: automatically connect to PGBouncer instead of PG itselfoperations/deployment-chartsmaster+35 -7
cloudnative-pg-cluster: enable ingress from the join pod to PG/5432operations/deployment-chartsmaster+34 -1
airflow: run the initdb as an initcontaineroperations/deployment-chartsmaster+14 -29
airflow: run the initdb job outside of hooksoperations/deployment-chartsmaster+2 -8
airflow: add missing spec and volumes to the initdb joboperations/deployment-chartsmaster+9 -8
airflow: initialize the DB via a hook, at first installoperations/deployment-chartsmaster+62 -26
cloudnative-pg-cluster: enable ingress traffic to the PG pod APIoperations/deployment-chartsmaster+16 -1
cloudnative-pg: configure the operator to watch the airflow-test-k8s nsoperations/deployment-chartsmaster+1 -0
airflow: fetch PG connection URI from the cloudnative PG cluster secretoperations/deployment-chartsmaster+69 -4
airflow: deploy postgresql cluster before airflow itselfoperations/deployment-chartsmaster+18 -7
rbac: grant RBAC permissions on cloudnative-pg CRDs to our view/deploy ClusterRolesoperations/deployment-chartsmaster+10 -0
airflow: add conditional dependency to cloudnative-pg-clusteroperations/deployment-chartsmaster+14 -5
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Expose the pooler URI in the application secretrepos/data-engineering/postgresql-kubernetes!11brouberolT372286-expose-pooler-urimain
Customize query in GitLab

Related Objects
Search...

Event Timeline

brouberol created this task.Aug 12 2024, 2:12 PM
gerritbot added a comment.Aug 12 2024, 2:17 PM

Change #1062030 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: add conditional dependency to cloudnative-pg-cluster

https://gerrit.wikimedia.org/r/1062030

gerritbot added a project: Patch-For-Review.Aug 12 2024, 2:17 PM
brouberol moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.07.29 - 2024.08.16) board.Aug 12 2024, 2:38 PM
gerritbot added a comment.Aug 13 2024, 12:29 PM

Change #1062030 abandoned by Brouberol:

[operations/deployment-charts@master] airflow: add conditional dependency to cloudnative-pg-cluster

Reason:

Deploying PG via helm dependencies gives no deployment order guarantees

https://gerrit.wikimedia.org/r/1062030

Maintenance_bot removed a project: Patch-For-Review.Aug 13 2024, 12:30 PM
gerritbot added a comment.Aug 13 2024, 1:19 PM

Change #1062397 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: deploy postgresql cluster before airflow itself

https://gerrit.wikimedia.org/r/1062397

gerritbot added a project: Patch-For-Review.Aug 13 2024, 1:19 PM

Change #1062398 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: fetch PG connection URI from the cloudnative PG cluster secret

https://gerrit.wikimedia.org/r/1062398

gerritbot added a comment.Aug 13 2024, 3:18 PM

Change #1062431 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] rbac: grant RBAC permissions on cloudnative-pg CRDs to our view/deploy ClusterRoles

https://gerrit.wikimedia.org/r/1062431

Gehel triaged this task as High priority.Aug 14 2024, 8:30 AM
Gehel edited projects, added Data-Platform-SRE (2024.08.17 - 2024.09.06); removed Data-Platform-SRE (2024.07.29 - 2024.08.16).Aug 16 2024, 9:43 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.08.17 - 2024.09.06) board.
gerritbot added a comment.Aug 19 2024, 9:12 AM

Change #1062431 merged by Brouberol:

[operations/deployment-charts@master] rbac: grant RBAC permissions on cloudnative-pg CRDs to our view/deploy ClusterRoles

https://gerrit.wikimedia.org/r/1062431

gerritbot added a comment.Aug 19 2024, 9:25 AM

Change #1062397 merged by Brouberol:

[operations/deployment-charts@master] airflow: deploy postgresql cluster before airflow itself

https://gerrit.wikimedia.org/r/1062397

gerritbot added a comment.Aug 19 2024, 9:27 AM

Change #1062398 merged by Brouberol:

[operations/deployment-charts@master] airflow: fetch PG connection URI from the cloudnative PG cluster secret

https://gerrit.wikimedia.org/r/1062398

gerritbot added a comment.Aug 19 2024, 11:00 AM

Change #1063769 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] cloudnative-pg: configure the operator to watch the airflow-test-k8s ns

https://gerrit.wikimedia.org/r/1063769

gerritbot added a comment.Aug 19 2024, 12:15 PM

Change #1063769 merged by Brouberol:

[operations/deployment-charts@master] cloudnative-pg: configure the operator to watch the airflow-test-k8s ns

https://gerrit.wikimedia.org/r/1063769

gerritbot added a comment.Aug 19 2024, 12:38 PM

Change #1063803 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] cloudnative-pg-cluster: enable ingress traffic to the PG pod API

https://gerrit.wikimedia.org/r/1063803

gerritbot added a comment.Aug 19 2024, 12:56 PM

Change #1063803 merged by Brouberol:

[operations/deployment-charts@master] cloudnative-pg-cluster: enable ingress traffic to the PG pod API

https://gerrit.wikimedia.org/r/1063803

gerritbot added a comment.Aug 19 2024, 1:24 PM

Change #1063808 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] cloudnative-pg-cluster: enable ingress from the join pod to PG/5432

https://gerrit.wikimedia.org/r/1063808

gerritbot added a comment.Aug 19 2024, 1:57 PM

Change #1063814 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: initialize the DB via a hook, at first install

https://gerrit.wikimedia.org/r/1063814

gerritbot added a comment.Aug 19 2024, 2:22 PM

Change #1063814 merged by Brouberol:

[operations/deployment-charts@master] airflow: initialize the DB via a hook, at first install

https://gerrit.wikimedia.org/r/1063814

CodeReviewBot added a comment.Aug 19 2024, 2:30 PM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/postgresql-kubernetes/-/merge_requests/11

Expose the pooler URI in the application secret

gerritbot added a comment.Aug 19 2024, 2:50 PM

Change #1063823 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: add missing spec and volumes to the initdb job

https://gerrit.wikimedia.org/r/1063823

gerritbot added a comment.Aug 19 2024, 2:57 PM

Change #1063823 merged by Brouberol:

[operations/deployment-charts@master] airflow: add missing spec and volumes to the initdb job

https://gerrit.wikimedia.org/r/1063823

gerritbot added a comment.Aug 19 2024, 3:21 PM

Change #1063830 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: run the initdb job outside of hooks

https://gerrit.wikimedia.org/r/1063830

gerritbot added a comment.Aug 19 2024, 3:22 PM

Change #1063830 merged by Brouberol:

[operations/deployment-charts@master] airflow: run the initdb job outside of hooks

https://gerrit.wikimedia.org/r/1063830

gerritbot added a comment.Aug 19 2024, 3:34 PM

Change #1063836 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: run the initdb as an initcontainer

https://gerrit.wikimedia.org/r/1063836

gerritbot added a comment.Aug 19 2024, 3:46 PM

Change #1063836 merged by Brouberol:

[operations/deployment-charts@master] airflow: run the initdb as an initcontainer

https://gerrit.wikimedia.org/r/1063836

gerritbot added a comment.Aug 20 2024, 7:51 AM

Change #1063808 merged by Brouberol:

[operations/deployment-charts@master] cloudnative-pg-cluster: enable ingress from the join pod to PG/5432

https://gerrit.wikimedia.org/r/1063808

gerritbot added a comment.Aug 20 2024, 2:16 PM

Change #1064039 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: automatically connect to PGBouncer instead of PG itself

https://gerrit.wikimedia.org/r/1064039

CodeReviewBot added a comment.Aug 20 2024, 2:17 PM

brouberol closed https://gitlab.wikimedia.org/repos/data-engineering/postgresql-kubernetes/-/merge_requests/11

Expose the pooler URI in the application secret

gerritbot added a comment.Aug 20 2024, 2:26 PM

Change #1064039 merged by Brouberol:

[operations/deployment-charts@master] airflow: automatically connect to PGBouncer instead of PG itself

https://gerrit.wikimedia.org/r/1064039

gerritbot added a comment.Aug 21 2024, 1:38 PM

Change #1064384 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: disable sqlalchemy pooling when deployed against a cloudnative cluster

https://gerrit.wikimedia.org/r/1064384

gerritbot added a comment.Aug 21 2024, 1:42 PM

Change #1064384 merged by Brouberol:

[operations/deployment-charts@master] airflow: disable sqlalchemy pooling when deployed against a cloudnative cluster

https://gerrit.wikimedia.org/r/1064384

brouberol closed this task as Resolved.Aug 21 2024, 2:03 PM
brouberol moved this task from In Progress to Done on the Data-Platform-SRE (2024.08.17 - 2024.09.06) board.
gerritbot added a comment.Aug 23 2024, 12:12 PM

Change #1065203 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-test-k8s: Move pooler.imageTag to a value file common to all PG clusters in dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1065203

gerritbot added a comment.Aug 23 2024, 1:30 PM

Change #1065203 merged by Brouberol:

[operations/deployment-charts@master] airflow-test-k8s: Move pooler.imageTag to a value file common to all PG clusters in dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1065203

gerritbot added a comment.Aug 23 2024, 1:51 PM

Change #1065227 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] cloudnative-pg-cluster: stop printing variable in resource manifest

https://gerrit.wikimedia.org/r/1065227

gerritbot added a comment.Aug 23 2024, 1:54 PM

Change #1065227 merged by Brouberol:

[operations/deployment-charts@master] cloudnative-pg-cluster: stop printing variable in resource manifest

https://gerrit.wikimedia.org/r/1065227

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits