On DPE side I believe we need to cover the following items in order to support this new instance:

• Implement necessary changes to the airflow-dags repository
  • Add the new exp_platform folder with new dags and config subfolders
  • Add new config/dag_config.py and config/artifacts.yaml files specific to the exp_platform instance
  • Add the correct tests folder tests/exp_platform
  • Update .gitlab-ci.yml and add the new instance to linters, pytest, Blunderbuss cache updater etc
• Add new service user for exp_platform
• Add the new instance to Helm deployment charts

Feel free to update the list if I forgot something!

Thanks! I copied that into the acceptance criteria and updated exp_platform to experiment_platform

thanks SRE!

@mpopov Who should be in the future airflow-experiment-platform-ops LDAP group? I was thinking Mikhail and Sam?

In T416709#11686299, @brouberol wrote:

@mpopov Who should be in the future airflow-experiment-platform-ops LDAP group? I was thinking Mikhail and Sam?

Oh goodness me I'm so sorry I missed your ping back in March! 😅 I think it should probably be Sam, Katherine, and me.

I'm taking it from here.

Change #1327550 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] deployment_server: define airflow config

https://gerrit.wikimedia.org/r/1327550

Change #1327551 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dse-k8s-eqiad: provision new airflow namespace

https://gerrit.wikimedia.org/r/1327551

Change #1327552 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dse-k8s-eqiad: add the airflow-experiment-platform ns to the ceph tenant list

https://gerrit.wikimedia.org/r/1327552

Change #1327553 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] dse-k8s-eqiad: provision the airflow instance

https://gerrit.wikimedia.org/r/1327553

Change #1327550 merged by Atsuko:

[operations/puppet@production] deployment_server: define airflow config

https://gerrit.wikimedia.org/r/1327550

Change #1327551 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: provision new airflow namespace

https://gerrit.wikimedia.org/r/1327551

During the application of 1327551 I got a problem. admin_ng applied just fine on first run, and then it has displayed another diff that was impossible to apply:

$ sudo -i charlie --services_dir admin_ng -e dse-k8s-eqiad apply
[. dse-k8s-eqiad] airflow-experiment-platform, deploy, RoleBinding (rbac.authorization.k8s.io) has changed:
[. dse-k8s-eqiad] ...
[. dse-k8s-eqiad]     name: deploy
[. dse-k8s-eqiad]     namespace: airflow-experiment-platform
[. dse-k8s-eqiad]   roleRef:
[. dse-k8s-eqiad]     apiGroup: rbac.authorization.k8s.io
[. dse-k8s-eqiad]     kind: ClusterRole
[. dse-k8s-eqiad] -   name: deploy
[. dse-k8s-eqiad] +   name: deploy-airflow
[. dse-k8s-eqiad]   subjects:
[. dse-k8s-eqiad]   - apiGroup: rbac.authorization.k8s.io
[. dse-k8s-eqiad]     kind: User
[. dse-k8s-eqiad]     name: postgresql-airflow-experiment-platform-deploy
[. dse-k8s-eqiad]   - apiGroup: rbac.authorization.k8s.io
[. dse-k8s-eqiad]   Error: UPGRADE FAILED: release namespaces failed, and has been rolled back due to atomic being set: cannot patch "deploy" with kind RoleBinding: RoleBinding.rbac.authorization.k8s.io "deploy" is invalid: roleRef: Invalid value: rbac.RoleRef{APIGroup:"rbac.authorization.k8s.io", Kind:"ClusterRole", Name:"deploy-airflow"}: cannot change roleRef

Fixed by removing the resource

# kube_env admin dse-k8s-eqiad
# kubectl --namespace airflow-experiment-platform delete rolebindings.rbac.authorization.k8s.io deploy

See https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes/Operations

• admin_ng namespaces
• admin_ng misc
• generated airflow secrets
• provisioned s3 and obtained s3 secrets including ones for backup
• provision oidc
• provision user id
• provision sudo
• obtain krb

Change #1328573 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] idp: add the airflow_experiment_platform service

https://gerrit.wikimedia.org/r/1328573

Change #1328576 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/dns@master] provision the airflow-experiment-platform DNS records

https://gerrit.wikimedia.org/r/1328576

Change #1328582 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] Add analytics-experiment system user and groups

https://gerrit.wikimedia.org/r/1328582

Change #1328595 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] Grant sudo privileges for the analytics-experiment-users group

https://gerrit.wikimedia.org/r/1328595

Change #1328598 had a related patch set uploaded (by Atsuko; author: Atsuko):

[labs/private@master] idp: provision the airflow_experiment_platform placeholder secret

https://gerrit.wikimedia.org/r/1328598

Change #1328598 merged by Atsuko:

[labs/private@master] idp: provision the airflow_experiment_platform placeholder secret

https://gerrit.wikimedia.org/r/1328598

Change #1328573 merged by Atsuko:

[operations/puppet@production] idp: add the airflow_experiment_platform service

https://gerrit.wikimedia.org/r/1328573

Change #1327552 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: add the airflow-experiment-platform ns to the ceph tenant list

https://gerrit.wikimedia.org/r/1327552

Change #1328576 merged by Atsuko:

[operations/dns@master] provision the airflow-experiment-platform DNS records

https://gerrit.wikimedia.org/r/1328576

Change #1328582 merged by Atsuko:

[operations/puppet@production] Add analytics-experiment system user and groups

https://gerrit.wikimedia.org/r/1328582

• configure ldap airflow-experiment-platform-ops
• bring growthbook secrets
• merge sudo
• Create the HDFS folders P96273
• Configure the new instance in airflow-dags repo
• merge the airflow helmfile

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2577

Adding experiment_platform instance

@mpopov what users should I add to analytics-experiment-users group? Will need to add them to 1328595.

Change #1331529 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] Add airflow-experiment-platform-ops LDAP group

https://gerrit.wikimedia.org/r/1331529

Asked @MoritzMuehlenhoff to help with creating a LDAP group.
Initial member: mpopov
Group description: Allows managing DAG runs in the Experiments Platform Airflow instance, more information at https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow

In T416709#12265125, @atsuko wrote:

Asked @MoritzMuehlenhoff to help with creating a LDAP group.
Initial member: mpopov
Group description: Allows managing DAG runs in the Experiments Platform Airflow instance, more information at https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow

The group has been added now.

Mentioned in SAL (#wikimedia-operations) [2026-08-28T11:43:23Z] <moritzm> add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org T416709

Change #1331529 merged by Atsuko:

[operations/puppet@production] Add airflow-experiment-platform-ops LDAP group

https://gerrit.wikimedia.org/r/1331529

In T416709#12264915, @atsuko wrote:

@mpopov what users should I add to analytics-experiment-users group? Will need to add them to 1328595.

• bearloga ✔
• phuedx
  • also needs to be in airflow-experiment-platform-ops
• kareid
  • also needs to be in airflow-experiment-platform-ops
  • @KReid-WMF should also be specified as the approver for analytics-experiment-users
• cjming
• sfaci

airflow-experiment-platform-ops: users added, see the list. the group is self-service, users can add themselves thru idm.

@mpopov diff 1328595 is pending the review by Infrastructure Foundations.

Change #1331615 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] Typo in airflow-experiment-platform-ops LDAP group manager list

https://gerrit.wikimedia.org/r/1331615

Change #1331615 merged by Atsuko:

[operations/puppet@production] Typo in airflow-experiment-platform-ops LDAP group manager list

https://gerrit.wikimedia.org/r/1331615

Change #1328595 merged by Atsuko:

[operations/puppet@production] Grant sudo privileges for the analytics-experiment-users group

https://gerrit.wikimedia.org/r/1328595

Change #1333817 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/dns@master] provision the airflow-experiment-platform DNS records

https://gerrit.wikimedia.org/r/1333817

Change #1333820 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/puppet@production] trafficserver: enabling access to airflow-experiment-platform.w.o

https://gerrit.wikimedia.org/r/1333820

Change #1333817 merged by Atsuko:

[operations/dns@master] provision the airflow-experiment-platform DNS records

https://gerrit.wikimedia.org/r/1333817

Change #1327553 merged by jenkins-bot:

[operations/deployment-charts@master] dse-k8s-eqiad: provision the airflow instance

https://gerrit.wikimedia.org/r/1327553

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2577

Adding experiment_platform instance

Change #1333820 merged by Atsuko:

[operations/puppet@production] trafficserver: enabling access to airflow-experiment-platform.w.o

https://gerrit.wikimedia.org/r/1333820

atsuko opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2602

Draft: Adding maintenance DAGs to experiment_platform

Change #1333867 had a related patch set uploaded (by Atsuko; author: Atsuko):

[operations/deployment-charts@master] airflow-test-k8s: remove growthbook-sync-related access

https://gerrit.wikimedia.org/r/1333867

Instance https://airflow-experiment-platform.wikimedia.org/ is up. @mpopov should I close the task, or do you want it open to track the migration of DAGs mentioned in the task description?

• DAG(s) that are created as part of T414105: SDS 2.2.6 Improve experiment event data data lake management
• DAGs that currently power automated analytics (https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/main/analytics_product/dags/experiment_analytics) (currently running under analytics-product Airflow)

atsuko merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2602

Adding maintenance DAGs to experiment_platform

Change #1333867 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-test-k8s: remove growthbook-sync-related access

https://gerrit.wikimedia.org/r/1333867