Page MenuHomePhabricator

Decomission dedicated growthbook airflow pipeline(s)
Open, Needs TriagePublic

Description

T414105: SDS 2.2.6 Improve experiment event data data lake management required us to create dedicated airflow pipeline(s) to help boost Growthbook query performance. The main improvement being copying experiment data over to iceberg tables with timestamp partitions (in lieu if dd/mm/yyyy partitions in hive). Once Refine is migrated to Iceberg, this pipeline will become redundant and can be decommissioned.

Caveat: The dedicated growthbook pipeline will also perform filtering (WHERE action = 'X'). How that should be handled by native Iceberg tables, and if they need to be handled at all, will need to be thought about.