The following tables will need to be loaded for the metrics planned as of now in T366044. The tables are:
The idea is to have Spark jobs (similar to sqoop) that will simply fetch and load to the destination tables, and have separate pipelines for necessary aggregations.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | KCVelaga_WMF | T365813 Develop a unified Content Translation (CX) metrics dashboard | |||
| Resolved | KCVelaga_WMF | T393561 Setup pipelines to load CX extension tables into Data Lake, at wmf_product | |||
| Resolved | KCVelaga_WMF | T382706 Data pipeline to load cx_translators to Data Lake, at wmf_product | |||
| Resolved | KCVelaga_WMF | T393559 Data pipeline to load cx_translations to Data Lake, at wmf_product | |||
| Resolved | KCVelaga_WMF | T393560 Data pipeline to load cx_corpora to Data Lake, at wmf_product |
Mentioned in SAL (#wikimedia-operations) [2025-05-14T11:17:05Z] <kcvelaga@deploy1003> Started deploy [airflow-dags/analytics_product@22aa307]: T393561
Mentioned in SAL (#wikimedia-operations) [2025-05-14T11:17:59Z] <kcvelaga@deploy1003> Finished deploy [airflow-dags/analytics_product@22aa307]: T393561 (duration: 01m 10s)