Our central Metrics FY26-27 dashboard should be available to all WMF staff members.
Individual objective metrics are directly loaded and visualized into the dashboard, but there are also link outs to more detailed KR-level dashboards and metrics that need to be available to all staff members.
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Group jobs, add dependencies, set schedules | repos/data-engineering/airflow-dags!2578 | milimetric | reconfigure-fy_2026_2027-dbt-jobs | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Milimetric | T431697 Metrics Superset dashboard access for all WMF | |||
| Resolved | BTullis | T432104 Create a new PostgreSQL cluster database with read-only from Superset and write access from Spark | |||
| Open | APizzata-WMF | T432017 Move all FY2026-2027 chart data to PostgreSQL |
We have agreed to move forward with a pragmatic decision to implement a new back-end PostgreSQL database cluster in Superset that will hold these metrics.
It will be optimised for performance and for ease of access so that, unlike our Presto/Hive and Presto/Iceberg databases, there will not be a second layer of authorisation after the user has successfully logged into Superset.
We also want to make sure that query results can be efficiently cached by Superset.
This PostgreSQL cluster will have an ingress path, so that our Spark jobs running on YARN will be able to take the results from Hive on HDFS and load them into PostgreSQL.
@Milimetric has outlined this part of the work in T432017: Move all FY2026-2027 chart data to PostgreSQL
Please note that this is not currently considered a full Data Warehouse solution, but it is intended to help us to meet a specific and time-constrained requirement to make these metrics available more widely within the Foundation.
We already have ongoing strategic work in T428255: Define a comprehensive security model for the Data Platform S3/Ceph services and to migrate away from HDFS/YARN to Ceph/Kubernetes. This solution has been selected in order not to put the completion of that work on the critical path for the availability of a high-performance metrics dashboard.
This ticket has a lot in common with T258962: Investigate easier methods for WMF staff to access Superset, which is roughly six years old.
I don't want to merge that in and lose its history, but I'm just mentining it to establish the link.
Change #1314629 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] dse-k8s-eqiad/cloudnative-pg: add postgresql-superset-metrics to the tenant namespaces
Change #1314629 merged by Brouberol:
[operations/deployment-charts@master] dse-k8s-eqiad/cloudnative-pg: add postgresql-superset-metrics to the tenant namespaces
Change #1314804 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/dns@master] Visually re-organize the dse svc.eqiad.wmnet section
Change #1314805 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/dns@master] Make +postgresql-superset-metrics.svc.eqiad.wmnet point to the dse-k8s-eqiad ingress
Change #1314804 merged by Brouberol:
[operations/dns@master] Visually re-organize the dse svc.eqiad.wmnet section
Change #1314805 merged by Brouberol:
[operations/dns@master] Make +postgresql-superset-metrics.svc.eqiad.wmnet point to the dse-k8s-eqiad ingress
I'm assigning this to myself because the PostgreSQL cluster is ready and just waiting on me to mirror the model output and swap the graph datasets. I'm behind on this while fixing another bug with the models, so if anyone else wants to take it, I can help explain what needs to happen.
Moving to "waiting" as the work is being done by @Milimetric. Ping us if you need help from DPE SRE.
milimetric opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2578
Group jobs, add dependencies, set schedules
amastilovic merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2578
Group jobs, add dependencies, set schedules