I retitled to make it clear that the metrics are not about Airflow, but are about pipeline jobs that are scheduled by Airflow

@lmata is there a timeline for this sunsetting project? That will help us figure out when to focus on this work. Also is it tied to an OKR?

Hi @VirginiaPoundstone, the proposed timeline is to go read-only (Graphite) by the end of Q3-FY24/25; this is part of WE5.1.2. More info here: https://wikitech.wikimedia.org/wiki/Graphite/Deprecation_Roadmap. General announcements will follow in the next few days. Thanks!

Great. @Milimetric let's include this in the temp accounts work on pipelines.

Hi all!

Just FYI, in case you aren't aware. It will likely be difficult or impossible to migrate airflow generated metrics from Graphite to Prometheus.

Prometheus does not support producing metrics for historical timestamps. Any metrics produced to it are given the current timestamp.

Graphite did support historical metrics; this is why it was used instead of Prometheus.

If the output of the job does not use historical metrics, then Prometheus will work just fine! Usually though, airflow scheduled jobs are expected to complete for a specific time period, and can be delayed and rerun in the past.

It may be possible, but not easy to do this.
https://dasl.cc/2024/07/07/setting-custom-timestamps-for-prometheus-metrics/

We'd have to work with SRE observability to see if setting custom timestamp and exporting historical metrics would be feasible.

I believe @otto's point above means we have to turn off these metrics for now, until we can find a place to send them that will work with historical metrics. There are really only three options:

1. convince Prometheus to accept historical metrics
2. do not retry missed DAG runs for Prometheus-bound pipelines and accept the resulting gaps in data
3. maintain a different data store for historical metrics (preferably something we already have like an Iceberg table)

Druid is similar to Prometheus; it may be possible to push these metrics there.

However, I believe the reason they are in prometheus is to enable public dashboarding with Grafana, so Druid + Superset might not be sufficient.

3. maintain a different data store for historical metrics (preferably something we already have like an Iceberg table)

+1
I tried this some years ago with the anomaly_detection table, but there were problems with Hive partitioning affecting Superset's querying performance.
Now, with Iceberg, that should be cool and useful!

@lmata what is the timeline for sunsetting Graphite?

In T372855#10541618, @VirginiaPoundstone wrote:

@lmata, what is the timeline for sunsetting Graphite?

@VirginiaPoundstone The plan is to turn off metric intake by the end of this quarter, assuming a migration target of 90% is met by then;,. Otherwise, I've heard its possible to extend it to Q4, but that is not the preferred outcome.

The api metrics had been disabled. The wikidata metrics is pending,

Moving this ticket to blocked until towards end of march so that we can give the owners of wikidata metrics time to convert the metrics as discussed in this ticket.

@AndrewTavis_WMDE can we work with the date 28th March to finally disable the wikidata metric job in airflow?

In T372855#10619318, @Snwachukwu wrote:

@AndrewTavis_WMDE can we work with the date 28th March to finally disable the wikidata metric job in airflow?

Thanks for checking in on this, @Snwachukwu! I've communicated the above to stakeholders over here: @Ifrahkhanyaree_WMDE and @karapayneWMDE. I'm not sure that I'll be able to work on this this week, but if not I'd then have two weeks until the 28th, so basically one week and a few days to implement and a few days grace to make sure that all's working. This is certainly doable, but I'll need to have other things de-prioritized :) I'll confirm the decision on our end in the coming days!

Hey @Snwachukwu 👋 We're able to confirm March 28th as a goal for you all to disable the Wikidata metrics DAG. I'll try to get the individual DAGs migrated a few days before, and will be in touch on Slack if something comes up :) Will make new tasks as sub-tasks of T377352 today and subscribe you to them so you can follow the progress 😊

Hi @AndrewTavis_WMDE . Thank you for the confirmation. Please feel free to reach out if you need any form of support.

See mentioned tasks above for the individual migration tasks :) I'll let you know if anything's needed on my end, @Snwachukwu!

Cross posting from https://phabricator.wikimedia.org/T377352#10679786:

GitLab:data-engineering/airflow-dags#1185 adds five new DAGs for the following processes within wikidata_metrics_to_graphite_daily_dag that will populate tables in the Data Lake:

• wd_article_placeholder_metrics_daily: Runs a new version of wikidata_articleplaceholder_metrics.hql
• wd_entity_schema_namespace_metrics_daily: Runs a new version of wikidata_entity_schema_namespace_metrics.hql
• wd_reliability_metrics_daily: Runs a new version of wikidata_reliability_metrics.hql
• wd_special_entity_data_metrics_daily: Runs a new version of wikidata_specialentity_data_metrics.hql
• wd_special_entity_schema_text_metrics_daily: Runs a new version of wikidata_special_entity_schema_text_metrics.hql

Tests have passed except for the approval job. The plan is to merge and auto-deploy tomorrow and then run the metrics from the 1st of January on. We'll then be good to disable wikidata_metrics_to_graphite_daily_dag by the March 28th deadline.

Cross posting from https://phabricator.wikimedia.org/T377352#10683430

GitLab:data-engineering/airflow-dags#1185 is merged, all five DAGs are deployed and we're all green for daily job success from January 1st 2025 on 💚 We're thus ready to turn off wikidata_metrics_to_graphite_daily_dag :)

@AndrewTavis_WMDE This is good news. Thank you so much for the effort put into this.

Happy to help, @Snwachukwu, and thanks for your coordination here! 😊