## Problem statement
As a Wikimedia Deutschland Data Analytics team member, I would like to be able to leverage a standardized process to conveniently create publicly available dashboards from data that currently resides in HDFS so that insights can be presented to non-WMDE/WMF staff.
## Context
WMDE would like to make our Wikidata REST API metrics available for the public, but the process to do this isn't something that has been standardized. These metrics are generated by an [Airflow DAG](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/wmde/dags/wd_rest_api_metrics/wd_rest_api_metrics_monthly_dag.py?ref_type=heads) that leverages [jobs defined on GitLab](https://gitlab.wikimedia.org/repos/wmde/analytics/-/tree/main/hql/airflow-jobs/wd_rest_api_metrics?ref_type=heads).
Ideas brought up in the [original Slack discussion](https://wikimedia.slack.com/archives/CSV483812/p1711539117956279) were:
- Leveraging [Wikistates](https://stats.wikimedia.org/#/all-projects) (high effort)
- This would require creating a service and API via [AQS 2](https://wikitech.wikimedia.org/wiki/AQS_2.0)
- Pushing data to [Prometheus](https://wikitech.wikimedia.org/wiki/Prometheus) such that it can be used in [Grafana](https://wikitech.wikimedia.org/wiki/Grafana) (strongly discouraged)
- Prometheus only supports counters and timings
- For counters, it assumes additivity – that is, a weekly count is day 1 count + ... + day 7 count
- It's impossible to control the time of your data point (it's the current time at which you push the metric)
- Adding the [published datasets](https://analytics.wikimedia.org/published/datasets) directories as a target of the DAG jobs where TSVs would be saved and then ingested via an open [Turnilo](https://wikitech.wikimedia.org/wiki/Analytics/Systems/Turnilo) instance (best solution to date)
- An example of this is [wiki-search-referrals.wmcloud.org](https://wiki-search-referrals.wmcloud.org/)
- [Documentation for the above dashboard](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/referrer_daily/Dashboard)
## General ideas
- It would be great if the public dashboards were an instance of WMF long-term supported data visualization software
- Ideally the public dashboards could be directly integrated into current data pipeline/Airflow based workflows
- Including data stakeholder/admin oversight of what is added to this system would be ideal to protect against the inclusion of PII, regions on the [Country and Territory Protection List](https://foundation.wikimedia.org/wiki/Legal:Country_and_Territory_Protection_List), etc
- Oversight of the Protection List and updating the public dashboards would be necessary
- Maybe a specific database within HDFS could be the source where the public dashboards have access?
- Admins would be the only ones who could create tables within this database
- This would prevent the public dashboards from presenting information that has not been actively checked for vulnerabilities