Note that from the most recent discussions with WMF data engineering, there isn't a set workflow for getting information into a place where it can be accessed via the Public Superset instance. We would need to edit the DAG such that we include an export step for the data getting to a place where the public instance can access it. This would require some more research.

Maybe another thing to consider is whether we'd prefer to have Graphite be the end export location for the data and then make a Grafana dashboard for this? Grafana does serve as the current public facing data dashboards for Wikidata, so it might make sense to leverage it more.

Further checks on this: the dashboarding process for the public Superset seems to be based on a few preset databases that have the data from Wikimedia projects (see SQL Lab). As of now I'm doubting whether we'd be able to have active rights over one of these such that tables we'd generate in Airflow could be added to one and used for visualizations. I've asked in the WMDE data channel if there are people with domain knowledge for Prometheus that could help with setting up a process where it would be one of the targets of the Airflow jobs. This to me seems more simple, with the end situation being that we use the main Superset instance for data processes that rely on the data lake/private data access, and then use Prometheus/Grafana for dashboards that are meant to be public facing.

I'll wait for further word on people who can assist with this, but I'd suggest that we keep this task and switch it over to one based on Prometheus if we decide to go in that direction :)

Post a large discussion about this in the data-engineering-collab channel on Slack, the general findings for this are:

• The public Superset instance isn't suitable for this at this time and there's no time table for it to be (see above comments)
• A suggestion of putting this information on Wikistats was agreed to be too complex to setup and manage
  • We would need to use AQS 2 (Analytics Query Service) to make a service/API for this
• An initial suggestion from WMDE to target Prometheus with the DAG was decided against
  • It is possible to push data to Prometheus, but there are many complications with this
• A new suggestion is to leverage Turnilo for this
  • There is a private instance at turnilo.wikimedia.org
  • There are also public instances of this as seen at wiki-search-referrals.wmcloud.org
    • Wikitech docs for this can be found at wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/referrer_daily/Dashboard
    • The Turnilo dashboard is hosted on Cloud VPS
    • The code for the Turnilo instance can be found at github.com/wikimedia/research-api-endpoint-template/turnilo-druid
  • The way this would be achieved is that we would have the published datasets folder be another target of the DAG jobs, and we'd then ingest this TSV data via the Turnilo instance

This sounds like a good way forward, but the question of setting up the Turnilo instance and maintaining it then comes to mind. A big question is: how often are data pipelines supposed to be public, and would putting it all on a single Turnilo instance work well for our requirements?

Note that I've made T361214: Data Platform - Public dashboard support to explain our use case of a standardized public dashboard process :)

@karapayneWMDE, @Ifrahkhanyaree_WMDE: I'd say given the adoption of Superset that we're good to close this task. If and when WMF adds public metrics, we'll then be able to make a new task for this, and T361214: Data Platform - Public dashboard support that I made covers the requirements for this on our end :)