WMDE Analytics Request

This task was generated using the WMDE Analytics request form. Please use the task templates linked on our project page to create tasks for the team. Thank you!

Why (Context & Decision)

What problem are you trying to solve, and what decision will this analysis inform? Briefly explain the organisational or strategic context, why this matters now, and what action will be taken based on the outcome.

WMDE has the goal of having all KPI metrics for our initiatives being in a common place for easy exploration. As of now the Wikidata Embeddings Project has a separate dashboard for metrics that requires an API key that must be provided for each stakeholder. These metrics are derived via calls to an API. These metrics would ideally be on Superset, with calls to the same API being possible in an Airflow DAG.

Note: This would be business logic within an Airflow DAG, which is poor practice, but creating a Python project to simply run a few requests is not a good use of Analytics time.

What (Scope & Output)

Describe the specific question(s), metrics, segments, or deliverables (e.g., dashboard, deep dive, experiment analysis), including any relevant definitions or constraints.

• Outlining all metrics and making sure that KPIs have been derived
  • At least 20 unique user agents access the Wikidata Vector Database API each month
  • Maintain at least 4,000 page views per quarter for the Wikidata vector database website
  • The vector database contains embeddings in at least 5 languages and at least 3,000 requests are made to query data outside of English Wikidata
• Creating tables in the data lake for the metrics
• Creating a DAG that makes requests based on wd_dump_size_monthly_dag.py
  • We would need to save the API key in an environment (maybe HDFS) to be used in the DAG

By When (Timing & Priority)

Provide a clear deadline, any key milestones (e.g. launch, leadership review), and note if timing is flexible or fixed.

DD.MM.YYYY

Information below this point is filled out by the analyst.

Sub Tasks

A breakdown of the steps to complete this task.

• Verify that all metrics can be derived from the API endpoint
• Create needed Iceberg tables
• Write DAG to make API requests and populate tables
  • Note: Experiment with moving the request function into a separate file in the airflow-dags repo
  • Note: Experiment with the @task(do_xcom_push=True) decorator
• Test and deploy DAG

Data

The tables that will be referenced in this task and the sample sizes from them that will be used.

• link_to_table