Purpose: The current way to filter users is roughly looking for user agent strings that are browser names such as Mozilla, Chrome, Safari. We'd like to redo the logic behind who we consider as filtered users so that we have a better understanding of who is using our APIs.
Pre work needed:
- We need to rework the definition of what is considered filtered and decide whether it makes sense to pursue keeping this metric altogether before moving into implementation
NOTE: The decision was made to remove this column from the REST API and Action API processes. This metric is not included in the GraphQL API processes, so no need to consider it there. There will be a follow up task to measure bot activity using the new bot scoring of wmf.webrequest_v2 (T432344).
Acceptance criteria:
Distribution of user agent graphs on the Wikidata API dashboards are adjusted to reflect the new definition and logic of filtered user agents
- Removal of total_filtered_user_agents from the REST API and Action API queries (wmde/merge_requests!57)
- Do the same for the create table queries (wmde/merge_requests!57)
- Dropping the column from the tables
- Running the command ALTER TABLE wmde.TABLE_NAME DROP COLUMN total_filtered_user_agents;
- Updating the Airflow DAGs to use the new commit SHA-1s for the updated queries (data-engineering/airflow-dags!2411)
- Updating the Admin variables in Airflow to load in the new SHA-1s (once data-engineering/airflow-dags!2411 is merged)
- Refreshing the Superset datasets to reflect the changes to the tables
- Updating the charts on Superset to remove the values
Deadline: 15.8.2026 (flexible but ideally before planning starts full force in September)