Change Details

Purpose: The current way to filter users is roughly looking for user agent strings that are browser names such as Mozilla, Chrome, Safari. We'd like to redo the logic behind who we consider as filtered users so that we have a better understanding of who is using our APIs. Pre work needed: * We need to rework the definition of what is considered filtered and decide whether it makes sense to pursue keeping this metric altogether before moving into implementation NOTE: The decision was made to remove this column from the REST API and Action API processes. This metric is not included in the GraphQL API processes, so no need to consider it there. There will be a follow up task to measure bot activity using the new bot scoring of `wmf.webrequest_v2` (T432344). Acceptance criteria: * ~~Distribution of user agent graphs on the Wikidata API dashboards are adjusted to reflect the new definition and logic of filtered user agents~~ [ ] Removal of `total_filtered_user_agents` from the REST API and Action API queries ([wmde/merge_requests!57](https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/57)) - [wd_action_api_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_action_api/metrics_monthly/wd_action_api_metrics_monthly.hql) - [wd_rest_api_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_rest_api/metrics_monthly/wd_rest_api_metrics_monthly.hql) - [wd_dump_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_dump/metrics_monthly/wd_dump_metrics_monthly.hql) [ ] Do the same for the create table queries ([wmde/merge_requests!57](https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/57)) [ ] Dropping the column from the tables [ ] Updating the Airflow DAGs to use the new commit SHA-1s for the updated queries [ ] Updating the Admin variables in Airflow to load in the new SHA-1s [ ] Refreshing the Superset datasets to reflect the changes to the tables [ ] Updating the charts on Superset to remove the values Deadline: 15.8.2026 (flexible but ideally before planning starts full force in September)

Purpose: The current way to filter users is roughly looking for user agent strings that are browser names such as Mozilla, Chrome, Safari. We'd like to redo the logic behind who we consider as filtered users so that we have a better understanding of who is using our APIs. Pre work needed: * We need to rework the definition of what is considered filtered and decide whether it makes sense to pursue keeping this metric altogether before moving into implementation NOTE: The decision was made to remove this column from the REST API and Action API processes. This metric is not included in the GraphQL API processes, so no need to consider it there. There will be a follow up task to measure bot activity using the new bot scoring of `wmf.webrequest_v2` (T432344). Acceptance criteria: * ~~Distribution of user agent graphs on the Wikidata API dashboards are adjusted to reflect the new definition and logic of filtered user agents~~ [x] Removal of `total_filtered_user_agents` from the REST API and Action API queries ([wmde/merge_requests!57](https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/57)) - [wd_action_api_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_action_api/metrics_monthly/wd_action_api_metrics_monthly.hql) - [wd_rest_api_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_rest_api/metrics_monthly/wd_rest_api_metrics_monthly.hql) - [wd_dump_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_dump/metrics_monthly/wd_dump_metrics_monthly.hql) [x] Do the same for the create table queries ([wmde/merge_requests!57](https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/57)) [ ] Dropping the column from the tables [ ] Updating the Airflow DAGs to use the new commit SHA-1s for the updated queries [ ] Updating the Admin variables in Airflow to load in the new SHA-1s [ ] Refreshing the Superset datasets to reflect the changes to the tables [ ] Updating the charts on Superset to remove the values Deadline: 15.8.2026 (flexible but ideally before planning starts full force in September)

Purpose: The current way to filter users is roughly looking for user agent strings that are browser names such as Mozilla, Chrome, Safari. We'd like to redo the logic behind who we consider as filtered users so that we have a better understanding of who is using our APIs. Pre work needed: * We need to rework the definition of what is considered filtered and decide whether it makes sense to pursue keeping this metric altogether before moving into implementation NOTE: The decision was made to remove this column from the REST API and Action API processes. This metric is not included in the GraphQL API processes, so no need to consider it there. There will be a follow up task to measure bot activity using the new bot scoring of `wmf.webrequest_v2` (T432344). Acceptance criteria: * ~~Distribution of user agent graphs on the Wikidata API dashboards are adjusted to reflect the new definition and logic of filtered user agents~~ [ x] Removal of `total_filtered_user_agents` from the REST API and Action API queries ([wmde/merge_requests!57](https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/57)) - [wd_action_api_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_action_api/metrics_monthly/wd_action_api_metrics_monthly.hql) - [wd_rest_api_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_rest_api/metrics_monthly/wd_rest_api_metrics_monthly.hql) - [wd_dump_metrics_monthly.hql](https://gitlab.wikimedia.org/repos/wmde/analytics/-/blob/main/hql/airflow_jobs/wd_dump/metrics_monthly/wd_dump_metrics_monthly.hql) [ x] Do the same for the create table queries ([wmde/merge_requests!57](https://gitlab.wikimedia.org/repos/wmde/analytics/-/merge_requests/57)) [ ] Dropping the column from the tables [ ] Updating the Airflow DAGs to use the new commit SHA-1s for the updated queries [ ] Updating the Admin variables in Airflow to load in the new SHA-1s [ ] Refreshing the Superset datasets to reflect the changes to the tables [ ] Updating the charts on Superset to remove the values Deadline: 15.8.2026 (flexible but ideally before planning starts full force in September)