Change Details

== Wikidata Analytics Request == > This task was generated using the [Wikidata Analytics](https://phabricator.wikimedia.org/project/profile/5408) request form. Please use the task template linked on our project page to create issues for the team. Thank you! === Purpose === > Please provide as much context as possible as well as what the produced insights or services will be used for. Monitor Wikidata Co-Editors via Grafana. However, the current overall count of Co-Editors ([[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | first diagram on the page ]]) is misleading because it counts Co-Editors multiple times, once per Wikimedia project they edited in. === Specific Results === > Please detail the specific results that the task should deliver. To correct this, we need to create a new data pipeline that aggregates distinct Wikidata Co-Editors across all Wikimedia projects. This will ensure that each Co-Editor is only counted once. === Desired Outputs === > Please list the desired outputs of this task. **New** ✅ [x] Airflow pipeline to monitor the above metric - Nice to have: Output as CSV to https://analytics.wikimedia.org/published/datasets/wmde/analytics/ **Old** ❌ - Add new pipeline to [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/780749/5/hql/wikidata/coeditors_metrics.hql | coeditors_metrics.hql ]] to aggregate distinct Wikidata Co-Editors over all Wikimedia projects. - Update [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToGraphite.scala | HiveToGraphite.scala ]] to send the new data to Graphite. - Change the data source for the first diagram on [[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | wikidata-co-editors ]] to use the new aggregated data. ===Open questions=== * Should we backfill the historical data with the correct counts, and if so, is this possible using Graphite? * Yes if possible * Will be a separate task to keep this one scoped * Should we keep the old misleading diagram or replace it with the corrected one? * If we can keep a record of the old one somewhere and have the correct one as the one we all now refer to, that would be amazing * Will be kept in Grafana for now * Do we need engineering support to get the new data into the Graphite database? * Not needed given new scope * Do we need engineering support to display the new data in Grafana? * Not needed given new scope === Deadline === > Please make the time sensitivity of this task clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority. 10.1.2025 --- **Information below this point is filled out by the task assignee.** == Assignee Planning == === Sub Tasks === > A full breakdown of the steps to complete this task. [x] Check what the frequency of the job should be with stakeholders - Monthly [x] Setup job queries to get total co-editors - Done for both distinct and project based counts and including active coeditors in each [x] Test job queries on Pyspark [x] Setup Airflow DAG to run jobs - [x] Setup config and needed sensors - [x] Add query stage [x] Make table for the final output with the `analytics-wmde` user - `wmde.wd_coeditors_by_project_monthly` - `wmde.wd_coeditors_distinct_monthly` [x] Test Airflow DAG **without** CSV export step [x] Deploy Airflow DAG **without** CSV export [x] Create new task to get historical data into the data lake - Task id: T381572 [ ] Before **nice to have** further steps: Get approval from WMF for public data export via new Phab task - Task id: T381570 [ ] Test Airflow DAG **with** CSV export step [ ] Deploy new Airflow DAG **with** CSV export === Estimation === Estimate: 3 days for non-nice to haves Actual: === Data === > The tables that will be referenced in this task. - [wmf.mediawiki_history](https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,wmf.mediawiki_history,PROD)/Schema?is_lineage_mode=false&schemaFilter=) === Notes === > Things that came up during the completion of this task, questions to be answered and follow up tasks. - Note

== Wikidata Analytics Request == > This task was generated using the [Wikidata Analytics](https://phabricator.wikimedia.org/project/profile/5408) request form. Please use the task template linked on our project page to create issues for the team. Thank you! === Purpose === > Please provide as much context as possible as well as what the produced insights or services will be used for. Monitor Wikidata Co-Editors via Grafana. However, the current overall count of Co-Editors ([[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | first diagram on the page ]]) is misleading because it counts Co-Editors multiple times, once per Wikimedia project they edited in. === Specific Results === > Please detail the specific results that the task should deliver. To correct this, we need to create a new data pipeline that aggregates distinct Wikidata Co-Editors across all Wikimedia projects. This will ensure that each Co-Editor is only counted once. === Desired Outputs === > Please list the desired outputs of this task. **New** ✅ [x] Airflow pipeline to monitor the above metric - Nice to have: Output as CSV to https://analytics.wikimedia.org/published/datasets/wmde/analytics/ **Old** ❌ - Add new pipeline to [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/780749/5/hql/wikidata/coeditors_metrics.hql | coeditors_metrics.hql ]] to aggregate distinct Wikidata Co-Editors over all Wikimedia projects. - Update [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToGraphite.scala | HiveToGraphite.scala ]] to send the new data to Graphite. - Change the data source for the first diagram on [[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | wikidata-co-editors ]] to use the new aggregated data. ===Open questions=== * Should we backfill the historical data with the correct counts, and if so, is this possible using Graphite? * Yes if possible * Will be a separate task to keep this one scoped * Should we keep the old misleading diagram or replace it with the corrected one? * If we can keep a record of the old one somewhere and have the correct one as the one we all now refer to, that would be amazing * Will be kept in Grafana for now * Do we need engineering support to get the new data into the Graphite database? * Not needed given new scope * Do we need engineering support to display the new data in Grafana? * Not needed given new scope === Deadline === > Please make the time sensitivity of this task clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority. 10.1.2025 --- **Information below this point is filled out by the task assignee.** == Assignee Planning == === Sub Tasks === > A full breakdown of the steps to complete this task. [x] Check what the frequency of the job should be with stakeholders - Monthly [x] Setup job queries to get total co-editors - Done for both distinct and project based counts and including active coeditors in each [x] Test job queries on Pyspark [x] Setup Airflow DAG to run jobs - [x] Setup config and needed sensors - [x] Add query stage [x] Make table for the final output with the `analytics-wmde` user - `wmde.wd_coeditors_by_project_monthly` - `wmde.wd_coeditors_distinct_monthly` [x] Test Airflow DAG **without** CSV export step [x] Deploy Airflow DAG **without** CSV export [x] Create new task to get historical data into the data lake - Task id: T381572 [ ] Before **nice to have** further steps: Get approval from WMF for public data export via new Phab task - Task id: T381570 [ ] Test Airflow DAG **with** CSV export step [ ] Deploy new Airflow DAG **with** CSV export === Estimation === Estimate: 3 days for non-nice to haves Actual: 6 days given rewrite to make it more efficient === Data === > The tables that will be referenced in this task. - [wmf.mediawiki_history](https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,wmf.mediawiki_history,PROD)/Schema?is_lineage_mode=false&schemaFilter=) === Notes === > Things that came up during the completion of this task, questions to be answered and follow up tasks. - Note

== Wikidata Analytics Request == > This task was generated using the [Wikidata Analytics](https://phabricator.wikimedia.org/project/profile/5408) request form. Please use the task template linked on our project page to create issues for the team. Thank you! === Purpose === > Please provide as much context as possible as well as what the produced insights or services will be used for. Monitor Wikidata Co-Editors via Grafana. However, the current overall count of Co-Editors ([[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | first diagram on the page ]]) is misleading because it counts Co-Editors multiple times, once per Wikimedia project they edited in. === Specific Results === > Please detail the specific results that the task should deliver. To correct this, we need to create a new data pipeline that aggregates distinct Wikidata Co-Editors across all Wikimedia projects. This will ensure that each Co-Editor is only counted once. === Desired Outputs === > Please list the desired outputs of this task. **New** ✅ [x] Airflow pipeline to monitor the above metric - Nice to have: Output as CSV to https://analytics.wikimedia.org/published/datasets/wmde/analytics/ **Old** ❌ - Add new pipeline to [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/780749/5/hql/wikidata/coeditors_metrics.hql | coeditors_metrics.hql ]] to aggregate distinct Wikidata Co-Editors over all Wikimedia projects. - Update [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToGraphite.scala | HiveToGraphite.scala ]] to send the new data to Graphite. - Change the data source for the first diagram on [[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | wikidata-co-editors ]] to use the new aggregated data. ===Open questions=== * Should we backfill the historical data with the correct counts, and if so, is this possible using Graphite? * Yes if possible * Will be a separate task to keep this one scoped * Should we keep the old misleading diagram or replace it with the corrected one? * If we can keep a record of the old one somewhere and have the correct one as the one we all now refer to, that would be amazing * Will be kept in Grafana for now * Do we need engineering support to get the new data into the Graphite database? * Not needed given new scope * Do we need engineering support to display the new data in Grafana? * Not needed given new scope === Deadline === > Please make the time sensitivity of this task clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority. 10.1.2025 --- **Information below this point is filled out by the task assignee.** == Assignee Planning == === Sub Tasks === > A full breakdown of the steps to complete this task. [x] Check what the frequency of the job should be with stakeholders - Monthly [x] Setup job queries to get total co-editors - Done for both distinct and project based counts and including active coeditors in each [x] Test job queries on Pyspark [x] Setup Airflow DAG to run jobs - [x] Setup config and needed sensors - [x] Add query stage [x] Make table for the final output with the `analytics-wmde` user - `wmde.wd_coeditors_by_project_monthly` - `wmde.wd_coeditors_distinct_monthly` [x] Test Airflow DAG **without** CSV export step [x] Deploy Airflow DAG **without** CSV export [x] Create new task to get historical data into the data lake - Task id: T381572 [ ] Before **nice to have** further steps: Get approval from WMF for public data export via new Phab task - Task id: T381570 [ ] Test Airflow DAG **with** CSV export step [ ] Deploy new Airflow DAG **with** CSV export === Estimation === Estimate: 3 days for non-nice to haves Actual: 6 days given rewrite to make it more efficient === Data === > The tables that will be referenced in this task. - [wmf.mediawiki_history](https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,wmf.mediawiki_history,PROD)/Schema?is_lineage_mode=false&schemaFilter=) === Notes === > Things that came up during the completion of this task, questions to be answered and follow up tasks. - Note