Change Details

=== Purpose === When want to monitor [Wikidata Co-Editors via Grafana](https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now). However, what we currently show as overall Wikidata Co-Editors (first diagram on the page) is misleading at best. === Scope === The main problem of the diagram showing overall Co-Editors is likely that Co-Editors are currently counted multiple times (once per every wiki that they were editing in). This needs to be corrected. We therefore need to add a new data pipeline aggregating distinct Wikidata Co-Editors over all Wikimedia projects. === Desired output === [] Add new pipeline to [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/780749/5/hql/wikidata/coeditors_metrics.hql | coeditors_metrics.hql ]] [] Update [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToGraphite.scala | HiveToGraphite.scala ]] to send the new data to Graphite [] Change the data source for the first diagram on [[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | wikidata-co-editors ]] ===Open questrions=== * Does this need to be backfilled? Is this even possible using Graphite? * Should we still keep the old misleading diagram? * Do we need engineering support to get the new data in the Graphite db? * Do we need engineering support to show the new data in Grafana? === Urgency === When this task should be completed by. If this task is time sensitive then please make this clear. Please also provide the date when the output will be used if there is a specific meeting or event, for example. DD.MM.YYYY --- **Information below this point is filled out by the Wikidata Analytics team.** == General Planning == Information is filled out by the analytics product manager. == Assignee Planning == Information is filled out by the assignee of this task. === Estimation === Estimate: Actual: === Sub Tasks === Full breakdown of the steps to complete this task: [ ] subtask === Data to be used === See [Analytics/Data_Lake](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake) for the breakdown of the data lake databases and tables. The following tables will be referenced in this task: - link_to_table === Notes and Questions === Things that came up during the completion of this task, questions to be answered and follow up tasks: - Note

=== Purpose === Monitor Wikidata Co-Editors via Grafana. However, the current overall count of Co-Editors ([[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | first diagram on the page ]]) is misleading because it counts Co-Editors multiple times, once per Wikimedia project they edited in. === Scope === To correct this, we need to create a new data pipeline that aggregates distinct Wikidata Co-Editors across all Wikimedia projects. This will ensure that each Co-Editor is only counted once. === Desired output === [] Add new pipeline to [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/780749/5/hql/wikidata/coeditors_metrics.hql | coeditors_metrics.hql ]] to aggregate distinct Wikidata Co-Editors over all Wikimedia projects. [] Update [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToGraphite.scala | HiveToGraphite.scala ]] to send the new data to Graphite. [] Change the data source for the first diagram on [[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | wikidata-co-editors ]] to use the new aggregated data. ===Open questions=== * Should we backfill the historical data with the correct counts, and if so, is this possible using Graphite? * Should we keep the old misleading diagram or replace it with the corrected one? * Do we need engineering support to get the new data into the Graphite database? * Do we need engineering support to display the new data in Grafana? === Urgency === When this task should be completed by. If this task is time sensitive then please make this clear. Please also provide the date when the output will be used if there is a specific meeting or event, for example. DD.MM.YYYY --- **Information below this point is filled out by the Wikidata Analytics team.** == General Planning == Information is filled out by the analytics product manager. == Assignee Planning == Information is filled out by the assignee of this task. === Estimation === Estimate: Actual: === Sub Tasks === Full breakdown of the steps to complete this task: [ ] subtask === Data to be used === See [Analytics/Data_Lake](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake) for the breakdown of the data lake databases and tables. The following tables will be referenced in this task: - link_to_table === Notes and Questions === Things that came up during the completion of this task, questions to be answered and follow up tasks: - Note

=== Purpose === When want to monitor [Wikidata Co-Editors via Grafana](https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now). HoweverMonitor Wikidata Co-Editors via Grafana. However, the current overall count of Co-Editors ([[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | first diagram on the page ]]) is misleading because it counts Co-Editors multiple times, what we currently show as overallonce per Wikidata Co-Editors (first diagram on the page) is misleading at bestmedia project they edited in. === Scope === The main problem of the diagram showing overall Co-Editors is likely that Co-Editors are currently counted multiple times (once per every wiki that they were editing in).o correct this, This needs to be correctedwe need to create a new data pipeline that aggregates distinct Wikidata Co-Editors across all Wikimedia projects. We therefore need to add a new data pipeline aggregating distinct WikidataThis will ensure that each Co-Editors over all Wikimedia projects. is only counted once. === Desired output === [] Add new pipeline to [[ https://gerrit.wikimedia.org/r/c/analytics/refinery/+/780749/5/hql/wikidata/coeditors_metrics.hql | coeditors_metrics.hql ]] to aggregate distinct Wikidata Co-Editors over all Wikimedia projects. [] Update [[ https://gerrit.wikimedia.org/r/plugins/gitiles/analytics/refinery/source/+/refs/heads/master/refinery-job/src/main/scala/org/wikimedia/analytics/refinery/job/HiveToGraphite.scala | HiveToGraphite.scala ]] to send the new data to Graphite. [] Change the data source for the first diagram on [[ https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now | wikidata-co-editors ]] to use the new aggregated data. ===Open questrions=== * Does this need to be backfilled?* Should we backfill the historical data with the correct counts, Is this evenand if so, is this possible using Graphite? * Should we still keep the old misleading diagram? or replace it with the corrected one? * Do we need engineering support to get the new data ininto the Graphite db? atabase? * Do we need engineering support to showdisplay the new data in Grafana? === Urgency === When this task should be completed by. If this task is time sensitive then please make this clear. Please also provide the date when the output will be used if there is a specific meeting or event, for example. DD.MM.YYYY --- **Information below this point is filled out by the Wikidata Analytics team.** == General Planning == Information is filled out by the analytics product manager. == Assignee Planning == Information is filled out by the assignee of this task. === Estimation === Estimate: Actual: === Sub Tasks === Full breakdown of the steps to complete this task: [ ] subtask === Data to be used === See [Analytics/Data_Lake](https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake) for the breakdown of the data lake databases and tables. The following tables will be referenced in this task: - link_to_table === Notes and Questions === Things that came up during the completion of this task, questions to be answered and follow up tasks: - Note