@Ifrahkhanyaree_WMDE , could you answer these questions:

1. If we are able to backfill historical data, should this be done?
2. Do we keep the old diagram or replace it with the correct one? (https://grafana.wikimedia.org/d/000000560/wikidata-co-editors?orgId=1&from=now-5y&to=now)

This is @Ifeatu_Nnaobi_WMDE's ticket so handing over to her

Thanks @Ifrahkhanyaree_WMDE and @karapayneWMDE.

1. Yes.
2. If we can keep a record of the old one somewhere and have the correct one as the one we all now refer to, that would be amazing.

Thank you

Hi!

Just connecting some dots:

Graphite will be made read-only in 2025. There is a task: T372855: Disable Data Platform Engineering generated graphite metrics and dashboards, but it will not be easy or maybe even not possible to use prometheus.

Expanding on the above comment from @Ottomata and the Open questions discussed above:

1. Should we backfill the historical data with the correct counts, and if so, is this possible using Graphite?
   • As Graphite will be read only in 2025, we should not factor a backfill to Graphite into this work
   • Via the comment linked above and another comment on the same task, we can...
     • get historical metrics into Prometheus (way beyond the scope of this task and requires WMF support/work)
     • not backfill
2. Should we keep the old misleading diagram or replace it with the corrected one?
   • We can follow @Ifeatu_Nnaobi_WMDE's direction on this and keep the data and the old graph :)
   • The old data can just live on Graphite and be visible in Grafana, but we should mark the resulting dashboard as deprecated/archived

A question that comes to mind: Does the backfilled data have enough value that we would consider using Airflow for this instead? The data in this case would be on the private Hive database rather than Prometheus. This would mean that we wouldn't have visualization until we have public dashboarding (T361214). The path forward on this would be much cleaner though as the other Open questions above of engineering support for getting the data where it needs to be (Question 3) and getting the visualizations switched over (Question 4) would be put to the side.

Generally speaking some options forward could be:

1. Switch the current process over to Prometheus (necessary new plan based on the current task)
   • Grafana visualization from the start
   • Inclusion of historical metrics depends on WMF Engineering and might not happen
   • Getting data into Prometheus depends on WMDE Engineering support as I've not been onboarded to these processes
2. Use Airflow as we do with other pipelines and put the data in the Hive database
   • No visualization at first, so SWE Analytics needs to provide Product with the metrics on a monthly basis (this could be within the monthly repeating tasks)
     • I can also do a training with @Ifeatu_Nnaobi_WMDE on getting the data
   • Likely inclusion of historical metrics
   • Way forward is clear, no dependence on WMDE/WMF Engineering

Sticking point above is whether we need the metrics to be community facing from the start. I'm not sure how Product uses these metrics though, so some feedback on the above would be appreciated :)

Hi @AndrewTavis_WMDE,

I'm responding to your comment above here.

1. To the question: Should we keep the old misleading diagram or replace it with the corrected one? I confirm your understanding - "The old data can just live on Graphite and be visible in Grafana, but we should mark the resulting dashboard as deprecated/archived"

2. General points on moving forward

2.1. Historical data does not have to live on Grafana. It can be someone else in an archived form. My question is if you want to go with the first option you've proposed, is there any WMDE Engineering support that I can help prioritize? I'm not sure who exactly you need support from.

2.2 The second option works for me :) This option sounds like the most sensible given the restraints you've mentioned. Metrics do not need to be community-facing from the start. We need them primarily for reporting and working on the product strategy and can separately share important communication with the community where relevant.

Thank you so much for laying all your points across so thoughtfully and please let me know if there's anything I haven't addressed.

Hi @Ifeatu_Nnaobi_WMDE :)

Thanks for the response and further questions! Replies to them:

2.1. I think that @karapayneWMDE would be able to organize the support that we need for Prometheus/Grafana if need be. Thanks for the offer!

2.2. Sounds good to me! Have started on the needed queries, and should be able to get a MR up for this next week :)

Thanks for the thorough responses! I'll send along some updates when I have them, and rework the task description to match what we're discussing (will keep the old one for documentation).

Testing for the Spark SQL jobs was finished last week and the merge request is now up :)

• Merge request: airflow-dags#964

Outputs of the query tests are in the MR and are listed below. Based on discussions with @Ifeatu_Nnaobi_WMDE, we now have this split into two DAG pipelines that lead to a distinct count of coeditors in one table and a wiki based count of coeditors in another table. This will allow the team to measure Wikidata coeditring on individual pilot wikis that they're working with.

Beyond this we agreed that splitting the outputs to include monthly active coeditors as well as content and non-content pages would be good to include so that the results here can cover as many possible scenarios as possible. Please note that this is user information, so outputs <25 need to be reported as <25 and not their actual value.

I went ahead and also include very active monthly coeditors (>=100 edits on each and restricting active coeditors to <100), but this can be removed if there's no value. I felt that including them as a separate group made sense given how many there already are and the goals of the team to increase coediting.

• wd_coeditors_distinct_monthly

• wd_coeditors_by_wiki_monthly (top 5 rows by total_active_wd_coeditors)

Next steps:

• Testing the DAGs on an Airflow instance using my schema
• Deploying them
• Moving onto historical metrics in T381572 when directed

DAG tests have been successful, so we need one more review that I have arranged for tomorrow and then we'll merge this in to deploy it 😊

The DAG has now been deployed 🚀 Thanks to @awight for the help with reviewing, fine tuning the logic and the suggestions to speed it up so this isn't using as many resources 🙏

The results are available in:

• wmde.wd_coeditors_by_wiki_monthly: Similar to prior dataset, this is Wikidata coeditors across all other wikis.
• wmde.wd_coeditors_distinct_monthly: The unique coeditors across all projects that were requested.

@Ifeatu_Nnaobi_WMDE, please let me know if you'd like to take a look at the datasets at some point or if you need any support in accessing it yourself :)

Once the work has been approved by @Ifeatu_Nnaobi_WMDE, this DAG task will move to Stalled on our board while the data release request in T381570 is processed.

Everything looks good! Thank you so much.

Thanks, @Ifeatu_Nnaobi_WMDE! Moving the remainder of the work for this task to stalled as we're waiting on the T381570 data release request for the CSVs going to the published datasets directories.

Moving to in review as WMDE SWE Analytics has decided to not export to the published datasets anymore - so this task is finalized.

Further related work in T388634.