@Pablo thanks for putting this task together. Is this task related to T342012? If yes, can we have one parent task for the support the Research team gives for SDS 1.1 and track both your work and Tanja's under that?
@Miriam cc.

@leila: this was authored by @Miriam and my understanding is that both tasks refer to the same work... is it ok to directly merge T342012 and this one?

Thanks for pointing this out @Pablo, I missed this task when I made T342012! I went ahead and merged in the task I made and we can use this one :)

Weekly updates

• @TAndic and I have scheduled a 1:1 for next week to start discussing this task.

Weekly updates

• @TAndic and I met and shared reports, literature and resources of interest and also discussed preliminary challenges to be addressed when characterising WM community health.

Weekly updates

• @TAndic and I reviewed the mapping created by @JAnstee_WMF of existing data and metrics for contributors.
  • Added/expanded documentation about metrics of the following projects: Knowledge Integrity Risk Observatory, Community Health Metrics, Standardized research metrics, Community Insights, and Developer Satisfaction Survey.
• The first contributors working group session has been scheduled for next week. @TAndic and I connected this week in order to draft questions and ideas for the session.

Weekly updates

• @TAndic and I attended the first contributors working group session, where some key concepts were discussed: contributor, community health, movement governance and health.
• A second session has been scheduled for next Tuesday to conclude the discussion on actionability criteria and the mapping of the existing community health metrics.

Weekly updates

• @TAndic and I attended the second contributors working group session, where participants shared and discussed reflections on actionability and what do we mean with "community health".
• As admins seem to be a relevant focus for contributors, I started to parse logs to retrieve the adminship status and period of editors on all Wikipedias (I am encountering some problems due to multiple logging formats, editors who changed their alias, and the inavailability of some early logs).
• Nex actions will be about expanding the mapping of existing data and metrics (we already contributed to that task but we could also enrich the mapping with some theoretical approaches from the Wikimedia research community, e.g., https://wikiworkshop.org/2023/papers/WikiWorkshop2023_paper_21.pdf)

Weekly updates

• Created a a first version of the dataset of sysops in each wiki including the start and end timestamps after parsing logs of every wiki and also Meta, since some sysops were added/removed by stewards. The parsing process was challenging because:
  • the log structure was not always the same,
  • some editors had different aliases when they were added and removed,
  • some early logs are not available, as logging was not always a part of MediaWiki (i.e., I found logs that removed editors from the sysop group without a previous log adding them).

• The first challenge was solved, the second one is partially addressed but the third one cannot be fully addressed as it relateds to non-available data (one could add current sysops that are missing in the dataset and set an approximate start_timestamp to those editors and to those missing a value in start_timestamp). Therefore, the dataset is not complete

• To describe the dataset, two plots with the number of sysops over time (monthly) on the largest wikis are attached below. Some seem to follow the well-known rise and decline pattern (a clustering algorithm could be run to better distinguish these patterns). Interestingly, some wikis start the decline period after a sudden drop (e.g., enwiki, eswiki). These cases are about the times when admin policies were set and, suddenly, a number of inactive syops lose their user rights (e.g., https://meta.wikimedia.org/wiki/?diff=prev&oldid=2697150 https://meta.wikimedia.org/?oldid=5338351#Several_usernames.40eswiki).
  
  

• As one approach to "community health" is about community resilience to project capture, another call with the UW researchers already working on governance capture has been scheduled for mid September.

Weekly updates

• Added a new item (a theoretical work on governance and project capture) to the mapping doc of existing data & metrics.
• We suggested having a group discussion about SDS 1.1 during the RDS offsite. The approach to such a session is currently being discussed with @JAnstee_WMF and @Mayakp.wiki (thanks again for your feedback :)).

Weekly updates

• @TAndic and I met with Zarine and @mako. We confirmed that last FY's idea of extending their theoretical approach with quantitative empirical work is more timely than ever, given its potential to bring complementary insights to SDS 1.1.
• I will discuss with @leila a proposal to explore a formal collaboration.

@Pablo thanks for this update and looking forward to our conversation. In preparation for our conversation: I'd like to understand the current state of the third metric that is being developed, what is ahead for that work (timelines and milestones), and how complementary research in this space at this moment can help. My understanding is that the stable metric needs be delivered in January. Depending on what you all have in mind this may be a time sensitive conversation so please book an office hour if you assess that it's better if we talk sooner.

Thank you @leila for your comment, it is not in my plans that this collaboration affects the calendar and workflows already set by @JAnstee_WMF. I have just booked an office hour to advance this conversation between the two of us.

Weekly updates

• Review of the Contributors: Community Health Metric Mapping Q1 FY23/24 spreadsheet to assess the target use case associations and to address the questions/comments raised by @JAnstee_WMF on scientific or empirical support (thanks for such an organized doc!)

Weekly update

• @JAnstee_WMF has started planning the two sessions on this project for the R&DS Onsite (a 15 minute overview and a 60 minute discussion session). We are currently reviewing some tbd aspects (roles/responsibilities, design questions, deck outline) to be decided in a sync meeting next week.

Weekly update

• Review of the new section "Step 3. Highest Potential Contributors Metric Review and Criteria Mapping" created by @JAnstee_WMF on the Miro board to examine pros/cons of the candidate metrics. Two sessions have been scheduled for next week to discuss this evaluation and choose a metric.
• @TAndic and I also met with @JAnstee_WMF to discuss the content and approaches for the R&DS Onsite related sessions.

Weekly update

• @TAndic and I attended the Contributors Metric Working Group Session #3 and provided feedback to the Miro board for metric evaluation and prioritization (awaiting voting results).
• We also met with @JAnstee_WMF and @Mayakp.wiki to prepare the R&DS Onsite related sessions on this project.

Weekly update

• Last week we presented our work in progress at the RDS offsite and gathered feedback.

• @JAnstee_WMF has shared challenge points and potential options for the selected metric, along with known pros and cons. A Google form will be shared early next week with the working group to rate options and make decisions.

Weekly update

• We have provided our feedback in the Google form for rating options on specifics for the proposal of the initial metric: approaches to define admins, thresholds of "stability", and criteria for aggregation and optimization to Annual Plan use cases.
• @JAnstee_WMF has drafted a 1-pager of the metric recommendation along with the optimization recommendations for early input before sharing with the Foundation Level Metrics Committee.

Weekly update

• No actions by @TAndic or myself (WMF's Research team members) this week. In the meantime:
  • @Mayakp.wiki shared with the working group a RACI document with the Data Governance model, focused on accountability for ensuring that tasks/processes are completed.
  • @JAnstee_WMF and Omari presented the metric recommendation to the Foundation Level Metrics Committee and then to key Product Leads (e.g., @Samwalton9-WMF).

Weekly update

• Business Data Steward and Technical Data Steward have been designated
• We received word from @JAnstee_WMF that no more tasks are expected of Research for this quarter, aside from keeping up with messages in case the stewards have questions or need feedback
• We may need to do some data exploration in Q3, which would come as a separate request, and input to the retrospective write-up of the process

For now, we can consider our work on this task complete for Q2 and we will only add new weekly updates if new information arises.

Weekly update

• We reviewed/contributed to the draft metric logic and relationships slide, the EasyRetro board, and the summary of the working group process.

As @TAndic mentioned above, our work on this task for Q2 is complete.