Weekly update:

• Progress update on the hypothesis for the week: Research brief drafted detailing research questions, proposal for approach/methods, and tentative phases and team member roles.
• Any new metrics related to the hypothesis: No
• Any emerging blockers or risks: No
• Any unresolved dependencies: No
• Have there been any new lessons from the hypothesis? Not yet
• Have there been any changes to the hypothesis scope or timeline? No
• Next steps: Coordination meetings on 24 and 25 July to align on approach and define roles for the different tasks of the project

Weekly update:

• Progress update on the hypothesis for the week: Met with team to discuss and gather input on the research brief, including a discussion of general approach, roles/responsibilities, and next steps. A weekly coordination meeting and notes doc has been set up to help keep work on track. Ongoing revisions to the research brief, which will serve as a sort of living document throughout the project. Phab subtasks have been added to capture some of the work that will be lead by different individuals working on this project. Met with steering committee to provide updates and flag that we are currently missing a lead for the part I quant work; this is the reason the project is currently noted at risk, Multiple meetings are scheduled for next week to explore possibilities for filling this gap
• Any new metrics related to the hypothesis: No, but in the research brief, quite a few revisions were made to the guiding questions for the part I quant analysis. In general, the more core metrics were identified, with others noted as nice-to-haves or likely out of scope.
• Any emerging blockers or risks: We currently don't have someone to lead the metrics definitions needed for the quant analysis. We do have someone who is able to pull and analyze data once they've been defined, but we currently have a gap that needs to be filled (see note above regarding this).
• Any unresolved dependencies - do you depend on another team that hasn’t already given you what you need? Are you on the hook to give another team something you aren’t able to give right now? See note above re:blocker/risk. This is the reason the project is status at risk.
• Have there been any new lessons from the hypothesis? n/a
• Have there been any changes to the hypothesis scope or timeline? No updates to the overall timeline, but the team is working on providing specific dates for important milestones. We aim to provide these tentative dates by the end of next week.
• Next steps: Notable mentions include priority topics for the team in the coming week: sampling approach and decisions, progress on a brief annotated bibliography, selection and definition of key terms, and work to identify someone who can lead the quant work.

Weekly updates:

• Finalized some project planning documents, including phases and timelines.
• Updated the project meta page with more content and ways for community to start getting involved at talk page
• Worked on the definition of key terms that we need to guide the work.
• Began work on an annotated bibliography to ground and guide the work.
• We met with various people this week to try to find a solution, but we currently still don't have someone to lead the metrics definitions needed for the quant analysis.
• Our main goals for the coming week are to make progress on the annotated bibliography and discuss and generate a sampling approach during the 8 Aug coordination meeting.

Weekly updates:

• Progress update on the hypothesis for the week:
  • Settling on a sampling approach :We have decided to focus on higher-activity wikis, which will go on to inform survey distribution plans, which in turn affects our interview plans. This is because prior research on moderators has consistently noted that activity level strongly correlates with the number of admins and perceived moderation task burden. Therefore, lower-activity wikis have substantially different administrator working conditions, which affects administrator push/pull factors. Additionally, the impetus behind this project came from concerns expressed by administrators of very high activity wikis, and therefore we believe it is most feasible to focus on these larger and more active wikis.
  • We are defining "high activity" as those with more than 1000 weekly non-bot edits, and more than 10 monthly active admins. This threshold is based on T348677#9353898 and our prior work on Content Moderation on Medium Sized Wikis, which used a similar cutoff to define small versus medium-sized wikis.
  • We also elaborated on the research goals behind the survey and interview, to allow these two components of the project to best compliment each other
  • We've scheduled a working meeting to work on a draft survey distribution plan
  • Continuing work on the annotated bibliography (internal use), and have also expanded our public Meta-Wiki page for this project
• Any new metrics related to the hypothesis: None
• Any emerging blockers or risks:
  • Unresolved lack of quantitative lead for metric definition, but no new blockers or risks
• Any unresolved dependencies?
  • See above, carried over from last week
• Have there been any new lessons from the hypothesis? None
• Have there been any changes to the hypothesis scope or timeline? None
• Next steps:
  • Draft the survey distribution approach
  • Define key distribution populations, particularly "potential administrators"
  • Finish annotated bibliography work, currently set for 15 Aug due date

Weekly updates:

• Diego (from Research) has joined the group to serve as lead for the quantitative portion of work (This is significant as it means the project is no longer currently at risk). Eli and Caroline met with him this week to catch him up, and Diego and Caroline will be working closely together in the coming weeks.
• Annotated bibliography is 95%+ complete. By Aug 21 we will add a few final entries and then do final minor cleanup of the document.
• We are quickly narrowing down the survey distribution criteria, including which language versions of Wikipedia (wikis) and which populations within those wikis will be included. As part of this work, we have researched topics such as administrator candidacy requirements across wikis given that there's significant variation. We also explored ways of defining categories such as 'potential administrator'. We plan to add summaries of some of these learnings to the public meta page.
• We've drafted some additional content for the public meta page pertaining to data use and retention. We plan to continue expanding the public presence on the meta project page in the coming week. We've already begun receiving some community input via prompts at the talk page.
• Main goals for coming week: finalized annotated bibliography, finalize inclusion criteria for language versions, begin drafting survey topics and questions, make progress on the quantitative component of work, and anticipate and coordinate a number of logistics around privacy policies, consent forms, and translation

@Easikingarmager thank you very much for this update and the work you and the team have been doing so far. The coming together of many expertise and the gradual community involvement are important for this work and I'm really thrilled to see that's happening.

two requests:

• I'd like to review the criteria for language inclusion before you all start executing on it. I can have a quick turnaround for this.
• Consider a "public first" approach for documentation and make the documents outside of meta-wiki when you have good reasons for. For example, I understand that for documents that are in flux and you need a lot of input and back-and-forth on, google docs are more efficient; or some documents will need to remain private. some other pieces of work may not need to be. (e.g., bibliography) Adopting this approach early on can save you all spending significant spike on documentation at the end of the project, and it also helps volunteers to benefit from them in their work. (I know there are tradeoffs here. I trust your judgement for making the right call on the tradeoff.)

Weekly updates:

• Annotated bibliography (part of pre-work) is complete: Commons file and linked to from meta page resources section.
• Notable progress on exploratory metrics, including determining data tables and variables needed (related, a L3SC request was submitted).
• Exploration of structured data that exists related to RfAs (i.e. successful vs. ‘failed’)
• Completed estimates for budget related to translation, interpretation, and transcripts.
• Group narrowed down the wiki (Wikipedia language version) selection criteria during weekly coordination meeting.

Upcoming goals:

• Prepare clear presentation of wiki criteria and selection from discussion notes, along with participant/respondent segment descriptions, and send to Leila to review.
• Begin drafting survey and interview question topics.
• Prepare plan and materials for pilot survey takers and plan various upcoming data collection logistics.
• Continue with quantitative analysis, and review/update questions we can answer from initial list.

Weekly updates:

• Wiki sampling selection criteria for surveys and interview was prepared and sent to Leila for review, who in turn provided feedback. This feedback was incorporated as we narrowed to a final sample selection (available soon on project meta page).
• Survey respondent and interview participant segments were reviewed and polished in anticipation of recruitment screening and materials.
• With quantitative work underway, the initial list of guiding questions was reviewed for feasibility and scope. Some initial results of investigations are presented below.
• Drafting of survey and interview discussion guide were started and will continue as priority in coming week's work.
• In the weekly coordination meeting, we reviewed a list of upcoming logistics for the survey and interviews that was prepared in advance.
• As a result of exploratory work for wiki selection, we have a dataset of every RFA made across 11 wikis.
• We have a few highlights of preliminary analysis of data for sysop activity over time, number of sysops over time, and bots with sysop rights across our initial shortlist of 21 Wikipedias to share. There is notable variation in patterns we observe across wikis, with some showing declines and others with increases for sysop activity, number of sysops, and amount of bot activity. To highlight 3 examples:
  • Examining the number of active sysops monthly, since 2018, across 21 wikis shows that many (e.g., English, Russian, French, and Portuguese) show a decline in active sysops, and others (e.g., Indonesian) show an increase.
  • Preliminary analysis shows that for the number of sysops over time, many (e.g., English, Russian, Portuguese) show a decline in sysops, while others (e.g., German, Italian, Indonesian) show an increase in sysops.
  • While all of the wikis we've examined have bots with admin rights, some wikis (e.g., Arabic, English, Spanish) have bots with extremely high monthly activity compared to others.

Upcoming goals:

• Continue drafting survey and interview discussion guide
• Continue metrics investigations

Weekly updates:

• Survey draft and interview discussion guide draft are both nearing completion. We will complete these drafts by Sept 13 and invite a few stakeholders to provide feedback. We've reached out in advance to let them know about this opportunity.
• We did a status check this week, and reviewed the remaining tasks for phase 1 of the project. Given many interdependencies and sequencing, we created a plan for these remaining tasks along with goal dates.
• Completed quantitative exploration of “active admins” vs. “all admins”, and now working on way to systematically define 'increasing', 'decreasing', and 'stable' for admins over time.
• Completed quantitative exploration of the Special:ListGroupRights data, per wiki, via scraping/parsing and plotting.
• Wiki sampling selection criteria for surveys and interview are available on the project Meta page.
• We continue to learn about cross-wiki variation in terms of user groups with administrative rights (e.g., reversores on ptwiki; eliminators on fawiki, jawiki, and ptwiki). One ongoing finding is that rights are divided up in different ways across different wikis, which also presents some challenges for analysis.

Upcoming goals:

• Complete survey and interview discussion guide drafts and send for review by selected stakeholders no later than Sept 13.
• Given cross-wiki variation we've observed, propose operating definition for "administrator", "administrator action", "inactive admin", etc. using a broader definition of "administrator" centered on an activity-based definition, "users who take admin actions". Discuss on Sept 12.
• Approach potential WMF staff who might serve as translation reviewers and ask them to bookmark some time around when we'll have translations of materials complete.

Weekly updates:

• Headway on multiple fronts this week. Shout out to the group for all the work that's gone into the highlights offered below. 🎉
• Completed survey drafts (current admins/prospective admins), as well as interview discussion guide (multiple participant segments included), and sent these to Leila Zia and Sam Walton for review.
• Finalizing definitions for admins, admin activity, active admins, inactive admins, former admins, and potential admins. This will be added to the Meta project page in the upcoming week. This is significant because it not only allows quantitative inquiry to move forward, but unblocks the group on sampling for the qualitative components of the project.
• Determined user groups to include in the “admin” definition (See Plot 1 here for user groups per wiki)
• Determined method for classifying admin activity over time as: strongly decreasing, decreasing, increasing, or strongly increasing. We ran linear regressions and allocated slopes into quartiles. (See Plot 3A, and Plot 3B for regression lines.)
• Various meetings to discuss additional considerations around how to reach and recruit survey respondents (current plans are through MassMessage or EmailUser, and we've arranged with Movement Communications to use MassMessage).
• Obtained participant consent forms in all target languages for the interviews (Russian in progress), and loaded them into Docusign. Also related to translations, we've received the commitments of Russian, French, Spanish, and Indonesian-speaking staff to review the survey translations once complete.
• Consulted with Leila and arrived at decision around research participant incentives for this project, and identified an opportunity to align the Research Team around participant incentive practices; ticket capturing this was added to the Research Team Phab backlog (T374581)

Upcoming goals:

• Revise survey and interview discussion guide drafts based on incoming feedback anticipated next week.
• Pilot English draft of survey ahead of translation.
• Update project Meta Page with recent updates and work.

Weekly updates:

• Requested survey privacy statement and submitted LS3C request
• Received feedback from Sam Walton and Leila Zia on research instrument drafts; may need some follow-up feedback from Leila next week around trimming down survey further.
• Based on group deliberations and consultations with other WMF staff, we arrived at a few decisions for survey distribution, including the decision to use anonymous links.
• Created outreach and follow-up messaging, and determined incentive plan for interviews.
• Prepared draft presentation of key definitions for inclusion on project meta page.
• Multiple consultations with folks from Movement Comms pertaining to the project generally, and specific topics such as survey distribution and working with admins
• Began compiling updates needed for project meta page.

Upcoming goals:

• Meetings with Ramzy and Nathan to consult on working with admins
• Revise survey and interview discussion guide drafts based on feedback received this week.
• Pilot English draft of survey ahead of translation.
• Update project Meta Page with recent updates and work.
• Begin recruitment of ex-admins for interviews

Weekly updates:

• Consulted with Trust & Safety Disinformation about working with admins and discussed implications of this information as team
• Made plans, in coordination with Movement Comms, for some recruitment outreach at WikiCon North America happening next week.
• Wording mostly complete for interview outreach messaging.
• Incorporating feedback on survey draft
• Waiting on needed responses from Privacy@ and Legal
• Having reviewed tasks in progress and TBD for survey preparation, we're anticipating a 'go-live' date of Oct 15 (previously Oct 9). No changes to scope.

Upcoming goals:

• Consult with Security and Human Rights team to ensure we're taking sufficient precautions to protect research participants.
• Generate pre-publication security review plan to ensure we're protecting project participants.
• Set up LimeSurvey and pilot English version of survey.
• Continue interview recruitment.

Weekly updates:

• Consulted with Movement Comms on recruitment plans for target wikis
• Discussing with Human Rights team on ruwiki recruitment plans
• Piloted English version of the surveys, incorporating feedback
• Translations obtained for privacy releases and outreach messages, as well as scheduling and privacy release portals
  • Waiting on translations for survey
• Waiting on needed responses from Privacy@ and Legal
• Querying lists of potential and current admins for survey recruitment
  • Current admin query is finished
  • Potential admin query is ongoing
• Recruitment for interviews (ex-admin segment) has begun
• Survey target start date remains the same, Oct 15

Upcoming goals:

• Consult with Security team to ensure we're taking sufficient precautions to protect research participants.
• Generate pre-publication security review plan to ensure we're protecting project participants.
• Continue interview recruitment and scheduling
• Deploy the survey

Weekly updates:

• Admin interviews are underway! With more scheduled and others being recruited for and scheduled. Following up with MoveComms for eswiki recruitment support.
• Updated definitions on Meta page.
• Revised and programmed surveys on Limesurvey based on feedback from survey pilot testers. 
• Queried, formatted, and shared lists of potential survey respondents for the purpose of outreach.
• Met with the human rights team, and heard back via email/slack from the security team, regarding safety precautions for research participants. Discussed as group the feedback from security, human rights, and trust and safety teams.
• Met with movement comms team to discuss sharing about the project at upcoming Norwegian and Polish community events; discussed options for folks at these events to share experiences around the topic with the team on the project talk page.
• Requested translations for surveys, and coordinating review of translations by WMF staff members
  • Initial translations expected complete on 14Oct
  • Expect to receive WMF translation review back by 16Oct (east coast am)
  • With need to finalize programming and do some testing for bugs, anticipating a launch of 21Oct (prefer to avoid launching right before the weekend)

Upcoming goals:

• Program translations, test, and launch survey
• Continue recruiting, scheduling, and running interviews

Weekly updates:

• Received and reviewed survey translations; wrapping up programming surveys; target go-live date remains Oct 21
• Interviews remain underway, with more scheduled for next week; transitioning to recruiting for the current admin segment
• Finished notebook for determining dates of admin milestones (linked notebook is enwiki only; there is an additional notebook that runs for all 21 focus wikis for the project). These milestones include events such as date of first admin action, date of becoming sysop or other admin group, etc.
• Finished notebook compiling 'admin inflow and outflow' for all 21 shortlisted wikis.
• Developed some additional project talk page prompts that can be used to gather input from groups that have worked in the area of 'admin support activities'.
• In the weekly coordination meeting, we revisited the firm delivery date of Dec 20 and worked backwards to find how we can meet this goal. To reduce the risk, we identified adjustments to the survey run time, requested some additional support for the interviews, and are clarifying expectations for reporting.
• There have been some additional learnings around the technical details of user groups and querying for users who have performed administrative actions. For anyone interested, more details can be found in updates at the sub-task level (https://phabricator.wikimedia.org/T371139)

Upcoming goals:

• Finish programming surveys and go live with them
• Continue conducting interviews

Weekly updates: Overall, we're in a very active stage of data collection as the summary below shows.

• 8 of 12 surveys are currently fielding responses. Pending final review of translations, 4 additional surveys will launch by mid-next week, resulting in a total of 12 surveys across 6 languages.
  • As part of this, the Polish Wikipedia surveys are having translations finalized and will launch next week. The project meta page has been updated to reflect the additional scope expansion to Polish Wikipedia (and Natalia from Movement Comms informed).
• We've received 675 survey responses and 210 partial responses as of Oct 24. We're observing a close relationship between the time of survey communication (i.e., invites) and responses coming in (and thus confidence we can still launch plwiki surveys next week without expanding timeline).
  • Currently lower response rates for eswiki potential admins; may need to expand outreach (decision to be made next week).
  • Compiled list of interview prospects stemming from subset of survey respondents.
• Interviews remain underway and in-progress. We're coordinating logistics on the team to figure out how to handle interviews while the interview lead on the team is away.
• Clarified and discussed expectations around reporting plans, and anticipated some details for how we'll coordinate/collaborate as the analysis and reporting process begins.
• Continue to review and respond as needed to talk page discussions happening: (as well as other inquiries/questions from individuals who've been invited to the survey) https://meta.wikimedia.org/wiki/Research_talk:Wikipedia_Administrator_Recruitment,_Retention,_and_Attrition
• Created notebook for analysis of admin tenure (see Plot 1) for comparison across 21 wikis in shortlist)
• Various technical learnings while investigating bot vs human actions and edits (see subtask T371139)

Upcoming goals:

• Continue recruiting, scheduling, and running interviews.
• Finalize and launch remaining 4 surveys; send out invites to respondents.
• Continue to monitor survey responses, and coordinate around survey respondents expressing interest in interviews.

Weekly updates:

• We continue to recruit, schedule, and run interviews.
• As of 31 Oct, we've received 966 complete (and 350 partial) survey responses. In addition to the 8 surveys across 4 languages currently collecting responses, 4 additional surveys in Russian and Polish (2 each language) have been programmed, reviewed, and activated. Recruitment messages will be sent for the Russian and Polish surveys starting on 4 Nov.
• We continue work to explore the relationships between bot and human administrative actions.
• We've updated queries of admin milestones to include desysop'ings via stewards, and continue to debug.
• Began outlining and preparing for analysis and reporting.

Upcoming goals:

• Continue recruiting, scheduling, and running interviews
• Messaging outreach for remaining 4 surveys; additional outreach for eswiki surveys
• Continue to monitor survey responses, and coordinate for purposes of interviews

Weekly updates:

• Onboarded Daisy to help with interviews, who has been helping to recruit participants over the last few days. Continue to run interviews.
• Currently sitting at just over 1k complete survey responses as of Nov 4, with ~350 partial completes.
• Handled a number of responses to community messages and inquiries about the project this week.
• Survey outreach lists compiled for plwiki and ruwiki (as well as second outreach group for eswiki); survey reminder message lists also compiled. Today and tomorrow will be sending lists to Uzoma, who is helping with outreach messaging. This should be the final step in unblocking us on survey response collection for ruwiki and plwiki.
• Continued work on reporting template.

Upcoming goals:

• Continue recruiting, scheduling, and running interviews.
• Send survey invite messaging for ruwiki and plwiki surveys, as well as second group of eswiki potential respondents. Also, send survey invite reminder messages.

Weekly updates:

• Reviewed interview and survey progress.
• Although we don't really have proper/appropriate benchmarks against which to compare, survey response rates have been relatively high (i.e., roughly 50-60%+).
• Survey response collection will close 22 Nov, and interviews will close by 27 Nov. A meta page update was added to reflect these plans.

Upcoming goals:

• Analysis and reporting

Weekly updates:

• Cleaned survey data and began analysis
• Interview data analysis
• Coordination meetings related to analysis and reporting
• Revisions of bots/human/abusefilter analysis
• Current sysop trends, including question of prevalence of multiple project sysops

Upcoming goals:

• Continue with analysis and reporting

Weekly updates:

• Completed survey analyses and visualizations
• Continued work on synthesis and reporting of quant metrics
  • Normalized regression slopes in active_admins_over_time.ipynb
  • Added project sizes to contextualize current admin Ns for the 21 wikis in current_admins.ipynb
• Significant progress made on final report
• Sending in-progress final report to Leila Zia today for review and input.

Upcoming goals:

• Continue drafting and editing final report
• Significantly shorten 'brief' portion of report (currently brief is comprised of sections 1 and 2; section 3 is appendix/full reporting)
• Workshop recommendations section
• Finish any remaining data analysis

Weekly updates: Overall, we made significant progress on analysis and reporting
Next steps: Project wrap-up tasks, including finalizing and sharing report by uploading it to Commons and Meta page

(moved to in progress; some tail-end tasks to be wrapped up)

Weekly update: We gathered feedback and input from collaborating WMF staff this week and are working on very final edits. Current expectation is to publish to Commons and share on meta project page approximately 22 January.

Weekly update:

• On 22 January, we published the final report for this project on Commons. Prior to report publication, we consulted with our Human Rights team as well as the Movement Communications team, to ensure our report would not contain potentially identifying or other sensitive information.
• For a quick read, we suggest reading the Section 1 ‘Key Findings’ headlines and short explanatory text, and then proceeding to Section 2 ‘Recommendations’.
• For a deeper understanding, the reader will find that the Key Findings section text in the PDF version of the report contains links to corresponding sections in Section 3 ‘Full Reporting’. This is where significant expansions can be found, including additional analyses, figures, and tables.
• We also included the key findings and recommendations sections (§ 1 and 2, respectively) of the final report as a project report subpage. Again, the key findings section contains section number references to locations in section 3 of the full report, where additional expansions of reporting are available.