Page MenuHomePhabricator

Lead survey portion of project inquiry
Closed, ResolvedPublic

Description

With support from the group, lead the survey portion of the study. This includes contributions to sampling approach used for the project, development and refinement of the survey materials, some support and overseeing of translation process(es), contributing to the development of a recruitment approach, and analysis and reporting of findings.

Preliminary Work (8 August - 27 September 23 October)

  • Identify Wikipedias to be surveyed: at moment looks like binding constraint may be translation budget
  • Develop sampling plan for:
    • Current admin survey
      • Sample size: census (even enwiki sysop user group population (n=911) is so small that sampling is impractical given plausible survey response rates)
      • Survey administration: mass message to talk pages with link to Limesurvey
      • Sampling method: N/A (census)
    • Potential admin survey
      • Sample size: TBD based on population sizes of "potential admins" per Wikipedia -- see calculations here.
      • Sampling method: SRS with qualification based on following:
        • Account age
        • Global edits
        • Edits in past X period (X TBD)
        • Availability of email (not feasible to sample on this, so can use a screener)
        • User group membership
      • Survey administration: massmessage
  • Draft survey instrument
  • Facilitate survey translations

Data Collection (Oct 23 - TBD)

  • Field current and potential admin surveys (mass message)
    • enwiki (Oct 23)
    • eswiki (Oct 23)
    • frwiki (Oct 23)
    • idwiki (Oct 23)
    • ruwiki
    • plwiki

Analysis (1 November - 6 December)

  • Clean survey data
  • Coordinate on analysis goals with rest of team
  • Prepare report on survey data

Reporting and socialization (6 -20 December)

  • finalize report
  • presentation/discussion with stakeholders (Q3)

Event Timeline

Easikingarmager renamed this task from Lead interview portion of project inquiry to Lead survey portion of project inquiry.Jul 26 2024, 6:55 PM
Easikingarmager updated the task description. (Show Details)

Weekly update:

  • Began work on survey timeline and preliminary sampling frame in advance of standing meeting to take Thursday August 1

Weekly update:

  • Tentative sampling plan parameters have been established and are summarized here
  • While I am OoO (until September 3), the following survey work will continue so that my absence doesn't block progress:
    1. Wikipedias to be surveyed are selected (based on translation/interpretation costs)
    2. Research questions to be gathered (I will transform these into survey questions on my return)
    3. Eli to connect with Leila and Sam Walton to determine whether and how the planned admin survey research can serve the needs of PES 1.2 (see T370439). Options could include (but are not limited to):
      • expanding the planned SDS 1.2.2 survey research to include satisfaction questions for moderators who say they use current tools
      • modifying the planned SDS 1.2.2 survey instrument to provide some baseline satisfaction measure without explicitly including an additional survey module
Easikingarmager changed the task status from Open to In Progress.Aug 21 2024, 8:30 PM
Easikingarmager triaged this task as High priority.
Easikingarmager updated the task description. (Show Details)

I added our targeted phase dates to sync these activities with the timeline at: https://meta.wikimedia.org/wiki/Research:Wikipedia_Administrator_Recruitment,_Retention,_and_Attrition#Timeline. I also added the final reporting/socialization phase and dates to capture work at that time.

Weekly update:

  • Wikipedias to be surveyed have been tentatively identified as English, Spanish, Russian, French, and Indonesian
  • Sampling plan currently depends on work being done as part of T371139 which will operationalize a definition of "admin" to be used in defining the population to be sampled
  • Survey script (English) is currently being drafted and will be shared with SDS 1.2.2 working group for feedback by September 13
  • Aligned with Moderator tools team and confirmed that work here and in T370439 will proceed in parallel (as separate survey projects).

Weekly update:

  • Met with Bethany and Tanja (separately) to better understand survey logistics. Tentative decision is to adopt Community Insights-style survey contact (through mass message or emailuser) for both the current and potential admin populations and to adopt CI-style sampling for the potential admin survey.
  • Received updates on work on defining "admin" (see T371139). This definition will be adopted for the current admin survey.
  • Shared draft versions of the current and potential admin survey scripts (English-only) with the SDS 1.2.2 working group. These will be shared with Leila and Sam Walton for feedback next week.

Weekly update:

  • LS3C submission made for risk assessment for admin survey on ruwiki (see Asana task here)
  • Decisions were made during the weekly working group meeting to:
    1. Distribute anonymous links rather than unique links to survey participants
    2. Request usernames, while disclosing in the survey item text that we would seek to link account attributes to survey responses for analysis purposes
  • Still awaiting Leila's feedback on survey scripts, but should be ready to update next week

Weekly update:

  • Still waiting update on LS3C submission
  • Survey scripts have been revised according to feedback from Leila and Sam and will be programmed as Limesurvey projects for piloting by EOD Wednesday October 3
  • Tentative decision to proceed with ruwiki preparations with arwiki survey to be prepared as a "backup"

Weekly update:

  • Survey scripts have been programmed and shared with volunteer pilot testers
  • Privacy statement has been received and translated

Weekly update:

  • Survey scripts and programmed surveys have been further revised based on feedback from pilot testers
  • Awaiting translations and further feedback on potential alternative survey targeted to ruwiki (see broader update on T368791)

Weekly update:

  • translations received for survey scripts
  • translations reviewed by staff for frwiki, eswiki, idwiki--still awaiting reviews for plwiki and ruwiki
  • frwiki and eswiki surveys programmed into Limesurvey projects and shared for review with working group; idwiki survey programming underway.
  • surveys to field (if ready) by Monday Oct 21, otherwise by EOW (at latest).
YLiou_WMF updated the task description. (Show Details)

Weekly update:

  • Surveys for enwiki, eswiki, frwiki, idwiki fielded on October 23
  • Still awaiting programmed survey testing for ruwiki surveys
    • Awaiting re-translation of plwiki potential admins survey to resolve translation discrepancies with plwiki current admins survey
    • See this tracking document for response rates and partials
      • noting that we're seeing low response rates for eswiki potential admins (12%) which may require us to expand the contact universe--will monitor for the next week and decide at the team meeting whether to do so
    • Respondents volunteering for in-depth interviews shared with team

Weekly update:

  • Translations and programmed surveys for ruwiki and plwiki are ready to field
  • Response rates and partials for enwiki, eswiki, frwiki, idwiki continue to be tracked here
  • Reminders for all current contact lists as well as newly sampled contact list respondents for potential eswiki admins survey to be sent out November 11

Weekly update:

  • Survey data collection to close at EOD November 22.
  • Decision made at working group meeting November 21 based on high overall response rates and need to begin survey data cleaning and analysis.

Update:

  • survey data is cleaned and analysis is underway
  • met with Caroline to coordinate on analysis tasks
  • preliminary analysis should be ready to share by December 12

Update:

  • survey analysis and visualizations are being collected here and are being added to the draft report
This comment was removed by YLiou_WMF.