Page MenuHomePhabricator

Testing of messaging for downloading the Wikipedia app
Closed, ResolvedPublic

Description

This is a research request from Jazmin, as described and discussed with Debra. Documenting here to facilitate confirming with Mike whether there's a quick turnaround, low-effort way for us to do this by the end of this month, so we can confirm to Jazmin this week.

Context:

  • Marketing team will be doing a big marketing push in May towards getting people to download the Wikipedia app, with campaigns happening both on platform and off
  • For on-platform messaging they plan to go with messaging around "did you know there is an app". Based on a past test of on-platform messaging with people in Japan, they found that what resonated most with users who didn't have the app, was just awareness that there's an app -- versus talking about features
  • For messaging to use on off-platform marketing channels (press releases to news sites, socials including TikTok and Instagram, possibly paid media), they don't know what messaging to go with. Other than social media posts and seeing which get the most views, they haven't done any sort of testing. There was a spike in downloads and account creation that is suspected to be related to how it went viral on TikTok, so that did seem to resonate.
  • There's been survey work of current Wikipedia app users before about what they most like about the app, and they've said tabs. That is not seen as enough of a hook for use for a marketing campaign.
  • Comms has come up with 3 contenders for the key message of the off-platform campaign - seen here on this Miro board under "Brainstorm 2": https://miro.com/app/board/uXjVJyINgaI=/?share_link_id=412613058887

Research question:

  • Which is the best "hook" to use in messaging, for the messaging that happens off-platform? (that is appealing to the most people, so as to drive the most downloads)

Estimated Effort.

  • Should be feasible to answer via a short survey via Qualtrics, using Prolific panel.
  • Still need to confirm exact survey population desired: infrequent/casual, which market.

Priority
Jazmin wants insights by the end of February, so that Marketing can prep the messaging campaign before the planned launch date.

Open question Debra sent to Jazmin on Feb 9 via Slack: "In this doc (https://docs.google.com/document/d/1ulXkHHp5WvrRS61y0IoaS--YIiBlW1-rrFcD69VXKOc/edit?tab=t.3cmw03b7q6tg) , there is mention that "In early January, we’ll see first results back from the Wave 1 CNB tests which include copy for each of these 3 messaging areas which may also help us make a final selection". Did that test happen? It is describing the same sort of testing that I had described to you as what I recommended: a quick survey of the text of the three options."

Event Timeline

MRaishWMF claimed this task.

We have produced a first draft of the research report. The method used to arrive here entailed:

  • developing an Android survey around 7 GIFs+static images, and an iOS survey around 5 GIFs+static images
  • watching ~9 pilot testers take these surveys, adjusting wording as needed (and also adjusting wording after feedback from the stakeholders and Design Research colleagues)
  • administering the survey to ~130 Android testers (they saw only 5/7 features each) and ~90 iOS testers (who saw all 5 of their features)
  • analyzing the numerical ratings for individual features in conjunction with participants' open-text responses

Closing this task at this point because we've completed data collection and the bulk of analysis. Coming weeks may see requests for clarification or additional analysis, but at this point it's probably safe to consider this project concluded.

Methodological lesson:

in the future, plan ahead of time to aggressively clean data using open-text responses to identify likely AI or extremely low-effort respondents. Structure the study so that these responses can be removed without compromising the dataset (i.e., it's less problematic to remove these people if you're not aiming for a representative sample, or to make direct comparisons between two samples). This affects the way we phrase our research questions and structure our analysis.

Thank you Mike for your work on this, and for also capturing these methodological learnings!