Page MenuHomePhabricator

Conduct research on WLM centralnotice banners
Open, In Progress, MediumPublic

Description

In order to better understand the effectiveness of the centralnotice banners and the landing pages run for WLM, it would be great if we could receive some more statistics. In this request I'll lay out an initial request which may be too broad, happy to discuss how we can make it easier to manage. I'll just describe what would be our ideal :)

There are two underlying questions that are prevalent:

  1. For each country, in which 'phase' do we loose the most potential participants (funnel analysis)
  2. What diet would be appropriate for future campaigns.

Generally, the anticipated workflow of participants looks like this:

  • User sees banner
  • User goes to landing page
  • User goes through a number of steps, different per country
  • (non-registered contributors only) User creates account
  • User uploads image through WLM campaign

Some more details:

  • Central Notice: following the convention "wlm_2021_xx" with xx being the country code
  • Landing page: can be either on-wiki or off-wiki. An overview is here, but happy to create a more succinct overview later with only the relevant onwiki pages. Ideal would be to draw the landing page directly from the campaigns.
  • user upload would be uploadwizard with the campaign of the type "wlm-xx" (possibly with suffix). All are listed in the 'upload link' column here.

For the landing pages that are on-wiki, we would love a funnel analysis of some type, if possible. We can collect raw numbers through stats pages, but it would be great if we know whether they actually came through the banner or not. If that is possible by day that would be awesome, but I suspect for most countries this would get too hard on privacy grounds.

We would love to get some insight in how many people on the landing page come from different sources (broadly speaking), but it would already be helpful to know how many from banner, and how many 'other'.

Directly relevant for the understanding of the diets, the following questions would be relevant (per country, because different settings/cultures):

  • Given a unique landing page visitor, what is the first banner they click on? If their first clicked banner is 'm', what is the distribution of m? How many people clicked on the first banner, how many on the second, etc.
  • Given a unique uploader (eventual success!) what was the *last* banner they clicked on?
  • Given a banner click, how many people click on the banner a second time? Third time?
  • How does this differ from mobile to desktop?

What would be really cool to know, but secondary to this request: what are the language settings of visitors to our landing pages? (probably out of scope, but great if it happens to be in the data)

Event Timeline

AndyRussG added a subscriber: AndyRussG.

Hi! Interesting and useful questions, for sure... Adding Fundraising tag just in case we can provide some very general guidance, or maybe refer you to someone who could actually work on this.

@AndyRussG it would be great if you could help lay the connection with the right people :)

This request was the result from a thread between community organizers, that had some doubts about the effectiveness of a campaign with certain diets. Hopefully with this kind of digging, we get a feeling for what the impact of a diet would be on effectiveness of campaigns.

This request was the result from a thread between community organizers, that had some doubts about the effectiveness of a campaign with certain diets. Hopefully with this kind of digging, we get a feeling for what the impact of a diet would be on effectiveness of campaigns.

Yes, that makes a lot of sense. :) Just in case this is useful, I can mention that Fundraising has found that only the first few banners shown to a user have much likelihood of getting a response. However, reactions could certainly be different for a campaign like WLM.

Yeah, one of the main reasons why people think this might be different, is that people may have to take a photo before they can actually upload them. So they may see the banner, go take a picture, and then come back to the banner to upload.
Not sure if that actually happens in practice, but that may be one of the plausible reasons I heard why things may be different. There is some intuition that there is a diminishing return though - where the optimum lies, is not obvious to me :) .

adding @Sadads for visibility. I hear he is fielding community campaign questions these days.

@Effeietsanders this is not something we (Product Analytics) will be able to look into in the foreseeable future, but I'm adding @Iflorez and @ifried as subscribers so they can keep this need in mind as they work on Campaign-Tools

(Disclaimer: I have volunteered for Wiki Loves Monuments for multiple years and understand and empathize with the needs of the international team and local organizers first-hand.)

@Effeietsanders I discussed multiple options with the different teams within WMF who could potentially support your request and here is what I can offer:

  • Our team, Research, would be happy to set up an experiment to see how we can support WLM's needs in this case.
  • We do this as part of our commitment to improve equity in decision making.
  • We won't be able to answer the specific questions you asked as we expect you all to run into more questions as you learn more from the data and we feel it's important to enable you all to answer them directly. Of course, this won't be super easy, hence the Experiment.

Below are more details about the experiment:

  • We ask that you nominate (privately to me) 1 person from your community who will enter into a formal relationship with WMF. This means, they will have to sign a Memorandum of Understanding (valid for 6 months) and Non-disclosure agreement with WMF. (We will be closely following the process that we have in place for Formal Collaborations.)
  • We will give your approved nominated person access to the relevant data and @MGerlach, a research scientist in our team, will be their Point of Contact for the duration of the six months.
  • Martin will help the individual learn how to process very large data on our servers. He will also support them in developing their research skills as it relates to the request.
  • I'll send you some requirements for the individual you can nominate.
  • If we go with the above, I expect that in six months time we review and decide whether to extend the relationship.

For transparency,

  • At this point, we will only be able to pick up this experiment with one group, WLM.
  • At this moment, our team is not equipped to scale this kind of access and relationship should we find the experiment to be mutually desirable and successful. If we see this is the kind of service we want to offer to the communities at large, we need to find answers to many questions that will arise and will have impact on the work of many teams within WMF: Legal, Security, Research, Data Engineering and more.
  • The above being said, we have an alignment of stars at this point and I believe it's important to seize the moment, experiment, and learn to make more informed decisions.

Sooo, if the above sounds good to you, let me know here and let's pick up the nomination discussion over email.

Thanks @leila for this helpful response. We will discuss internally and follow up with you over email following that.

leila renamed this task from Statistics on WLM centralnotice banners to Conduct research on WLM centralnotice banners.Oct 6 2021, 9:36 PM
leila changed the task status from Open to In Progress.
leila assigned this task to Effeietsanders.
leila triaged this task as Medium priority.
leila added a project: Research.

Update:

  • I heard back from Effeietsanders and they confirmed that the WLM international team has nominated them as the 1 person who will work on the team's questions.
  • MGerlach will be Effeietsanders' point of contact for the duration of their access and research.
  • I have added the Research tag to this task so we don't lose track of it on our team's end.
  • MGerlach is coordinating the next steps with Effeietsanders over email. Sub-tasks may be created for this task to track that work or not. I will leave that decision to MGerlach and Effeietsanders.
  • To all other teams/projects: feel free to remove your tags if this task is no longer of interest to you. I didn't touch existing tags. :)

Thanks all!