Page MenuHomePhabricator

[REQUEST] Analyze Impact of Event Invitations on Event Registration
Closed, ResolvedPublicMar 14 2024

Description

What team/program is this request for?

Campaigns, product

What are you requesting?

I would like to analyze the potential impact of Event Invitations as a way for organizers to reach new audiences to invite to their events, with the hope that some of the invited editors register for their events. This ticket specifically focuses on the impact of Event Invitations on event registration. To do this, the Campaigns team will provide the following data:

  • Usernames of who was included in the Event Invitation list for an event and their scores
  • Usernames of who the organizer chose to invite to the event
  • Link to information/direct information on who/who did not actually registered for the event

With this information, we would like to collect data on:

  • The total number of people who were invited
    • Per event
    • All events
  • The percentage of those who were invited and registered per event
    • Per event
    • All events

Note that, by "event registration," I do not specifically mean the Event Registration tool created by the Campaigns product team. Rather, I am referring to the general event registration number of the event, regardless of which event registration tool is used.

What is the problem you're trying to solve?

Summary: Right now, it is very hard for organizers to promote their events to new audiences who may be interested in the topic. This means that some events may have less impact than the organizers would hope for, since they are reaching out to the same audiences again and again. We want to make it easier for organizers to reach new audiences who may be interested in their events, so more people can join their events and make campaign contributions.

More background: Some organizers can use the CentralNotice banner to do a very wide promotional campaign for their event, which does not target by topical interests of editors, or they can reach out to their existing networks for promotion. However, if they have an event that focuses on topics X and Y, and they want to find editors who are really engaged in editing on those topics on certain wikis, they have no easy way of doing this. Meanwhile, our research on Event Discovery found that experienced editors are most likely to join an event due to the topic of the event.

As part of our Event Discovery work for WE 1.3, we are experimenting with a prospective feature called Event Invitations, which allows organizers to identify editors who may be interested in their event due to their edit history. We do this by requesting worklists from organizers (which need to include existing Wikipedia articles). We then generate an Invitation List that has a scoring system (see T353459), which finds all editors who contributed to the article in the past three years and assigns a score to the editors based on their level of contributions to the article and their recent editing history. Once we have the Invitation List, we give it to the organizers, who can choose to invite the editors on the list via wikimail or talk page messages.

Note that we originally planned to only give the organizers a list of editors who have high scores. However, we plan to now indicate which editors have higher scores, but we will also give the full list of all the editors, including those with lower scores, in case the organizer wants to invite a wider group of people, since some of the editors with lower scores may still have interest in the topic or the event.

We want to analyze the impact of Event Invitations by looking at the results from a certain minimum number of events (perhaps 10 events, which we should discuss as a team). This way, we can minimize risk if, for example, one or two events experience issues (such as being on less popular topics, being at a time that is inconvenient for many people, etc).

What decision will you make or action will you take with the deliverable?

  • Decision: Do we want to invest further team resources (product, design, and technical work) in the Event Invitations feature?
    • Details: Right now, we don't know how effective Event Invitations are as a tool to encourage more editors to join an event (which is being analyzed in this ticket) and make campaign contributions (which is being analyzed in another ticket). For this reason, we want to know if we're seeing any notable impact on the registration (and to what degree?). With this data, we can make a more informed decision about how impactful such a feature may be for event organizers and if we want to invest more development into the feature or to pivot to a different potential feature. It is important to note that we have not built a front-end for Event Invitations and we have invested limited team resources into the project so far.
  • Decision: Do we want to update or improve the current the model for generating the Event Invitations list -- and, if yes, what improvements do we want to make?
    • Details: We do not yet know who is or is not showing up to the events from the Invitation List. Once we have a better sense of its overall effectiveness and audiences it is or is not reaching, we may want to improve the logic behind the current model so that it can be more precise in who it identifies as a good candidate to invite to a given event.

Details

Due Date
Mar 14 2024, 3:30 AM

Event Timeline

Hi @ifried if the below looks good, let's identify those workbooks/events to include.

A) Inputs provided to Irene for each event...received in a Gworkbook doc (example):

  • list of usernames included in the Event Invitation list - Column A in each "impact analysis" sheet
  • list of usernames invited to the event by the organizer(s) - Column D in each "impact analysis" sheet
  • list usernames that registered for the event - Column F in each "impact analysis" sheet
  • date the invitation list is provided to the organizer

As well as a list of campaigns to include. For now I believe that we have:

B) To output data on:

  • The total number of people who were invited: Per event - Column K in each "impact analysis" sheet as well as on this ticket at the culmination.
  • Invitation list registration rate: Per event - Column L in each "impact analysis" sheet as well as on this ticket at the culmination

And provide an aggregate report on this ticket on:

  • The total number of people who were invited
  • Invitation list registration rate for all participating events
Iflorez triaged this task as High priority.Feb 17 2024, 1:30 AM
Iflorez edited projects, added Product-Analytics (Kanban); removed Product-Analytics.
Iflorez moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

This makes sense to me & sounds good, @Iflorez! Thank you for putting together this proposal and explanation.

ifried set Due Date to Mar 14 2024, 3:30 AM.Mar 6 2024, 6:00 PM

Interim checkup numbers - compilation sheet - for the campaigns and metrics noted above

The total number of people who were invited: 51
Invitation list registration rate for all participating events (we haven't discussed this in detail, so I'm offering two options):

  • registration rate - on the full Invitation list : 1.5% (includes all of those that we identified in the full list)
  • registration rate - for folks that were invited from the invitation list: 3.9% (includes all of those that we identified AND that organizers invited on the full list)

Added events from the worksheet Index of Events that used Event Invitations to the data compilation sheet

  • The total number of people who were invited: 214

Invitation list registration rates for all participating events (we haven't discussed this in detail, so I'm offering two options):

  • registration rate - on the full Invitation list : 2.4% (includes all of those that we identified in the full list)
  • registration rate - for folks that were invited from the invitation list: 24.77% (includes all of those that we identified AND that organizers invited from the full invitation list that we provided)

EQUATIONS

H Match function

=IF(ISNUMBER(MATCH(CLEAN(TRIM(G4)), ARRAYFORMULA(CLEAN(TRIM(A:A))), 0)), "Found", "Not Found")

What it's doing:

  • cleans and trims the names in column A "invite list names" and column G "attendee list names"
  • returns "found" when the value in cell G# is found anywhere in column A

N number of people who were invited and registered

=SUM(ARRAYFORMULA(IFERROR(IF(VLOOKUP(FILTER(G:G, H:H="Found"), A:D, 4, FALSE)=1, 1, 0), 0)))

What it's doing:

  • Sum the occurrences where the value in column H is "Found", the corresponding value in column G matches a value in column A, and then the corresponding value in column D (where the match in column A occurs) is 1
  • check only the rows where column G has a value, and ensure error handling for any potential errors that might arise from the missing values

Steps:

  1. Filter Column G Values Where Column H is "Found": get all values from column G where the corresponding value in column H is "Found".
  2. Match These G Values in Column A: For each value obtained from step 1, find if it matches any value in column A.
  3. Check Corresponding D Value: For each match found in step 2, verify if the corresponding value in column D is 1.
  4. Sum the Results: Sum up all instances where the above conditions are met.

.
.
.
an updated function for N to clean/trim participant-name columns:

=SUM(ARRAYFORMULA(IFERROR(IF(VLOOKUP(FILTER(TRIM(CLEAN(G:G)), TRIM(CLEAN(H:H))="Found"), {TRIM(CLEAN(A:A)), B:B, C:C, D:D}, 4, FALSE)=1, 1, 0), 0)))

Met with @ifried today and discussed the calculated rates.
Ilana is interested in the number of people invited from the list and the rate of registration for those people that were on the list AND were invited.

data compilation sheet

  • Total number of people who were invited: 294
  • Registration for those people that were on the list AND were invited: 12%

Caveats: input data is incomplete

Met with Ilana today and Identified/fixed:
Identified the final list of events (12)
Identified which workbooks had different columns than the others (3)
Identified workbooks that needed additional data entry (multiple)
Identified workbooks which had additional extra characters in the username columns (multiple)
Identified workbooks with duplicated names in G username column (1)

  • ‘template’ file for column architecture and equations: Event_Template

data compilation sheet

  • Total number of people who were invited: 338
  • Registration for those people that were on the list AND were invited: 12%

Caveats: some input data is incomplete/messy; There may be a few more names to include in the tabulations once the input data is cleaned and complete. However, I don't expect the rate to change significantly once the remaining names, data are included.

Thank you for this work and this analysis, @Iflorez. I think we can close this task as Resolved. Do you agree?

We have published our findings (see status update on project page), so I am closing this task as Resolved.