Page MenuHomePhabricator

Develop a prototype for Event Invitations with scoring on likelihood of valuable participation
Closed, ResolvedPublic

Description

Background: Before we build out a full-fledged Event Invitation tool, we want to develop a prototype for Event Invitations that is smaller in scope and in technical difficulty. This way, we can still generate an Invitation List for organizers based on their campaign event worklists, but the Campaigns team will provide more hands-on support to organizers who want to use the Invitation List rather than the support being made available through a user interface. The goal is to build a basic prototype that can be used for testing the impact of an Event Invitation tool, so we can decide if a) Event Invitations can increase event registrations and therefore campaign contributions for organizers who use the tool, and b) if we do or do not want to continue investing in building an Event Invitation tool based on the findings of the prototype experiment.

Acceptance Criteria:

  • We should be able to generate an Invitation List based on a list of Wikipedia articles
    • The invitation list should be a list of usernames of people who are recommended to invite to the event based on early ideas for potential criteria, which we can iterate on & improve over the course of developing the prototype and talking to various stakeholders
  • Note that we do not need to build a user interface for organizers to generate an Invitation List for the prototype. It is sufficient to an engineer generate the list and to give the list to the ambassadors, who will then give the list to the organizers.

To test:

  • Create a text file with the worklist
    • This file should contain one page per line, and each line must start with the wiki ID (optional, leave empty to use the current wiki), then a colon, then the page title. See example.
  • Run the following command (from the MW root directory): php maintenance/run.php CampaignEvents:GenerateInvitationList --listfile path/to/worklist

Event Timeline

Restricted Application added a subscriber: Aklapper. ยท View Herald TranscriptDec 14 2023, 3:36 PM

@ifried I just realized this task is still a placeholder with no description. Do you have any ideas of what should go in the task description? I'm not really sure how to best put it from a product perspective.

Change 979982 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/extensions/CampaignEvents@master] Add maintenance script for invitation list generator prototype

https://gerrit.wikimedia.org/r/979982

ifried renamed this task from [placeholder] Develop a prototype for Event Invitations with scoring on likelihood of valuable participation to Develop a prototype for Event Invitations with scoring on likelihood of valuable participation.Jan 17 2024, 3:22 PM
ifried updated the task description. (Show Details)

Change 979982 merged by jenkins-bot:

[mediawiki/extensions/CampaignEvents@master] Add maintenance script for invitation list generator prototype

https://gerrit.wikimedia.org/r/979982

Daimona added a subscriber: vaughnwalters.

@vaughnwalters This should be ready for QA now; you can test locally or in beta.

Mentioned in SAL (#wikimedia-operations) [2024-01-22T16:42:24Z] <Daimona> T353459 Running mwscript /home/daimona/GenerateInvitationList.php to test the script before it reaches production

โœ… We should be able to generate an Invitation List based on a list of Wikipedia articles
โœ… The invitation list should be a list of usernames of people who are recommended to invite to the event based on early ideas for potential criteria, which we can iterate on & improve over the course of developing the prototype and talking to various stakeholders
โœ… Note that we do not need to build a user interface for organizers to generate an Invitation List for the prototype. It is sufficient to an engineer generate the list and to give the list to the ambassadors, who will then give the list to the organizers.

==Articles==
[0:Dogs]@my_wiki
[0:Elephants]@my_wiki
[0:Cows]@my_wiki

==Contributions==
Admin - [0:Dogs]@my_wiki - 19000
Admin - [0:Cows]@my_wiki - 199
Admin - [0:Elephants]@my_wiki - 3793
Test1 - [0:Dogs]@my_wiki - 1005
Test1 - [0:Cows]@my_wiki - 600
Test2 - [0:Elephants]@my_wiki - 43


==Scoring debug info==
User Admin max bytes 19000 with score 0.99995545740812
User Admin bytes score #1: 0.72119161881604
User Admin bytes score #2: 9.101735421629E-47
User Admin overall bytes score 0.99998758115206
User Admin edit count 113, score 0.34144920980106
User Admin last edit 0.26790509259259 days ago, score 0.99999969942329
User Test1 max bytes 1005 with score 0.20807005680788
User Test1 bytes score #1: 0.11895953588636
User Test1 overall bytes score 0.30227767530452
User Test1 edit count 6, score 0.051202494919298
User Test1 last edit 0.0030439814814815 days ago, score 0.99999999999955
User Test2 max bytes 43 with score 0
User Test2 overall bytes score 0
User Test2 edit count 7, score 0.056674843149996
User Test2 last edit 4.6296296296296E-5 days ago, score 1



==Contributor scores==
Admin - 90
Test1 - 46
Test2 - 0

@Daimona this is very cool! It seems that anyone who makes any edit on the articles in the text file are noted in the results (though with a score of zero), even when they make the smallest edit possible. I guess we would just only submit to organizers people who have a score over X, whatever that score may be then?


Also noting that if there is an article that does not exist named in the text file, then no results will be shown, even if the other articles do have matches. I don't think this matters since this is an internal tool run by developers only, but just noting it here.

I have no name!@5ac87427c9d7:/var/www/html/w$ php maintenance/run.php CampaignEvents:GenerateInvitationList --listfile testlist.txt
Page does not exist: derp

Sending this to product sign off.

@Daimona this is very cool! It seems that anyone who makes any edit on the articles in the text file are noted in the results (though with a score of zero), even when they make the smallest edit possible. I guess we would just only submit to organizers people who have a score over X, whatever that score may be then?

Yup, that is correct. We don't really know what the threshold X would be in practice (it depends on the event, the size of the resultset etc.), but we definitely don't want to include anyone with a score of 0. If and when we built a frontend for the tool, we would most likely implement the threshold in the code.

Also noting that if there is an article that does not exist named in the text file, then no results will be shown, even if the other articles do have matches. I don't think this matters since this is an internal tool run by developers only, but just noting it here.

Indeed. I wanted to make failure states as obvious as possible given the nature of the tool.

Mentioned in SAL (#wikimedia-operations) [2024-02-06T16:26:59Z] <Daimona> T353459 Running mwscript CampaignEvents:GenerateInvitationList --wiki=metawiki --listfile=/home/daimona/list.txt

@Daimona, thank you for this fantastic work! Perhaps we can put somewhere in the documentation of this ticket how the score is calculated/the current equation? While it will probably change over time, it could be useful for us to refer to in the future (and apologies if this is already documented somewhere and I just missed it).

@Daimona, thank you for this fantastic work! Perhaps we can put somewhere in the documentation of this ticket how the score is calculated/the current equation? While it will probably change over time, it could be useful for us to refer to in the future (and apologies if this is already documented somewhere and I just missed it).

I had documented this quite extensively in the code itself. I reckon it might not be the best place, but it's also true that, as you mention, the criteria might (will?) change. Did you have a specific idea of documentation in mind?

Thanks for the response, @Daimona! I think the link you provided in the comments is sufficient; I just wanted to get something in the comments for general documentation. Much appreciated!

The model is now being used to generate Invitation Lists, following an internal validation test with Community Programs, so I am marking this work as Done.