Gather data about number of praise-worthy mentees
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Urbanecm_WMF
	Nov 15 2022, 4:43 PM

Description

Within Personalized praise, we will recommend mentees to send praising messages to mentees that are doing well. In the first iteration, we want to do that by suggesting mentors praise mentees that meet all the following conditions:

have more than N edits in the last H hours,
have zero reverted edits

According to T322446, mentors will be able to customize the values N and H. We still, however, need to provide a reasonable set of defaults for mentors to start with.

To avoid guessing the right values of N and H, we should see how many praise-worthy mentees would be suggested per mentor (on average) for the following configurations:

exactly 1 unreverted edits in (timeframe)
2-9 unreverted edits in (timeframe)
10-19 unreverted edits in (timeframe)
20-29 unreverted edits in (timeframe)
30-39 unreverted edits in (timeframe)
40-49 unreverted edits in (timeframe)
50 or more unreverted edits in (timeframe)

where (timeframe) is 48 hours, 1 week, 2 weeks and 1 month.

The data should be gathered for the following wikis (pilots + two big wikis):

arwiki,
bnwiki,
cswiki,
enwiki,
eswiki,
frwiki

Related Objects
Search...

Status	Assigned	Task
Open	KStoller-WMF	T248330 [EPIC] Growth: Positive reinforcement
Open	None	T317817 [EPIC] Positive Reinforcement: Personalized Praise
Resolved	Urbanecm_WMF	T323136 Gather data about number of praise-worthy mentees

Event Timeline

Urbanecm_WMF created this task.Nov 15 2022, 4:43 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 15 2022, 4:43 PM

Urbanecm_WMF added a parent task: T317817: [EPIC] Positive Reinforcement: Personalized Praise.Nov 15 2022, 4:43 PM

Urbanecm_WMF moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.

Maintenance_bot moved this task from New Tasks to In Progress on the GrowthExperiments-Mentorship board.Nov 15 2022, 5:15 PM

KStoller-WMF awarded a token.Nov 15 2022, 6:56 PM

It might be nice to also see data for variations of "N" (Number of edits), but if that's too complex, I'm fine with starting with this. Thanks!

In T323136#8397252, @KStoller-WMF wrote:

I might be nice to also see data for variations of "N" (Number of edits), but if that's too complex, I'm fine with starting with this. Thanks!

Good point. That should be reasonable easy to do. I'd like to cover as many different Ns as possible, without also generating a high amount of data (to ensure we can go through the results).

Perhaps as a first step, we can bucket the numbers together (and in the future, we can look closely at any of the buckets). In other words, let's see how many users have:

exactly 1 unreverted edits in (timeframe)
2-9 unreverted edits in (timeframe)
10-19 unreverted edits in (timeframe)
20-29 unreverted edits in (timeframe)
30-39 unreverted edits in (timeframe)
40-49 unreverted edits in (timeframe)
50 or more unreverted edits in (timeframe)

and use timeframes from the task description/mockups. We'd then have 6 tables (one per wiki) full of numbers to analyze.

Does that seem good to you, @KStoller-WMF?

Sounds great, thanks, @Urbanecm_WMF !

Urbanecm_WMF changed the task status from Open to In Progress.Nov 16 2022, 2:22 PM

Urbanecm_WMF triaged this task as High priority.

Urbanecm_WMF moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.

Urbanecm_WMF updated the task description. (Show Details)Nov 17 2022, 1:33 PM

Most of the wikis defined in the description are below. Each number is an average number of praise-worthy mentees per mentor (from 2022-01-01 to 2022-09-31).

arwiki

time_bucket	1-1 edits	2-9 edits	10-19 edits	30-39 edits	50-∞ edits
48h_id	9.124607799193187	7.227941176470588	1.38370720188902	1.0388349514563107	1.02
day_id	5.63326602725896	4.408125819134994	1.1718249733191035	1.0	1.0
doubleweek_id	36.76523702031603	30.17808219178082	3.95	1.2962962962962963	1.280373831775701
month_id	68.35	56.56880733944954	7.006622516556291	1.7294117647058824	2.0555555555555554
week_id	22.302681992337163	18.102464332036316	2.5054945054945055	1.1504424778761062	1.1368421052631579

bnwiki

time_bucket	1-1 edits	2-9 edits	10-19 edits	30-39 edits	50-∞ edits
48h_id	2.3285606631499625	2.31140350877193	1.1355932203389831	1.0	1.0
day_id	1.6996007984031936	1.7367101303911736	1.065625	1.0	1.0
doubleweek_id	8.193548387096774	7.66358024691358	1.6764705882352942	1.0526315789473684	1.18
month_id	14.840764331210192	14.45	2.414141414141414	1.2105263157894737	1.4444444444444444
week_id	5.0	4.828264758497316	1.371559633027523	1.0434782608695652	1.0606060606060606

cswiki

time_bucket	1-1 edits	2-9 edits	10-19 edits	30-39 edits	50-∞ edits
48h_id	2.503417772416566	2.398480371464753	1.1165254237288136	1.0	1.0
day_id	1.8151950718685832	1.7200112107623318	1.0728051391862956	1.0232558139534884	1.0
doubleweek_id	8.62217659137577	8.92931392931393	1.9166666666666667	1.0757575757575757	1.0862068965517242
month_id	16.06451612903226	17.09090909090909	3.094674556213018	1.2241379310344827	1.28125
week_id	5.4502793296089385	5.529209621993127	1.4388297872340425	1.0483870967741935	1.0196078431372548

eswiki

time_bucket	1-1 edits	2-9 edits	10-19 edits	30-39 edits	50-∞ edits
48h_id	6.818385650224215	7.5414118954974985	1.7341772151898733	1.063953488372093	1.0535714285714286
day_id	4.392641737032569	4.5288835677940735	1.3217317487266553	1.0272108843537415	1.0555555555555556
doubleweek_id	23.225913621262457	30.036303630363037	5.881226053639847	1.7310344827586206	1.8741721854304636
month_id	37.43835616438356	50.244897959183675	10.5	2.6736842105263157	3.701923076923077
week_id	15.113636363636363	18.32758620689655	3.511879049676026	1.3793103448275863	1.3121019108280254

frwiki

time_bucket	1-1 edits	2-9 edits	10-19 edits	30-39 edits	50-∞ edits
48h_id	5.574947132727951	5.440308899860742	1.431184407796102	1.036613272311213	1.017921146953405
day_id	3.5616630442403423	3.427305093366272	1.2069844506755034	1.010126582278481	1.0
doubleweek_id	20.91218872870249	20.823031880286273	3.9595687331536387	1.4096385542168675	1.4412371134020618
month_id	38.357440890125176	38.4525447042641	6.973684210526316	2.1315789473684212	2.3263157894736843
week_id	13.083333333333334	12.797132616487454	2.5559265442404007	1.2193675889328064	1.1513859275053304

Interesting, thank you so much for pulling this data! I have opinions... but I'll keep them to myself for now and let's chat with Ambassadors on Tuesday. I've added the topic to our agenda: Setting a default “praiseworthy mentee” baseline based on data.

In T323136#8436685, @KStoller-WMF wrote:

Interesting, thank you so much for pulling this data! I have opinions... but I'll keep them to myself for now and let's chat with Ambassadors on Tuesday. I've added the topic to our agenda: Setting a default “praiseworthy mentee” baseline based on data.

Sounds great!

FTR, enwiki was originally in the specifications, but it turns out enwiki's too huge to be worked with in the same way as what I did with other wikis. Leaning towards keeping it out for now, and deciding only based on the wikis already listed in my previous comments.

Moving to Watching, as this is done. Keeping it opened for a while so we can discuss the data.

Urbanecm_WMF changed the task status from In Progress to Open.Dec 1 2022, 10:58 PM

Urbanecm_WMF moved this task from In Progress to Ready for QA on the GrowthExperiments-Mentorship board.

KStoller-WMF updated the task description. (Show Details)Dec 2 2022, 9:44 PM

Discussed with Ambassadors. I believe there was consensus that we should allow defaults to be set in community configuration, but that we can use this data to determine a decent baseline that communities can configure.

Gather data about number of praise-worthy menteesClosed, ResolvedPublicActions