Page MenuHomePhabricator

Quick Analysis: impact of adding Thanks to more pages
Closed, ResolvedPublic

Description

User story & summary:

As the Growth team PM, I want to understand the impact T51541 and T90404, because that will ensure I make better, data-driven prioritization decisions (especially in relation to the Positive Reinforcement project)

As a Wikimedian, I'm curious about the impact of features like Thanks, because I care about the long-term health of the wikis.

Background & research:

This task is important because understanding the impact will help us make more informed decisions in the future.

This task covers just the initial quick impact analysis of:

Hypothesis:

This research should help support or reject following hypothesis:

  • Adding Thanks to more pages on Wikipedia will increase the percentage of Edits that get thanked.
Acceptance Criteria:

Quick analysis with a short summary of methodology and results posted in this task.

Note: since we didn't set up A/B testing for these changes we can only look at before and after data to measure the impact. Both Thanks tasks (T51541 and T90404) were on the 2023-01-02 release train, so should have been added to Group 2 wikis on 2023-01-05.

Event Timeline

@Jdlrobson & @NRodriguez - pinging you in case you are interested in following this task.

Adding this to the Product Analytics kanban board, keeping the High priority. The plan is to start this work on March 20 to get an initial perspective on the usage of Thanks as it relates to edits made in the most recent months, then go on from there based on what our needs are.

This comment was removed by Aklapper.

This analysis has been completed. We created a slide deck presenting the findings and decided to upload that to Commons for wider accessibility: https://commons.wikimedia.org/wiki/File:T332441_-_Thanks_Usage_-_Introduction.pdf The notebooks used for data gathering and visualization are in this GitHub repository

Our quick analysis of edits being thanked does not find a clear signal that the wider availability of the feature resulted in more edits being thanked. Patterns of usage are different between wikis, both in the sense of what proportion of edits are thanked as well as whether we see differences in January and February compared to December. There are also differences when it comes to thanks given by editor tenure.

I had asked for the Thanks link to be added to RC pages. I'm wondering if the study maybe doesn't take into account whether most of the user base even knows this feature is now available, as it was conducted right after implementation.

Also, as for the confusing Thanks confirmation, I do wonder if it needs to exist - I mean, is there actually a non-public thanks? Or, is it just a warning that their thanks will be public? At any rate, it's possible that many wishing to give thanks don't notice the confirmation request and move on.

I had asked for the Thanks link to be added to RC pages. I'm wondering if the study maybe doesn't take into account whether most of the user base even knows this feature is now available, as it was conducted right after implementation.

Yes, good point. This analysis was quick and imperfect, since the feature wasn't released as an A/B test. Normally the Growth team would have A/B tested this change to get a better understanding of the impact. However, since the work was tackled outside of the Growth team as part of the Wishathon, it wasn't A/B tested.

Although we didn't find a clear signal that the wider availability of the feature resulted in more edits being thanked, I'm still in support of having Thanks available on pages where it is logical to Thank another editor.

Also, as for the confusing Thanks confirmation, I do wonder if it needs to exist - I mean, is there actually a non-public thanks? Or, is it just a warning that their thanks will be public? At any rate, it's possible that many wishing to give thanks don't notice the confirmation request and move on.

I totally agree. The confirmation message copy and UX is confusing. There is no "non-public thanks", it's just warning that Thanks is public.

My hope is that we can actually instrument Thanks: T71804: Thanks: add instrumentation to Thanks and then eventually work on T333027: Growth design: finalize designs for thanks confirmation UX. As you can see in that second task, we already have some initial design ideas prepared.
The work has somewhat stalled as the Growth team needed to move onto other urgent priorities, but I'm still hoping this is work we can fit in.