Page MenuHomePhabricator

Analyze "junior contributor" talk page participation
Closed, ResolvedPublic

Assigned To
Authored By
ppelberg
Sep 18 2019, 8:39 PM
Referenced Files
F30911751: senior_contributor_yoy_plot.png
Oct 29 2019, 12:49 AM
F30911745: junior_contributor_yoy_plot.png
Oct 29 2019, 12:49 AM
F30911724: contributors_byuserexperience.png
Oct 29 2019, 12:49 AM
F30912233: contributors_monthly_bywiki_stackedbar.png
Oct 29 2019, 12:49 AM
F30478605: talk_contributors_yoy_plot.png
Sep 27 2019, 12:32 AM
F30408558: talk_contributors_yoy_plot.png
Sep 20 2019, 5:56 PM

Description

We are curious to know the number of distinct junior contributors [1] contributing to talk pages and how that number has changed over time.

This information will be helpful for us in establishing a baseline for one of the key results for the project: Increasing the number of junior contributors participating in conversations on talk pages by 5%. [2]

Done

  • A table that compares the number of distinct "junior contributors" who have contributed to a talk page in the past 30 days and the number of distinct "junior contributors" who contributed to talk pages in a similar 30-day window, 18 months ago.
  • Time series chart showing the year-over-year variation in the distinct number of "junior contributors" who have contributed to a talk page from August 2014 - August 2019
  • Expand analysis to look at number of distinct contributors by experience level and by wikipedia to help inform selection of target wikis
  • Proportion of distinct contributors participating on talk pages each month (calculated by dividing by the overall number of distinct contributors each month for each wiki).

  1. Junior contributor: A contributor who has made <100 cumulative edits.
  2. Our current placeholder is a 5% increase over the lifespan of the project which we're currently estimating to be 18 months

Details

Due Date
Oct 28 2019, 12:00 AM

Event Timeline

ppelberg renamed this task from Analyze talk page participation by experience level to Analyze "junior contributor" talk page participation.Sep 18 2019, 8:48 PM
ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
MNeisler moved this task from Triage to Doing on the Product-Analytics board.

I looked at junior contributors (<100 cumulative edits) talk page participation for the identified time periods. I removed editors tagged with a bot flag.

This data reflects the total number of distinct junior contributors to talk pages and the total number of edits to talk pages from junior contributors across all wikis and all talk page types.

We currently only have data until the end of August 2019 so I did a comparison of August 2018 to 18 months prior (March 2018). While both are 31 day periods, directly comparing different months like this doesn't give us the best comparison of change because of seasonal fluctuations.

Talk Page Junior Contributors 18-Month Period Comparison

March 2018August 2019Percent Change
distinct # of junior contributors3652929272-19.87%
total # of edits by junior contributors9157370760-22.73%

A year over year comparison (YoY) comparing the same month or months helps account for these seasonal changes. A comparison of August 2018 to August 2019 shows a -3.45% percent YoY decline in the monthly total # of unique junior contributors who edited a talk page while the total number of edits to talk pages by junior contributors slightly increased YoY.

Talk Page Junior Contributors YoY Comparison (August 2018 and August 2019)

August 2018August 2019Percent Change
distinct # of junior contributors3031829272-3.45%
total # of edits by junior contributors70290707600.67%

Here's a time series plot showing the changes in the total monthly number of junior contributors who edited a talk page. There's been some fluctuations in this metric that past year with YoY changes varying from a -6.5% YoY decline seen in April 2019 to a 3% increase seen in July 2019.

talk_contributors_yoy_plot.png (1×1 px, 86 KB)

Based on these trends, I'm currently thinking we should target a 5% year over year increase in the # of distinct junior contributors editing a talk page by the end of the project to help account for the seasonal fluctuations in this metric.

@ppelberg - Let me know if you have any questions or other ideas you'd like to look into.

Thank you for turning this around as quickly as you did and for adjusting the time period on the fly to account for the seasonal changes, @MNeisler. It's helpful to have this baseline and to know that targeting a 5% year over year increase in the # of distinct junior contributors editing a talk page by the end of the project is realistic.

A resulting question:

  • Are we able to look further back than August 2018? To your point RE seasonality, I'm thinking there would be value in seeing how these numbers vary over a longer time horizon...something like 5 years (e.g. August 2014 - August 2019).

...if it would be helpful for me to write up this question as new task, please let me know. What works for you, works for me...I only ask considering it extends beyond the initial scope of the task.

@ppelberg - Documenting the decision from our meeting today - I will extend the analysis to see how the numbers vary over the past 5 years. I'll update the description to reflect the new scope.

Here's an updated chart showing year over year changes in the number of distinct junior contributors contributing to talk pages from August 2014 through August 2019.

talk_contributors_yoy_plot.png (1×1 px, 121 KB)

A few observations re trends:

  • This data follows similar seasonal trends we see with overall pageview data. The number of junior contributors contributing to talk pages decreases around the holidays/Christmas and from May to June.
  • From February to June 2019, there was a decline in the number of junior contributors compared to the previous years; however, there was a year over year increase from September to October 2018 and starting again in July 2019.
  • The year over year changes for the past 5 years range from about 7.4% increase (seen in September 2015) to a -9.6% increase (seen in July 2016). The average year over year change is about 0.86%.
MNeisler set Due Date to Oct 28 2019, 12:00 AM.Oct 24 2019, 4:32 PM
MNeisler updated the task description. (Show Details)
MNeisler updated the task description. (Show Details)
MNeisler moved this task from Needs Review to Doing on the Product-Analytics (Kanban) board.

Overall number of distinct contributors participating on Wikipedia talk pages each month by experience level:

Here is a time series plot showing the overall number of monthly distinct junior contributors participating on talk pages over the past 5 years.

contributors_byuserexperience.png (1×1 px, 125 KB)

There are greater fluctuations with junior contributors with under 100 edits; however, there are also more unique contributors in this experience level group than the 500+ or the under 100 edits experience level groups.

Here are the year over year percent changes comparing September 2019 to September 2018 for each group.

dateNo of contributorsedit_countyoy_percent_change
2019-09-0125438under 100+3.68%
2019-09-015302100+-0.80%
2019-09-0119401500++0.48%

A year over year breakdown for junior contributors (under 100 edits) and senior contributors (500 plus edits) helps clarify seasonal impacts and how these have changed over the year.

junior_contributor_yoy_plot.png (1×1 px, 170 KB)

senior_contributor_yoy_plot.png (1×1 px, 159 KB)

Overall there has been a year over year decline in both junior and senior contributors participating on talk pages; however,
however, there were some recent year over year increases in July and September 2019 for senior contributors and from September to October 2019 for junior contributors.

Overall number of distinct contributors participating on Wikipedia talk pages each month by Wikipedia
Next up: Per my comments on T233760, I'll review the proportion of unique junior contributors that participate on talk pages by wiki.

In the meantime, I took a quick look at the top 10 wikis based on the average total of monthly distinct junior contributors that participate on talk pages:

wikiavg monthly contributors
enwiki33032.077
dewiki7774.692
frwiki3822.615
eswiki3263.077
ruwiki2908.769
itwiki2046.385
jawiki1733.000
fawiki1312.846
hewiki1269.846
zhwiki1207.615

Here's a breakdown showing the proportion by user experience for each of these top 10 wikis

contributors_monthly_bywiki_stackedbar.png (1×1 px, 74 KB)

See current notebook for analysis and detailed results

MNeisler updated the task description. (Show Details)

I pulled together a table showing the average number of talk page contributors, average overall monthly contributors, and proportion of talk page contributors for all Wikipedias. You can add filters to sort by the size of each column or look at specific wikis.

https://docs.google.com/spreadsheets/d/1Owvy__kSmnqB2bETZmcyFnuVF6vo7sTRFjcxZfBc2fc/edit#gid=1949681400

Some notes:

  • The time period for this data is over the last year (October 1, 2018 through October 31, 2019).
  • I updated the data to include the latest month so they differ slightly from what is posted above in T233261#5613752. Please refer to the numbers reflected in the linked google doc for the most up to date numbers.
  • The linked google doc includes two sheets (1) monthly talk page contributors overall, which shows the overall numbers by wiki and (2) monthly_talk_page_contributors_byexperience, which includes the numbers for each wiki by user experience level.
  • The highest proportion of talk page contributors end up being the very small Wikipedias because they have so few contributors on their site. It might be useful to filter these out to find wikis that have both a high number of unique contributors and a high proportion of talk page editors.

@ppelberg - Let me know if you have any questions or need any further information.

Notebook
Repo

I pulled together a table showing the average number of talk page contributors, average overall monthly contributors, and proportion of talk page contributors for all Wikipedias. You can add filters to sort by the size of each column or look at specific wikis.

https://docs.google.com/spreadsheets/d/1Owvy__kSmnqB2bETZmcyFnuVF6vo7sTRFjcxZfBc2fc/edit#gid=1949681400

The information you've collected in this sheet and the way you've represented it is what we needed to move T233627 along. Thank you, Megan – this is great.

For context, you can see how I've applied this here in the "Wiki selection" sheet of this Google doc: OWC/Target wiki selection.

An aside: it's encouraging to see that of the number of distinct talk page contributors who use talk pages each month, Junior Contributors make up the greatest percentage of those contributors. This, to me, suggests the enhancements we'll release have the potential to make a big impact on Junior Contributors' experiences participating on the wikis.