Page MenuHomePhabricator

Add Experience Level calculations and flags, including for Newcomer, to the ReviewStream feed
Closed, DeclinedPublic

Description

Goal: Research shows that new editors are particularly vulnerable to rejection. To add prominence to the special needs of new users, standardize a new-user definition that’s useful for reviewing purposes, and enable downstream tools to identify and filter for new users more easily, we’ll calculate and flag new user status explicitly in the feed. To provide users the ability to filter with more granularity, we'll also flag a few other standardized "experience levels."

  • Research shows that the first few days of activity are when users are most vulnerable to rejection.
  • When practical, thresholds will correspond with existing definitions.
  • That said, thresholds should be easily configured and should be configurable on a per-wiki basis.
  • Also, the terminology for ReviewStream will be independent of individual wiki terms. (E.g., in the example below, the feed property will be called "more experienced" not "extended confirmed," though the two will be equal out of the gate.)
  • After some debate, we decided that the flag should apply to edits by all users, registered and unregistered alike .

With that, we have defined the following Experience Levels:

Newcomer
Fewer than 10 edits and 4 days of activity.

Experienced
More days of activity and edits than “Newcomers” but fewer than “More experienced.” (corresponds to autoconfirmed)

More experienced
More than 30 days of activity and 500 edits. (corresponds to extended confirmed)

Event Timeline

Flag High Experience Editors as Well?
Newcomers are a particular interest of the ERI project. However, in considering how reviewers might use experience level in filtering, @Pginer-WMF suggested it might be desirable to have a high-end user data flag as well. Reviewers might, for example, decide to filter out such users as presumed safe. So, two questions:

  • A) Does this sound like a good idea for a) including as a filter on the RC page, b) including also in the new feed, c) both?
  • B) there are many types of privileged uses with different access levels; which groups would we include in this "presumed innocent" category?
  • C) What would we call this? Something like "Admins and Trusted Editors"?

Some experienced users may do bad faith edits sometimes, on purpose or by accident. For instance, I've removed categories in an article last week because of a bad copy/pasting. I think there is no counter-indication to have on one side good faith and bad faith edits and on the other side filters by editors status (IP, beginners, autoconfirmed, autoreviewed).

However, I would not add a filter to target users by maintenance status (admins, bureaucrats, etc.). These users don't have a status with more rights concerning edits than others. They are basic users, with extended tools. If they have a status it is a cultural one, but not official.

Some experienced users may do bad faith edits sometimes, on purpose or by accident.

Note that by providing separate filters those aspects are not coupled.
A user aimed at helping those users that may not be familiar with the rules, may want to exclude "experienced users" from the edits to review.

So I would not presume those are "innocent", but I think they are relevant cases where you may want to include or exclude them on certain activities. In other words, it seems an often referred group in different contexts (even if there is no formal definition universally accepted) for it.

@Pginer-WMF and @Trizek-WMF, thanks for your comments. I was assuming this was a kind of "Trusted user" category—similar to how Autopatrolled works. So, my proposal would have been to include similar groups, which the system seems to regard as trusted users in various senses. Something like this:

Admins, Bureaucrats, Pending changes reviewers, Rollbackers, Autopatrolled, Oversighters.

But it sounds like Pau has something different in mind. Pau, what is your scenario? And what would the definition be?

Some experienced users may do bad faith edits sometimes, on purpose or by accident.

I don't think that "bad faith" is the right word here. If you accidentally damaged Wikipedia, I'd say you did it in "good faith". But with that said, I certainly agree that experienced editors will accidentally do damage all the time.

IMO, the most concerning biasing factors are "newcomer" and "IP editor" and if we're flagging those already, we're not doing much harm by flagging advanced permissions.

Each wiki has a different set of user groups, so I don't think it's a good idea to create a merged class of "Trusted user", but rather to just report the user group memberships in raw form like Special:ListUsers.

@Halfak wrote:

just report the user group memberships in raw form like Special:ListUsers.

If I understand you here, you're suggesting we could enable filtering by groups (a data type that will be in the new feed, BTW). That's an interesting idea for a future enhancement.

Re. the question at hand, do you see a use case for the suggested "Most Experienced" filter? Which would be grouped with Newcomers, something like so (we'd have to define the levels better):

User experience level

Newcomer
Editors that joined recently or with few edits.

More experienced
Edits by those who are not Newcomers, but not the most experienced editors either.

Most experienced
Edits by the most productive editors.

But it sounds like Pau has something different in mind. Pau, what is your scenario? And what would the definition be?

I think that we should aim for groups of filters to be defined around the same concepts. If we are talking about expertise in terms of number and frequency of editing, I'd expect all filters to represent ranges based on those. Mixing different aspects (edit productivity and user groups) makes groups harder to understand and generate problems when filtering (potentially creating relevant intersections you won't be able to filter).

I was considering a spectrum where we define three levels of expertise based on edits and time. I'm ok with any specific values. We may consider reusing the "active" or "very active" user definition which can define the following ranges:

  • Newcomer. Editors with less than 50 edits or editing for less than one week.
  • More experienced. Editors with at least 50 edits or editing for more than a week but not reaching 100edits per month.
  • Most experienced. Editors with 100 edits per month or more.

Some example scenarios where targeting the most experienced editors may make sense:

  • As a Teahouse host, I want to look for users that may become hosts to help answering newcomer questions. I can filter for very experienced users making good contributions in order to invite them to act as hosts in the Teahouse.
  • As a reviewer for the Wikiproject Astronomy, I want to focus on helping editors making common mistakes to give them guidance or invite them to discuss their edits in the talk page. So I prioritise the damaging contributions made in good faith by users that are not "the most experts" who are likely to know how talk pages work and basic editing rules already (although I may include them in other vandalism related filtering activities).

Editors with 100 edits per month or more.

The 'per month' of this makes it strangely defined. This seems to conflate activity level with experience level. E.g. I've made 4514 edits to English Wikipedia, but only 20 of those were in the last month[1]. Am I "more experienced" or "most experienced"?

  1. https://quarry.wmflabs.org/query/12525

I'm thinking that it might be best to associate with community consensus definitions that are baked into the software. If we use these definitions, then we can encourage individual communities to optimize these permissions-giving thresholds to actually match the expected experience level of editors.

  • Newcomer == Pre-autoconfirmed
  • More experienced == Autoconfirmed (enwiki: 4 days tenure and at least 10 edits)
  • Experienced == Extended confirmed (enwiki: 30 days tenure and at least 500 edits)

FWIW, I find these thresholds intuitive. The danger zone where most good-faith newcomers leave Wikipedia is pre-autoconfirmed. We send people to the Teahouse when they are pre-autoconfirmed. If you are autoconfirmed and not blocked, you are probably good-faith and a little bit more robust feedback. If you are extended confirmed, you're already old and crusty whether you believe it or not. You probably don't believe it until you've reached ~2000 edits and even then some people feel like impostors.

Editors with 100 edits per month or more.

The 'per month' of this makes it strangely defined.

I was trying to reuse existing definitions. I'm ok with other criteria especially if it is useful but simpler to understand. My main point was that in those analysis, it is common to think in three levels of expertise (as opposed to two:t "noobs" and "the rest"). So regardless of the specific measurement I think it would be useful to have these three levels reflected in our filters about user experience.

The measurement needs of analysis differ from the needs of a user doing reviewing work. In the case of analysis, we often choose to use something that is easy to compute historically over something that is most intuitive if they are strongly correlated at the scale of large populations. When we are looking at individuals, it's probably best to use the most intuitive measures -- especially when the difficulty of generating them on demand has already been solved within MediaWiki.

  • Experienced == Extended confirmed (enwiki: 30 days tenure and at least 500 edits)

You can have people who have reached that goal years ago and resume contributing after a loooong wikibreak. Communities are not considering them as experienced, because most of time rules have evolved so as tools and practices. I don't know if we can (or want) to include them into our filtering system.

Communities are not considering [people come back after a substantial wikibreak] as experienced

That seems extreme. Can you point to an example of this being the case?

Communities are not considering [people come back after a substantial wikibreak] as experienced

That seems extreme. Can you point to an example of this being the case?

I remember a case: a friend of mine, an experienced Wikipedia user who has written featured contents and get involved into community discussions moved to Commons, because that projects fits better his expectations. He came back infrequently to write some article and get once reverted by a patroller because he didn't followed some rules that have changed since he left Wikipedia. A discussions happen then, my friend was like "I'm experienced" and the other person was like "I can't see that into your contributions".

There is also cases of sysops who loose their tools due to inactivity and complain against the system because they can't have them back.

IMO, "Experienced" doesn't mean knowing all of the rules, but rather knowing where the rules are and how to manage disagreements using talk pages and other spaces. In this case, I don't think your friend came back less "experienced" but maybe he did come back less "familiar".

Correct. Plus, I think these cases are relatively rare.

We discussed this in the ERI team discussion on 9/14. We decided to base the top two categories on the autoconfirmed and extended confirmed rights, respectively. All categories apply to Registered users only. Here is suggested wording for these:

User experience level

Newcomer
Very new editors: fewer than 10 edits and 4 days of activity.

Experienced
More days of activity and edits than "Newcomers" but fewer than “More experienced.”

More experienced
More than 30 days of activity and 500 edits

This raises a couple of questions: :

  1. Though the actual definitions may vary (see below), do all wikis have the concepts of autoconfirmed and extended confirmed?
  2. If not, what definitions should we use for wikis that lack them? The en.wiki standards above?
  3. My understanding is that wikis that do employ these concepts may define them differently. Can we specify these levels (in the feed, in the page) in such a way that they automatically conform to the local wiki standards? Or do we need to research and set them individually for each wiki?

autopatrolled and extendedconfirmed rights

"autoconfirmed" and extended confirmed :)

Oh! Also, change "or" to "and".

10 edits and 4 days of activity.

30 days of activity and 500 edits

I have made the following corrections (thanks @Halfak!) to the definitions, links and language above:

"autoconfirmed" (not autopatrolled)
10 edits and 4 days of activity.
30 days of activity and 500 edits

jmatazzoni renamed this task from Add New User calculation and flag to the ERI feed to Add Experience Level calculations and flags, including for Newcomer, to the ReviewStream feed.Sep 23 2016, 12:15 AM

I have a couple of issues to report/discuss related to this ticket:

Add group membership data to the stream

@Halfak suggested above that, instead of creating a special group of most experienced users, we could

just report the user group memberships in raw form like Special:ListUsers.

I answered that that would be a good future enhancement. But I just remembered that group membership data is in fact included in the spec for ReviewStream (look under "metadata about users"). So, good for us!

Including Unregistered Users

At @Halfak's suggestion, we'd made a decision to restrict the so-called "Experience level" categories to registered users only. But in testing of the RC page filters, we've seen this causing problems. Essentially, it introduces a "hidden" property, so even though we state that the "Newcomer" filter, for example, includes registered only, people keep using it incorrectly and not understanding why they get odd results. (E.g., they can, in effect, find themselves searching for the intersection of Registered + Unregistered.)

Since Registered/Unregistered is already a filter of its own on the page (and properties included in the feed), it seems better to simplify this definition so that it relates only to experience. Which is to say, we're leaning towards including IP users when looking at Experience Levels.

If I understand properly, this could lead to the following cases where the filter would be inaccurate:

  • Shared IP addresses: This would result in us missing some new users, who would look like regulars. But we wouldn't have caught the newbies anyway if IPs weren't included, so no harm done.
  • Mobile IPs: This is more serious, because experienced users will show up as "new," which could waste the time of Welcomers, for example. This would, however, affect only the class of experienced users who edit from mobile but don't log in. At present, it's hard to imagine that's a big problem.

The other reason to do this is that the intersection of Newcomer + Unregistered seems to be a space that interests vandalism fighters. Did I miss anything here? Please speak up if including IP users to keep things clean doesn't seem like a reasonable tradeoff.

@JTannerWMF, @Catrope: As both Collaboration-Team-Triage and Edit-Review-Improvements-ReviewStream got archived, please either assign an active project to this task or decline this task. Thank you!