Page MenuHomePhabricator

Create metrics to measure admin community health
Closed, ResolvedPublic

Description

Based on the discussions with AHT team, guardrail metrics for IP masking can be categorized to several groups. Create this subtask to note all metrics related to admin community. Currently, collected metrics includes:

  • Number of active admins

Definition: an active administrator is a user who has the administrator user right and has made at least one administrative action during a given month. Administration activities include block, protect, delete, and rights. (document).

  • Number of new admins per month

Definition: a new admin is a user who made one administrative action for the first time ever during a given month.

  • Number of admins w/ advanced permissions

Definition: active admins w/ advanced permissions, including bureaucrat admins, oversight admins, local checkuser admin, and steward (global checkuser) admins. Bureaucrat admins and oversight admins are counted in metrics of "Number of active admins" and "Number of new admins per month". Local checkuser admins and steward admins are not counted in metrics of "Number of active admins" and "Number of new admins per month" as they are from another data source. Below four metrics are the numbers for each category.
Bureaucrat admins: admins who grant the sysop user right during a given month
Oversight admins: admins who delete revision during a given month
Local checkuser admins: admins who use checkuser during a given month on wikis with local checkusers. Wikis with local checkusers are 'metawiki','commonswiki', 'specieswiki', 'wikidatawiki','arwiki', 'bnwiki','cawiki','cswiki','dawiki' , 'dewiki', 'enwiki','eswiki','fawiki','fiwiki','frwiki', 'hewiki','hrwiki','huwiki', 'idwiki','itwiki','jawiki', 'kowiki', 'mlwiki', 'nlwiki', 'plwiki', 'ptwiki', 'ruwiki', 'simplewiki', 'slwiki', 'srwiki', 'svwiki', 'trwiki', 'ukwiki', 'viwiki','enwikibooks','enwiktionary'.
Steward admins (global checkuser): admins who use checkuser during a given month on all wikis without local checkusers

  • Admin to content ratio

Definition: Active admins per million edits during a given month

  • Admin to active editor ratio

Definition: Active admins per thousand active editors during the given month. Active editors are registered users who made at least 5 content edits across all projects in the given month.

Event Timeline

jwang triaged this task as Medium priority.May 4 2020, 6:03 PM
jwang moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

An active administrator is a user who has the administrator user right and has made at least one administrative action during a given month. Currently, user is not flagged as admin in any table. Therefore, we count users who executed administration activities ( block, protect, delete, and rights) as administrator. (document).

Use the method mentioned above, here is the trend of monthly active administrator in 2019 and 2020

year_nmonth_nadmins
201914140
201923258
201933521
201943133
201953138
201963172
201973996
201983288
201993251
2019103320
2019113216
2019123364
202013521
202023455
202034106

image.png (590×1 px, 75 KB)

jwang updated the task description. (Show Details)

@cwylo, can you clarify what are advanced permissions?

There are 2 ways to define new admins. Method 1, a new admin is a user who newly registered and has made at least one administrative action during a given month. Method 2, a new admin is a user who made one administrative action for the first time during a given month.

Using the first method, the monthly new admin in past 2 years is very low. Seems it's a rare case that user acts as a admin once he/she has registered an account.

month_nnew admins
2019013
2019021
2019032
2019046
2019052
2019063
2019073
2019081
2019092
2019101
2019114
2019122
2020011
2020021
2020035

Using the second method, the monthly new admin in past 2 years varied around 200. Compared with the previous, this method provides more insight as a guardrail metric.

year_nmonth_nnew_admin
20191223
20192149
20193252
20194136
20195112
20196155
20197179
20198272
20199226
201910168
201911167
201912248
20201196
20202272
20203287

new_admin.png (632×1 px, 86 KB)

jwang updated the task description. (Show Details)
jwang changed the visibility from "Public (No Login Required)" to "All Users".

@cwylo, can you clarify what are advanced permissions?

I would define it as "groups with access level higher than administrators". I'm thinking of these groups in particular:

  • Bureaucrat (users who can grant the sysop user right)
  • Oversight (user right deleterevision)
  • Checkuser (user right checkuser)
  • Steward (user right userrights)

As far as I know these are extended rights in use on all of our wikis.

Aklapper changed the visibility from "All Users" to "Public (No Login Required)".May 5 2020, 6:23 PM

@jwang changed the visibility from "Public (No Login Required)" to "All Users".

@jwang: For the records, I've reset the task visibility, as this can easily be worked around by creating an account which can be done by anyone. (Also see the docs).

Niharika renamed this task from Create metrics to measure admin community to Create metrics to measure admin community health.May 6 2020, 2:17 AM
jwang updated the task description. (Show Details)
jwang updated the task description. (Show Details)

As far as I know these are extended rights in use on all of our wikis.

Correct. The "rights" are defined in code, the groups those rights are assigned to are configurable per-wiki. From the code perspective, the group is irrelevant, we only use the right to determine if the user can do something or not.

  • Admin to content ratio

image.png (606×1 px, 86 KB)

  • Admin to active editor ratio

image.png (610×1 px, 88 KB)

  • Number of admins w/ advanced permissions

Bureaucrat admins: admins who grant the sysop user right during a given month

image.png (614×1 px, 82 KB)

Oversight admins: admins who delete revision during a given month

image.png (624×1 px, 82 KB)

Local checkuser admins: admins who use checkuser during a given month on wikis with local checkusers.

image.png (622×1 px, 85 KB)

Steward admins (global checkuser): admins who use checkuser during a given month on all wikis without local checkusers
image.png (604×1 px, 80 KB)

Have some updated on metrics of Local checkuser admins, Steward admins. Based on the comment from @Strainu at ticket https://phabricator.wikimedia.org/T254375, I removed 'rowiki' from local checkuser wiki list and refreshed the data.

Local checkuser admins: admins who use checkuser during a given month on wikis with local checkusers.

image.png (634×1 px, 86 KB)

Steward admins (global checkuser): admins who use checkuser during a given month on all wikis without local checkusers

image.png (620×1 px, 82 KB)