Page MenuHomePhabricator
Create Task
Maniphest T275999

track editor numbers split by namespace
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Problem:
We are tracking the number of editors who make at least 1/5/100 edits in the past 30 days at https://grafana.wikimedia.org/d/000000162/wikidata-site-stats?orgId=1. This is for all edits across all namespaces. It'd be useful to also keep this statistic split by namespace. The main namespace, Property namespace and the Lexeme namespace are of special interest but we will want to look at the others as well.

Acceptance criteria:

Notes:

  • We count only the edits an editor makes in a specific namespace: e.g. if a user makes 10 edits in Lexemes and 100 edits in the main namespace, then they should be counted as active (5 or more edits) for the Lexeme namespace and very active (100 or more edits) for the main namespace.
  • Testing locally cannot happen out of the box

Links that may be useful

Details

ProjectBranchLines +/-Subject
analytics/wmde/scriptsproduction+5 -13Final fixes to get editors split by namespace
analytics/wmde/scriptsproduction+1 -1Fix SQL query field name
analytics/wmde/scriptsproduction+1 -1Fix PHP Fatal error
analytics/wmde/scriptsmaster+5 -13Final fixes to get editors split by namespace
analytics/wmde/scriptsmaster+1 -1Fix SQL query field name
analytics/wmde/scriptsmaster+1 -1Fix PHP Fatal error
analytics/wmde/scriptsproduction+75 -0Track editor numbers split by namespace
analytics/wmde/scriptsmaster+75 -0Track editor numbers split by namespace
Customize query in gerrit

Related Objects
Search...

Event Timeline

Lydia_Pintscher created this task.Feb 28 2021, 3:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2021, 3:15 PM
Lydia_Pintscher triaged this task as Medium priority.Feb 28 2021, 3:16 PM
Lydia_Pintscher moved this task from Incoming to Unconnected Stories on the Wikidata-Campsite board.
Lydia_Pintscher added a subscriber: Lucas_Werkmeister_WMDE.Mar 2 2021, 10:55 AM

FYI Lucas

• amy_rc added a subscriber: • amy_rc.Mar 2 2021, 11:02 AM

Adding Lucas comments from mattermost discussion - As a database query

MariaDB [wikidatawiki]> SELECT rc_namespace, COUNT(DISTINCT rc_actor) AS editors FROM recentchanges WHERE rc_namespace >= 0 GROUP BY rc_namespace ORDER BY editors DESC;

result with registered users only

MariaDB [wikidatawiki]> SELECT rc_namespace, COUNT(DISTINCT rc_actor) AS editors FROM recentchanges JOIN actor ON rc_actor = actor_id WHERE rc_namespace >= 0 AND actor_user > 0 GROUP BY rc_namespace ORDER BY editors DESC;
darthmon_wmde updated the task description. (Show Details)Mar 3 2021, 11:47 AM
darthmon_wmde updated the task description. (Show Details)
darthmon_wmde set the point value for this task to 8.Mar 3 2021, 11:52 AM
darthmon_wmde moved this task from Unconnected Stories to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
darthmon_wmde edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Mar 3 2021, 12:15 PM
Silvan_WMDE claimed this task.Mar 10 2021, 2:11 PM
Silvan_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Lydia_Pintscher mentioned this in T276918: Get statistics about the Lexemes editors.Mar 11 2021, 5:16 PM
gerritbot added a comment.Mar 12 2021, 4:53 PM

Change 671195 had a related patch set uploaded (by Silvan Heintze; owner: Silvan Heintze):
[analytics/wmde/scripts@master] Track editor numbers split by namespace

https://gerrit.wikimedia.org/r/671195

gerritbot added a project: Patch-For-Review.Mar 12 2021, 4:53 PM
Silvan_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 15 2021, 8:29 AM
Silvan_WMDE updated the task description. (Show Details)Mar 16 2021, 8:44 AM
gerritbot added a comment.Mar 17 2021, 11:19 AM

Change 671195 merged by jenkins-bot:
[analytics/wmde/scripts@master] Track editor numbers split by namespace

https://gerrit.wikimedia.org/r/671195

Silvan_WMDE mentioned this in rADES9943b31c0c93: Track editor numbers split by namespace.Mar 17 2021, 11:22 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 17 2021, 12:10 PM
• Tonina_Zhelyazkova_WMDE moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 17 2021, 12:11 PM
Addshore added a subscriber: Addshore.Mar 17 2021, 12:19 PM

This will need to also land in the "production" branch in order to be deployed.
See https://wikitech.wikimedia.org/wiki/WMDE/Analytics#analytics/wmde/scripts_repo

gerritbot added a comment.Mar 17 2021, 2:48 PM

Change 672823 had a related patch set uploaded (by Silvan Heintze; owner: Silvan Heintze):
[analytics/wmde/scripts@production] Track editor numbers split by namespace

https://gerrit.wikimedia.org/r/672823

gerritbot added a project: Patch-For-Review.Mar 17 2021, 2:48 PM
gerritbot added a comment.Mar 19 2021, 7:50 AM

Change 672823 merged by jenkins-bot:
[analytics/wmde/scripts@production] Track editor numbers split by namespace

https://gerrit.wikimedia.org/r/672823

Silvan_WMDE mentioned this in rADES8a8e909c4cd6: Track editor numbers split by namespace.Mar 19 2021, 7:52 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 19 2021, 8:10 AM
Silvan_WMDE added a comment.Mar 19 2021, 6:18 PM

waiting for some data to show up in graphite before the grafana panel can be added

Silvan_WMDE moved this task from Doing to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 19 2021, 6:18 PM
Daniel_Mietchen added a subscriber: Daniel_Mietchen.Mar 22 2021, 5:38 PM
Lea_Lacroix_WMDE added a parent task: T276918: Get statistics about the Lexemes editors.Mar 25 2021, 8:41 AM
• Tonina_Zhelyazkova_WMDE moved this task from Stalled/Waiting to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Mar 29 2021, 8:59 AM
• Tonina_Zhelyazkova_WMDE added a subscriber: • Tonina_Zhelyazkova_WMDE.
In T275999#6929441, @Silvan_WMDE wrote:

waiting for some data to show up in graphite before the grafana panel can be added

Moving this back to Doing because there should already be enough data in graphite.

Silvan_WMDE added a comment.Apr 8 2021, 11:52 AM

PHP error message in production, according to people in #wikimedia-analytics:

PHP Fatal error:  Uncaught TypeError: Argument 1 passed to WikidataActiveUsersByNamespace::collectNamespaces() must be of the type array, object given, called in /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_users_by_namespace.php on line 26 and defined in /srv/analytics-wmde/graphite/src/scripts/src/wikidata/site_stats/active_users_by_namespace.php:55
gerritbot added a comment.Apr 8 2021, 12:32 PM

Change 677871 had a related patch set uploaded (by Silvan Heintze; author: Silvan Heintze):

[analytics/wmde/scripts@master] Fix PHP Fatal error

https://gerrit.wikimedia.org/r/677871

gerritbot added a project: Patch-For-Review.Apr 8 2021, 12:32 PM
Silvan_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 8 2021, 12:38 PM
gerritbot added a comment.Apr 8 2021, 2:41 PM

Change 677722 had a related patch set uploaded (by Silvan Heintze; author: Silvan Heintze):

[analytics/wmde/scripts@production] Fix PHP Fatal error

https://gerrit.wikimedia.org/r/677722

gerritbot added a comment.Apr 8 2021, 2:44 PM

Change 677871 merged by jenkins-bot:

[analytics/wmde/scripts@master] Fix PHP Fatal error

https://gerrit.wikimedia.org/r/677871

Silvan_WMDE mentioned this in rADES0bc1f76bbe12: Fix PHP Fatal error.Apr 8 2021, 2:44 PM
gerritbot added a comment.Apr 8 2021, 3:13 PM

Change 677722 abandoned by Silvan Heintze:

[analytics/wmde/scripts@production] Fix PHP Fatal error

Reason:

more errors to fix, before this can be deployed

https://gerrit.wikimedia.org/r/677722

gerritbot added a comment.Apr 8 2021, 3:23 PM

Change 677935 had a related patch set uploaded (by Silvan Heintze; author: Silvan Heintze):

[analytics/wmde/scripts@master] Fix SQL query field name

https://gerrit.wikimedia.org/r/677935

gerritbot added a comment.Apr 8 2021, 3:28 PM

Change 677935 merged by jenkins-bot:

[analytics/wmde/scripts@master] Fix SQL query field name

https://gerrit.wikimedia.org/r/677935

Silvan_WMDE mentioned this in rADES6ad4fa30ea4e: Fix SQL query field name.Apr 8 2021, 3:29 PM
gerritbot added a comment.Apr 8 2021, 4:41 PM

Change 677975 had a related patch set uploaded (by Silvan Heintze; author: Silvan Heintze):

[analytics/wmde/scripts@master] Final fixes to get editors split by namespace

https://gerrit.wikimedia.org/r/677975

gerritbot added a comment.Apr 8 2021, 4:47 PM

Change 677975 merged by jenkins-bot:

[analytics/wmde/scripts@master] Final fixes to get editors split by namespace

https://gerrit.wikimedia.org/r/677975

Silvan_WMDE mentioned this in rADESf8d75826eca9: Final fixes to get editors split by namespace.Apr 8 2021, 4:48 PM
gerritbot added a comment.Apr 8 2021, 4:58 PM

Change 677722 restored by Addshore:

[analytics/wmde/scripts@production] Fix PHP Fatal error

https://gerrit.wikimedia.org/r/677722

gerritbot added a comment.Apr 8 2021, 5:00 PM

Change 677950 had a related patch set uploaded (by Addshore; author: Silvan Heintze):

[analytics/wmde/scripts@production] Fix SQL query field name

https://gerrit.wikimedia.org/r/677950

gerritbot added a comment.Apr 8 2021, 5:01 PM

Change 677951 had a related patch set uploaded (by Addshore; author: Silvan Heintze):

[analytics/wmde/scripts@production] Final fixes to get editors split by namespace

https://gerrit.wikimedia.org/r/677951

gerritbot added a comment.Apr 8 2021, 5:02 PM

Change 677722 merged by jenkins-bot:

[analytics/wmde/scripts@production] Fix PHP Fatal error

https://gerrit.wikimedia.org/r/677722

Addshore mentioned this in rADES5da5c5d50c55: Fix PHP Fatal error.Apr 8 2021, 5:02 PM
gerritbot added a comment.Apr 8 2021, 5:03 PM

Change 677950 merged by jenkins-bot:

[analytics/wmde/scripts@production] Fix SQL query field name

https://gerrit.wikimedia.org/r/677950

gerritbot added a comment.Apr 8 2021, 5:03 PM

Change 677951 merged by jenkins-bot:

[analytics/wmde/scripts@production] Final fixes to get editors split by namespace

https://gerrit.wikimedia.org/r/677951

Addshore mentioned this in rADES742bf1f73eb6: Final fixes to get editors split by namespace.Apr 8 2021, 5:06 PM
Addshore mentioned this in rADES7ae4bbb6f0e5: Fix SQL query field name.
Silvan_WMDE added a comment.Apr 8 2021, 5:17 PM

added three panels (1/5/100) for editor numbers split by namespace (T275999) to the "Wikidata Site Stats" dashboard on grafana

Side note: the way user activity is fetched from the database does not only include edits, but also other activities with log entries, such as patrolling or deleting. We assume this is expected, so maybe the dashboard description should be changed, as it currently says:

Active Users - The number of users that have made more than 1, 5 or 100 edits in the past 30 days.

Lydia_Pintscher added a comment.Apr 9 2021, 7:42 AM
In T275999#6984929, @Silvan_WMDE wrote:

added three panels (1/5/100) for editor numbers split by namespace (T275999) to the "Wikidata Site Stats" dashboard on grafana

Yay!
Any chance we can add human readable names for the namespaces? Having to look up which ID goes with which namespace is a bit cumbersome.

Side note: the way user activity is fetched from the database does not only include edits, but also other activities with log entries, such as patrolling or deleting. We assume this is expected, so maybe the dashboard description should be changed, as it currently says:

Active Users - The number of users that have made more than 1, 5 or 100 edits in the past 30 days.

That's also the case for the existing data that we have for all namespaces, right? In this case I'd leave it as is because then by convention those are all also treated as edits. If it differs then we should change it yeah.

Silvan_WMDE moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 9 2021, 10:06 AM
Silvan_WMDE added a comment.EditedApr 9 2021, 1:45 PM

Any chance we can add human readable names for the namespaces? Having to look up which ID goes with which namespace is a bit cumbersome.

grafik.png (119×576 px, 19 KB)

aka

grafik.png (191×1 px, 56 KB)

aka

sortByName(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasSub(aliasByNode(daily.wikidata.site_stats.active_users_by_namespace.*.1, 4), '^-1$', '-1: Special'), '^0$', '0: Main'), '^1$', '1: Talk'), '^2$', '2: User'), '^3$', '3: User Talk'), '^4$', '4: Wikidata'), '^5$', '5: Wikidata Talk'), '^6$', '6: File'), '^7$', '7: File Talk'), '^8$', '8: MediaWiki'), '^9$', '9: MediaWiki Talk'), '^10$', '10: Template'), '^11$', '11: Template Talk'), '^12$', '12: Help'), '^13$', '13: Help Talk'), '^14$', '14: Category'), '^15$', '15: Category Talk'), '^120$', '120: Property'), '^121$', '121: Property Talk'), '^122$', '122: Query'), '^123$', '123: Query Talk'), '^146$', '146: Lexeme'), '^147$', '147: Lexeme Talk'), '^640$', '640: EntitySchema'), '^641$', '641: EntitySchema Talk'), '^828$', '828: Module'), '^829$', '829: Module Talk'), '^1198$', '1198: Translations'), '^1199$', '1199: Translations Talk'), '^2600$', '2600: Topic'), true)

(for lack of a better location to document and version-control this)

Silvan_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 9 2021, 1:46 PM
Lydia_Pintscher added a comment.Apr 9 2021, 2:11 PM

\o/
Beautiful!

Silvan_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 12 2021, 11:16 AM
• amy_rc moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Apr 13 2021, 8:38 AM
• amy_rc closed this task as Resolved.Apr 14 2021, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL