Page MenuHomePhabricator
Create Task
Maniphest T186736

Track Git activity in canonical Wikimedia repositories located on Github in wikimedia.biterg.io
Closed, ResolvedPublic
Actions

Description

Currently on https://wikimedia.biterg.io we only track Gerrit and Wikimedia Git activity.

Tracking Git activity (not: PRs and tasks) of GitHub repos which are related to Wikimedia would give us more complete developer activity statistics.
This is complicated by T109939 as AFAIK there is no way to identify repos on GitHub whose canonical home is on GitHub and which are not mirrored from Gerrit/Git to GitHub.

(For the records, I once upon a time tried a bunch of things in T163576 to investigate.)

Related Objects
Search...

Event Timeline

Aklapper changed the task status from Open to Stalled.Feb 7 2018, 6:08 PM
Aklapper triaged this task as Lowest priority.
Aklapper created this task.
Aklapper added a subtask: T109939: For mirrored GitHub repositories, actually give the canonical source Gerrit URL in the repo description.
Aklapper added a comment.May 19 2018, 4:58 PM

This also affects https://github.com/wmde (as brought up by jayvdb) plus potential personal GitHub repos of people.

Aklapper mentioned this in T194799: Technical development statistics (Git and Gerrit) on https://wikimedia.biterg.io/.May 29 2018, 10:44 AM
Aklapper added a comment.Jun 4 2018, 11:52 AM

I think I mentioned this somewhere before and than forgot, but something [?] seems to be enabled already. At least I can see some GitHub repos for this query on Git: https://wikimedia.biterg.io/goto/b4231e8afff0f2865a75e2afb46ec2e7 (sepcifically: https://github.com/wikimdeia/jquery.uls and .ime).

Need to understand why exactly (and probably ask Bitergia about that, until we have Bestiary in place).

Aklapper added a comment.EditedJul 6 2018, 2:50 PM

https://wikimedia.biterg.io/app/kibana#/dashboard/Git already indexes:

Everything else is from https://gerrit.wikimedia.org/r/

It looks like this configuration was taken over from https://github.com/Bitergia/mediawiki-repositories/blob/master/git_repositories.conf

Aklapper added a comment.Jul 18 2018, 9:02 AM

Correcting the actual scope of this task (Bitergia to deploy Bestiary; Wikimedia to fix T109939) before we can do anything here.

Aklapper renamed this task from Consider enabling GitHub backend in wikimedia.biterg.io to cover canonical Wikimedia repositories not in Gerrit to Track canonical Wikimedia repositories on Github in wikimedia.biterg.io.Jul 18 2018, 9:02 AM
Aklapper mentioned this in T202186: Provide FY2017-18 update of dev stats for audit report.Aug 18 2018, 4:09 PM
Aklapper added a comment.Jan 10 2019, 1:55 PM

I've asked for clarification in (non-public) https://gitlab.com/Bitergia/c/Wikimedia/support/issues/52

Aklapper added a comment.Jan 16 2019, 6:58 PM

Documented in https://www.mediawiki.org/w/index.php?title=Community_metrics&type=revision&diff=3056548&oldid=3056018 what's possible and where.

Aklapper claimed this task.Jan 18 2019, 4:59 PM

Asked followup question in https://gitlab.com/Bitergia/c/Wikimedia/support/issues/54 (eliminating duplicates automatically for mirrored repositories?); plus want to sort out https://gitlab.com/Bitergia/c/Wikimedia/support/issues/55 first before trying this.

Aklapper changed the task status from Stalled to Open.Jan 20 2019, 8:30 PM
Aklapper edited projects, added Developer-Advocacy (Jan-Mar 2019); removed Developer-Advocacy.

Removing "stalled" status as I might do this manually. Note to myself: https://www.mediawiki.org/wiki/Gerrit/GitHub lists more Github orgs to potentially index.

Aklapper added a parent task: T179820: Advertise wikimedia.biterg.io more widely in the Wikimedia community.Jan 20 2019, 8:55 PM
Aklapper added a comment.Jan 23 2019, 7:29 PM

Many repos under https://github.com/wikimedia/ too blurry to check if a mirror and no time to investigate manually, but https://gitlab.com/Bitergia/c/Wikimedia/sources/merge_requests/2 should improve things already vastly and also cover orgs listed under https://www.mediawiki.org/wiki/Gerrit/GitHub

Aklapper removed a parent task: T179820: Advertise wikimedia.biterg.io more widely in the Wikimedia community.Mar 15 2019, 9:04 PM
Aklapper renamed this task from Track canonical Wikimedia repositories on Github in wikimedia.biterg.io to Track Git activity in canonical Wikimedia repositories located on Github in wikimedia.biterg.io.Mar 16 2019, 1:07 PM
Aklapper updated the task description. (Show Details)
Aklapper mentioned this in T218475: Enable Github backend on wikimedia.biterg.io for canonical Wikimedia repositories located on Github.Mar 16 2019, 1:10 PM
Aklapper added a parent task: T218529: Investigate indexing Issues and PRs of Github repositories canonical on wikimedia.biterg.io.Mar 17 2019, 11:52 PM
Aklapper removed a parent task: T218529: Investigate indexing Issues and PRs of Github repositories canonical on wikimedia.biterg.io.
Aklapper removed a subtask: T109939: For mirrored GitHub repositories, actually give the canonical source Gerrit URL in the repo description.Mar 17 2019, 11:56 PM
Aklapper added a parent task: T218529: Investigate indexing Issues and PRs of Github repositories canonical on wikimedia.biterg.io.
Aklapper raised the priority of this task from Lowest to Medium.EditedMar 18 2019, 12:01 AM

I finished going through all previously 209 repositories in Github under https://github.com/wikimedia/ that don't have the word mirror in their project description && do not have an empty description. That number is now 189.

And I also checked those previously 134 repositories without a description.

The result is in https://gitlab.com/Bitergia/c/Wikimedia/sources/merge_requests/3 which should allow us to index nearly all Git repositories currently canonical on Github. Once that merge request is accepted, deployed, and verified I am going to close this task.

The exceptions for "nearly all" are some repositories that confused me too much and I did not sort out:

I deliberately did not add 56 Wikimedia Git repositories on Github forked from another Github upstream. We could index them but we'd have more noise from activity that's not Wikimedia but upstream activity. Have not made up my opinion on this.

As a side effect, I added "this is a mirror from Gerrit" yaddayadda to a good bunch of repositories after checking that code review has actually happened on Gerrit (as we also mirror some repositories the other way round from Github to Gerrit; plus for completeness one repository in Github is mirrored from Diffusion but we don't index Diffusion).
As a side effect, I also reduced the number of Github repositories with an empty description from previously 134 to 95.

Followup issues:

Aklapper added a parent task: T179820: Advertise wikimedia.biterg.io more widely in the Wikimedia community.Mar 18 2019, 12:03 AM
Aklapper added a comment.Mar 18 2019, 12:45 AM
In T186736#4892729, @Aklapper wrote:

Asked followup question in https://gitlab.com/Bitergia/c/Wikimedia/support/issues/54 (eliminating duplicates automatically for mirrored repositories?)

Indexing is based on unique hashes hence usually no duplication.
Documented in https://www.mediawiki.org/w/index.php?title=Community_metrics&type=revision&diff=3136903&oldid=3136896

Aklapper closed this task as Resolved.Mar 21 2019, 9:45 AM

This is deployed; 141 additional repos on Github are indexed now on https://wikimedia.biterg.io .

Aklapper mentioned this in T194557: Add Wikimedia newcomer friendly projects / good first bugs in external repositories .Mar 23 2019, 11:18 PM
Aklapper mentioned this in T218528: Sort out a process to add new Github repositories to wikimedia.biterg.io Git indexing.Mar 29 2019, 2:06 PM
Aklapper removed a parent task: T218529: Investigate indexing Issues and PRs of Github repositories canonical on wikimedia.biterg.io.Jun 28 2019, 1:30 PM
Aklapper mentioned this in T261776: Provide FY2019-20 development contributors statistics for WMF Audit report.Sep 1 2020, 6:18 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL