how to differentiate (exclude) those repositories that are mirrors-only

Link header response for WM repos on GitHub says &page=1964>; rel="last" so there are 1964 repos.
For a comparison of numbers:

$:acko\> ssh aklapper@gerrit.wikimedia.org -p 29418 gerrit ls-projects | wc -l
1736

Trying to somehow get a complete list of repos that are only mirrors there seems to be no one approach. Different spellings; only some mirrored repos have set the homepage key:

$:acko\> wget -q "https://api.github.com/orgs/wikimedia/repos?page=XY&per_page=100" -O fooXY.json
$:acko\> grep -r "Github mirror" . | wc -l
1594
$:acko\> grep -r "GitHub mirror" . | wc -l
2
$:acko\> grep -r "actual code is hosted" . | wc -l
1595
$:acko\> grep -r "\"homepage\": \"https://gerrit.wikimedia.org" . | wc -l
575

Also some repos have no description at all:

$:acko\> grep -r "\"description\": null" . | wc -l
68
$:acko\> cat fooXY.json | jq '.[] | select(.description == null) | .name'       ## list GitHub repos with empty description

Chad was kind enough to point me to:

• T153614: GitHub repository descriptions should point directly to the Gerrit projects
• T114616: Review and update GitHub mirror repo descriptions with standard text
• T109939: For mirrored GitHub repositories, actually give the canonical source Gerrit URL in the repo description

So now there does not seem to be an "easy" way.

In practical terms, to me this task should be block by another one "Create a list of Featured Projects for new developers" and see whether we have any GitHub only projects in that list.

If we have any GitHub only projects, then we can see whether having an export/mirror in Gerrit makes sense. If not, then we can check the metrics problem again.

In other words, I think putting time on this task before having Featured Projects is not a good use of time.

• Created list of GitHub repositories (output from cat foo01.json | jq -r '.[] | .full_name > github.list' after concatenation)
• Created list of Gerrit repositories (output from ssh aklapper@gerrit.wikimedia.org -p 29418 gerrit ls-projects > gerrit.list)
• Stripped wikimedia/ prefix in github.list
• Replaced / by - in gerrit.list
• Sort entries in both files alphabetically
• Ran diff and only show changed lines (via grep and nothing else)
• - means in Gerrit only, + means in GitHub only:

In T163576#3234762, @Qgil wrote:

In other words, I think putting time on this task before having Featured Projects is not a good use of time.

True that. My curiosity was too strong though. :)

"Not a good use of time" is not an appropriate expression for this task. My apologies! :)

I just wanted to clarify how much we need this answer and under which circumstances. Regardless, I am reading with curiosity too. It is an interesting problem.

Would T109939 really be the easy way here? It seems very fragile. Surely reading the master configuration from Phabricator or gerrit or whatever does the mirroring is superior?

In T163576#3991009, @Tgr wrote:

Would T109939 really be the easy way here? It seems very fragile. Surely reading the master configuration from Phabricator or gerrit or whatever does the mirroring is superior?

@Tgr: "whatever does the mirroring": If I understand correctly, replication from Gerrit to Github is done by a Gerrit "replication" plugin. wikitech:Gerrit implies that plugin is com.googlesource.gerrit.plugins.replication (upstream code location). Which brought me to T109939...