Page MenuHomePhabricator

Create and maintain a list of organization repos that are maintained on Gerrit, GitHub, and Diffusion
Closed, ResolvedPublic

Description

There should be very few repos not maintained in gerrit, and for each one we should have a good, on-going reason why we do not maintain the repository in Gerrit. This list will be useful to do things like fix all the npm audit issues appropriately, etc.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 5 2019, 9:35 PM
Krenair added a subscriber: Krenair.
hashar added a subscriber: hashar.

Github had a feature to declare a repository has being a mirror but it can no more be used. There is a hint at https://help.github.com/en/github/getting-started-with-github/finding-open-source-projects-on-github#open-source-projects-with-mirrors-on-github and an example is chromium/chromium:

chromium/chromium Mirror
mirrored from https://chromium.googlesource.com/chromium/src

Github search documentation mentions [[ https://help.github.com/en/github/searching-for-information-on-github/searching-for-repositories#search-based-on-whether-a-repository-is-a-mirror | search qualifier mirror:true ]], which potentially would let us differentiate between mirror and mastered repositories.

Unfortunately there does not seem to be a way for us to flag a repository has being a mirror.


We also do not quite keep the mirrored repositories in sync with what is in Gerrit. For the replication to occur we need to manually create the project on github and when a project is deleted in Gerrit it is not necessarily deleted from GitHub, though lot of deletions are now handled via the Projects-Cleanup workflow.

I guess we could have a script listing all repositories on Gerrit and Github and do a diff. There are a few cases to deal with, on top of my mind:

  • GitHub renames such as mediawiki/core in Gerrit mapping to mediawiki-core in Github which got renamed to simply mediawiki ( https://github.com/wikimedia/mediawiki )
  • Repositories deleted in Gerrit but not deleted in GitHub. Though the project description on GitHub should have a line stating it is a mirror from Gerrit

Maybe we could instead replicate Gerrit repositories to a standalone GitHub organization dedicated to mirroring? It would be closed down to just the few people managing Gerrit / github-mirrors with master repositories staying under the current wikimedia organization.

Tarrow added a subscriber: Tarrow.Nov 11 2019, 12:18 PM
darthmon_wmde removed a subscriber: darthmon_wmde.
darthmon_wmde added a subscriber: darthmon_wmde.
mmodell added a subscriber: mmodell.Dec 4 2019, 6:09 PM
Aklapper renamed this task from Create and maintain somehow a list of repos mastered in GitHub (and in Differential) to Create and maintain somehow a list of repos mastered in GitHub (and in Phabricator Diffusion).Dec 31 2019, 10:15 PM
Aklapper edited projects, added Diffusion; removed Differential.

It looks like it's not documented how/if it's programmatically possible for us to set our mirrored repos as mirrors on GitHub; maybe someone (Tyler?) could ask in an official capacity if it'd be possible for them to tell us what the undocumented API magic is (or… build it?). If so, we could adjust our mirroring script, and then this would be trivial.

Can we crawl all of our GitHub repos and see which have .gitreview in them? Those that don't are presumably mastered in GitHub.

It looks like it's not documented how/if it's programmatically possible for us to set our mirrored repos as mirrors on GitHub; maybe someone (Tyler?) could ask in an official capacity if it'd be possible for them to tell us what the undocumented API magic is (or… build it?). If so, we could adjust our mirroring script, and then this would be trivial.

Reached out to GitHub, there is no api and we'd need to give them a list of repos that are "mirrors"; evidently the 301s in place will stay in place.
Quoting GitHub's reply:

This does have to be done through support I'm afraid, yes.

If you need to raise future requests to establish other mirrors, you could streamline things by providing us with a list of mirrors to establish in the format:

Exact link to external repo -> Exact link to GitHub repo

And a statement saying "I understand the current contents of the GitHub repos will be overwritten.

Can we crawl all of our GitHub repos and see which have .gitreview in them? Those that don't are presumably mastered in GitHub.

I made a thing: https://github.com/thcipriani/wikimedia-github-projects/blob/master/README

Methods

  1. Find all gerrit repos
  2. Find all github repos for the wikimedia organization
  3. Map gerrit repos to github repos (by using "github names"; e.g., mediawiki/core -> wikimedia/mediawiki-core AND by following any 301 redirects; e.g., wikimedia/mediawiki-core -> wikimedia/mediawiki)
  4. Remove the set of gerrit repos from the set of github repos

The remaining github repos are assured to be github only repos. There are 296 GitHub-only repos. Programatically determining whether these repos are developed on GitHub has been problematic.

I've tried a few methods:

GitHub only vs GitHub maintained/.gitreview file

Just because a repo is only present on Wikimedia's GitHub doesn't mean it's maintained there. Likewise a repo that has a .gitreview is not necessarily a repo active in Gerrit.

For example https://github.com/wikimedia/mediawiki-extensions-AddMetaAndTitle has no corresponding gerrit repo ( https://gerrit.wikimedia.org/g/mediawiki/extensions/AddMetaAndTitle ) but does have a .gitreview.

Pull requests

My assumption that repos with pull-requests are repos developed on GitHub turned out to have the following problems:

  1. Leaves out repos maintained on GitHub with a sole-contributor skipping pull requests (https://github.com/wikimedia/OKAPI for example)
  2. Leaves in forks

This opens a sub problem; how do you tell a "fork" in the github sense is a fork in the real, active, sense:

For example: https://github.com/wikimedia/Cyberbot_II seems to be an out-of-date "fork" in github terms, but https://github.com/wikimedia/less.php is maintained by us and is, seemingly, a real live fork.

I came up with the heuristic that a repo is an active for if its parent is archived or if the repo was updated more recently than its parent; however, that's also not quite right as https://github.com/wikimedia/NYTPhotoViewer has active development; however, ours is definitely a fork. I think something with git merge-base could work here; however, that still leaves problem #1.

Change-Ids

I also, briefly, had the theory that commit messages mentioning "Change-Id" are for repos maintained in Gerrit; however, that's not the case.

For example, https://github.com/wikimedia/eventgate is maintained on GitHub; however, there are commits that mention a "Change-Id": https://github.com/wikimedia/eventgate/commit/85d95ea4fde607ba7cd35ced2d2fa05fe888acd5

So what

The outcomes of this are:

  1. We have a list of repos definitely not in gerrit
  2. We could derive a list of repos definitely maintained in GitHub; however, that would be a subset of the repos actually maintained in GitHub

So...¯\_(ツ)_/¯

This is the current list:

https://github.com/wikimedia/AnimatedGIFImageSerialization
https://github.com/wikimedia/BlocksKit
https://github.com/wikimedia/CocoaLumberjack
https://github.com/wikimedia/CombineExt
https://github.com/wikimedia/CommonsNotifier
https://github.com/wikimedia/CopyPatrol
https://github.com/wikimedia/Cyberbot_II
https://github.com/wikimedia/DeadlinkChecker
https://github.com/wikimedia/InteractionTimeline
https://github.com/wikimedia/KafkaSSE
https://github.com/wikimedia/Leaflet.Sleep
https://github.com/wikimedia/MGSwipeTableCell
https://github.com/wikimedia/NSDate-Extensions
https://github.com/wikimedia/NYTPhotoViewer
https://github.com/wikimedia/OAStackView
https://github.com/wikimedia/OKAPI
https://github.com/wikimedia/SDWebImage
https://github.com/wikimedia/SSDataSources
https://github.com/wikimedia/SWStepSlider
https://github.com/wikimedia/SelfSizingWaterfallCollectionViewLayout
https://github.com/wikimedia/SurveyCop
https://github.com/wikimedia/TUSafariActivity
https://github.com/wikimedia/ToolforgeBundle
https://github.com/wikimedia/Tweaks
https://github.com/wikimedia/WhoWroteThat
https://github.com/wikimedia/WikiContrib
https://github.com/wikimedia/WikidataQueryServiceR
https://github.com/wikimedia/WikimediaUI-Style-Guide
https://github.com/wikimedia/YapDatabase
https://github.com/wikimedia/analytics-editor-geocoding
https://github.com/wikimedia/analytics-fundraising
https://github.com/wikimedia/analytics-fundraising-dashboard
https://github.com/wikimedia/analytics-gerrit-stats-data
https://github.com/wikimedia/analytics-quarry-puppet
https://github.com/wikimedia/analytics-query-service
https://github.com/wikimedia/android-mwlogin
https://github.com/wikimedia/aosp-morelangs-ime
https://github.com/wikimedia/aosp-morelangs-ime-dictionaries
https://github.com/wikimedia/apcu
https://github.com/wikimedia/articlequality
https://github.com/wikimedia/banana-checker
https://github.com/wikimedia/banana-i18n
https://github.com/wikimedia/bunyan-syslog-udp
https://github.com/wikimedia/camus
https://github.com/wikimedia/cassandra
https://github.com/wikimedia/change-propagation
https://github.com/wikimedia/codemirror.next
https://github.com/wikimedia/commtech-musikbot
https://github.com/wikimedia/community-tech-tools
https://github.com/wikimedia/composer-merge-plugin
https://github.com/wikimedia/content-type
https://github.com/wikimedia/dClass
https://github.com/wikimedia/data-warehouse
https://github.com/wikimedia/database-reports
https://github.com/wikimedia/docker-library-images
https://github.com/wikimedia/draftquality
https://github.com/wikimedia/drafttopic
https://github.com/wikimedia/dump-scheduler-eval
https://github.com/wikimedia/dumpgrepper
https://github.com/wikimedia/editquality
https://github.com/wikimedia/edittypes
https://github.com/wikimedia/envoy
https://github.com/wikimedia/eslint-config-wikimedia
https://github.com/wikimedia/eslint-docgen
https://github.com/wikimedia/eslint-plugin-mediawiki
https://github.com/wikimedia/eslint-plugin-no-jquery
https://github.com/wikimedia/eventgate
https://github.com/wikimedia/eventmetrics
https://github.com/wikimedia/extensions-Limn
https://github.com/wikimedia/fosdem20-web-performance-cfp
https://github.com/wikimedia/fosdem20webperf
https://github.com/wikimedia/git-client-plugin
https://github.com/wikimedia/git-fat
https://github.com/wikimedia/grunt-contrib-nodeunit
https://github.com/wikimedia/grunt-stylelint
https://github.com/wikimedia/grunt-tyops
https://github.com/wikimedia/hdfs-tools
https://github.com/wikimedia/hpple
https://github.com/wikimedia/html-metadata
https://github.com/wikimedia/htmldumper
https://github.com/wikimedia/hubot-twitter-stream
https://github.com/wikimedia/hyperswitch
https://github.com/wikimedia/incubator-cordova-android
https://github.com/wikimedia/incubator-cordova-ios
https://github.com/wikimedia/incubator-cordova-js
https://github.com/wikimedia/incubator-superset
https://github.com/wikimedia/java-morelangs
https://github.com/wikimedia/jmx_exporter
https://github.com/wikimedia/jquery.i18n
https://github.com/wikimedia/jquery.ime
https://github.com/wikimedia/jquery.uls
https://github.com/wikimedia/jquery.webfonts
https://github.com/wikimedia/json-stable-stringify
https://github.com/wikimedia/jsonschema-compatibility
https://github.com/wikimedia/jsonschema-tools
https://github.com/wikimedia/kad
https://github.com/wikimedia/keyholder
https://github.com/wikimedia/kraken-puppet
https://github.com/wikimedia/labs-tools-WikiConvFR-training-2016
https://github.com/wikimedia/labs-tools-editorinteract
https://github.com/wikimedia/labs-tools-kirstentest
https://github.com/wikimedia/labs-tools-wrcp
https://github.com/wikimedia/language-data
https://github.com/wikimedia/less-plugin-clean-css
https://github.com/wikimedia/less.php
https://github.com/wikimedia/limitation
https://github.com/wikimedia/limn-data
https://github.com/wikimedia/limn-debugging-data
https://github.com/wikimedia/limn-deploy
https://github.com/wikimedia/limn-fundraising-data
https://github.com/wikimedia/logstash-report
https://github.com/wikimedia/mediawiki-api-demos
https://github.com/wikimedia/mediawiki-bots
https://github.com/wikimedia/mediawiki-bots-PHPWikiBot
https://github.com/wikimedia/mediawiki-docker
https://github.com/wikimedia/mediawiki-extensions-AddMetaAndTitle
https://github.com/wikimedia/mediawiki-extensions-AmazonLookup
https://github.com/wikimedia/mediawiki-extensions-CategoryMagicWords
https://github.com/wikimedia/mediawiki-extensions-Censor
https://github.com/wikimedia/mediawiki-extensions-CloudSearch
https://github.com/wikimedia/mediawiki-extensions-ContextComments
https://github.com/wikimedia/mediawiki-extensions-CustomMagic
https://github.com/wikimedia/mediawiki-extensions-DebianISOCodes
https://github.com/wikimedia/mediawiki-extensions-EImage
https://github.com/wikimedia/mediawiki-extensions-ELearnware
https://github.com/wikimedia/mediawiki-extensions-ExtensionStatus
https://github.com/wikimedia/mediawiki-extensions-FlashPlayer
https://github.com/wikimedia/mediawiki-extensions-GoogleTagManager
https://github.com/wikimedia/mediawiki-extensions-HaloTripleStoreConnector
https://github.com/wikimedia/mediawiki-extensions-HotCat
https://github.com/wikimedia/mediawiki-extensions-ISO3166
https://github.com/wikimedia/mediawiki-extensions-ISO639
https://github.com/wikimedia/mediawiki-extensions-ImportBibliography
https://github.com/wikimedia/mediawiki-extensions-InterwikiExistence
https://github.com/wikimedia/mediawiki-extensions-Ipernity
https://github.com/wikimedia/mediawiki-extensions-Isbn
https://github.com/wikimedia/mediawiki-extensions-LightboxGallery
https://github.com/wikimedia/mediawiki-extensions-ListTransclusions
https://github.com/wikimedia/mediawiki-extensions-MailChimpSubscription
https://github.com/wikimedia/mediawiki-extensions-MetaDescriptionTag
https://github.com/wikimedia/mediawiki-extensions-MirrorTools
https://github.com/wikimedia/mediawiki-extensions-MobileSections
https://github.com/wikimedia/mediawiki-extensions-MultiAudioVideo
https://github.com/wikimedia/mediawiki-extensions-NoUnwrap
https://github.com/wikimedia/mediawiki-extensions-PDBHandler
https://github.com/wikimedia/mediawiki-extensions-PageCredits
https://github.com/wikimedia/mediawiki-extensions-PerformanceMonitor
https://github.com/wikimedia/mediawiki-extensions-PlaceNewSection
https://github.com/wikimedia/mediawiki-extensions-PrefixExport
https://github.com/wikimedia/mediawiki-extensions-ProtectedTitles
https://github.com/wikimedia/mediawiki-extensions-PubSubHubbubSubscriber
https://github.com/wikimedia/mediawiki-extensions-PurposeCentricSearch
https://github.com/wikimedia/mediawiki-extensions-QueryResult
https://github.com/wikimedia/mediawiki-extensions-RealNames
https://github.com/wikimedia/mediawiki-extensions-SMWHalo
https://github.com/wikimedia/mediawiki-extensions-SOLRSearch
https://github.com/wikimedia/mediawiki-extensions-ShareThisWidget
https://github.com/wikimedia/mediawiki-extensions-ShrinkTheWeb
https://github.com/wikimedia/mediawiki-extensions-SignupAPI
https://github.com/wikimedia/mediawiki-extensions-SimpleSamlAuth
https://github.com/wikimedia/mediawiki-extensions-StoryParagraph
https://github.com/wikimedia/mediawiki-extensions-SubpageWatchlist
https://github.com/wikimedia/mediawiki-extensions-SwedishCollation
https://github.com/wikimedia/mediawiki-extensions-TemplateDocumentation
https://github.com/wikimedia/mediawiki-extensions-VersionView
https://github.com/wikimedia/mediawiki-extensions-VisualWiki
https://github.com/wikimedia/mediawiki-extensions-WhatsNearby
https://github.com/wikimedia/mediawiki-extensions-WhichImageIsBetter
https://github.com/wikimedia/mediawiki-extensions-WikiCortex
https://github.com/wikimedia/mediawiki-extensions-WikiFarm
https://github.com/wikimedia/mediawiki-extensions-WikivotePageSchemas
https://github.com/wikimedia/mediawiki-extensions-WikivoyageMessages
https://github.com/wikimedia/mediawiki-services-push-notification
https://github.com/wikimedia/mediawiki-services-service-runner
https://github.com/wikimedia/mediawiki-services-service-template-node
https://github.com/wikimedia/mediawiki-title
https://github.com/wikimedia/mediawiki-types
https://github.com/wikimedia/mobile-html-converter
https://github.com/wikimedia/mw-node-qunit
https://github.com/wikimedia/node-htcp-purge
https://github.com/wikimedia/node-repo-template
https://github.com/wikimedia/node-txstatsd
https://github.com/wikimedia/nodejs-driver
https://github.com/wikimedia/oauth2-server
https://github.com/wikimedia/ocean
https://github.com/wikimedia/officeit-puppet
https://github.com/wikimedia/operations-debs-apertium
https://github.com/wikimedia/operations-debs-cg3
https://github.com/wikimedia/operations-debs-contenttranslation-apertium-af-nk
https://github.com/wikimedia/operations-debs-contenttranslation-apertium-api
https://github.com/wikimedia/operations-debs-contenttranslation-apertium-hi-ur
https://github.com/wikimedia/operations-debs-lttoolbox
https://github.com/wikimedia/operations-deployment
https://github.com/wikimedia/operations-network-diagrams
https://github.com/wikimedia/ores
https://github.com/wikimedia/ores-support-checklist
https://github.com/wikimedia/ores-wmflabs-deploy
https://github.com/wikimedia/ores_bias_project
https://github.com/wikimedia/parsoid-jsapi
https://github.com/wikimedia/pcs-html-converter
https://github.com/wikimedia/peformance-docroot
https://github.com/wikimedia/phabricator-translations
https://github.com/wikimedia/phoenix
https://github.com/wikimedia/pi-playbook
https://github.com/wikimedia/popularpages
https://github.com/wikimedia/preq
https://github.com/wikimedia/promise-polyfill
https://github.com/wikimedia/puppet-storm
https://github.com/wikimedia/pywikiapi
https://github.com/wikimedia/pywikibot-bots-catbot
https://github.com/wikimedia/pywikibot-bots-misc
https://github.com/wikimedia/pywikibot-bots-mjbmrbot
https://github.com/wikimedia/pywikibot-externals-six
https://github.com/wikimedia/recommendation-missing-sections
https://github.com/wikimedia/recommendation-translation-test
https://github.com/wikimedia/rescue-pxe
https://github.com/wikimedia/research-api-endpoint-template
https://github.com/wikimedia/research-api-interface-template
https://github.com/wikimedia/research-babel-parser
https://github.com/wikimedia/research-gapfinder-analytics
https://github.com/wikimedia/research-gapfinder-gadget
https://github.com/wikimedia/research-navigation-vectors
https://github.com/wikimedia/research-reader-survey-analysis
https://github.com/wikimedia/research-translation-recommendation-models
https://github.com/wikimedia/research-translation-recommendation-predictions
https://github.com/wikimedia/research-wmf-utils
https://github.com/wikimedia/resource-modules
https://github.com/wikimedia/restbase
https://github.com/wikimedia/restbase-mod-table-cassandra
https://github.com/wikimedia/restbase-mod-table-mysql
https://github.com/wikimedia/restbase-mod-table-spec
https://github.com/wikimedia/restbase-mod-table-sqlite
https://github.com/wikimedia/revscoring
https://github.com/wikimedia/riemann-jmx
https://github.com/wikimedia/service-runner
https://github.com/wikimedia/service-template-node
https://github.com/wikimedia/simplei18n
https://github.com/wikimedia/skinomatic
https://github.com/wikimedia/sqoopy
https://github.com/wikimedia/stylelint-config-wikimedia
https://github.com/wikimedia/subversion
https://github.com/wikimedia/subversion-svn.wikimedia.org
https://github.com/wikimedia/subversion-svn.wikimedia.org-trunk
https://github.com/wikimedia/svgtranslate
https://github.com/wikimedia/swagger-router
https://github.com/wikimedia/takedown-tools
https://github.com/wikimedia/testrepo
https://github.com/wikimedia/thank-you-note
https://github.com/wikimedia/tipsy
https://github.com/wikimedia/tool-gridengine-status
https://github.com/wikimedia/tool-keystone-browser
https://github.com/wikimedia/tool-my-first-flask-oauth-tool
https://github.com/wikimedia/tools-globalsearch
https://github.com/wikimedia/twcs
https://github.com/wikimedia/uxprototypes
https://github.com/wikimedia/ve-dirtydiffbot
https://github.com/wikimedia/ve-needcheck-reporter-bot
https://github.com/wikimedia/video.js
https://github.com/wikimedia/waxer
https://github.com/wikimedia/webextension-readinglists
https://github.com/wikimedia/webpagetest
https://github.com/wikimedia/wikibase-changes
https://github.com/wikimedia/wikidiff2-docker
https://github.com/wikimedia/wikilabels
https://github.com/wikimedia/wikilabels-wmflabs-deploy
https://github.com/wikimedia/wikimedia-IPSet
https://github.com/wikimedia/wikimedia-analytics-wikimetrics-deploy
https://github.com/wikimedia/wikimedia-android-data-client
https://github.com/wikimedia/wikimedia-fundraising-dashboard
https://github.com/wikimedia/wikimedia-logo
https://github.com/wikimedia/wikimedia-page-library
https://github.com/wikimedia/wikimedia-thumbor-djvu-engine
https://github.com/wikimedia/wikimedia-thumbor-ghostscript-engine
https://github.com/wikimedia/wikimedia-thumbor-multi-handler
https://github.com/wikimedia/wikimedia-thumbor-page
https://github.com/wikimedia/wikimedia-thumbor-proxy-loader
https://github.com/wikimedia/wikimedia-thumbor-request-storage
https://github.com/wikimedia/wikimedia-thumbor-tiff-engine
https://github.com/wikimedia/wikimedia-thumbor-video-loader
https://github.com/wikimedia/wikimedia-thumbor-xcf-engine
https://github.com/wikimedia/wikimedia-whatcanidoforwikipedia
https://github.com/wikimedia/wikimedia.github.io
https://github.com/wikimedia/wikimediafoundation-org
https://github.com/wikimedia/wikipedia-android-lite
https://github.com/wikimedia/wikipedia-ios-codemirror
https://github.com/wikimedia/wikipedia-ios-lite
https://github.com/wikimedia/wikipedia-ios-liter
https://github.com/wikimedia/wikipedia-kaios
https://github.com/wikimedia/wikipedia-preview
https://github.com/wikimedia/wikiresearchbot
https://github.com/wikimedia/wikitax
https://github.com/wikimedia/wmfdata-python
https://github.com/wikimedia/work-tracking
https://github.com/wikimedia/wvui
https://github.com/wikimedia/www.wikipedia.org
https://github.com/wikimedia/zotero

Maybe we could instead replicate Gerrit repositories to a standalone GitHub organization dedicated to mirroring? It would be closed down to just the few people managing Gerrit / github-mirrors with master repositories staying under the current wikimedia organization.

My proposal of replicating to a standalone organization still stand. I guess after my last comment I was seeking a dismissal or blessing. Which namely would mean creating all the repositories to mirror under the new organization then change the Gerrit replication configuration.

Maybe we could instead replicate Gerrit repositories to a standalone GitHub organization dedicated to mirroring? It would be closed down to just the few people managing Gerrit / github-mirrors with master repositories staying under the current wikimedia organization.

My proposal of replicating to a standalone organization still stand. I guess after my last comment I was seeking a dismissal or blessing. Which namely would mean creating all the repositories to mirror under the new organization then change the Gerrit replication configuration.

… and transferring 1000+ repos across between the two orgs so that GitHub creates a redirect and maintains any other config (old issues/pulls, meta data, post-merge integrations such as Composer or Coveralls), and then letting Gerrit continue replication on the other side.

I just want to point out some unexpected part of keeping canonical repos in github.

For really long time, github wouldn't allow anyone in Iran access the website. And even was blocking a whole organizational if a user of that org opened github in Iran (causing backlashes like this that led to their CEO's interfering) and finally they lifted the sanctions for Iran. I know we don't get much volunteer contributions from Iran but I want to point to the inherit racism in their decision and the fact that being heavily dependent to corporations might lead to discrimination against our volunteers in unexpected ways (and you might not suffer from it, I don't. But it leaves a bitter taste in my mouth as an Iranian). I highly recommend pushing codes to be in gerrit or (in future) in gitlab.

I just want to point out some unexpected part of keeping canonical repos in github.

For really long time, github wouldn't allow anyone in Iran access the website. And even was blocking a whole organizational if a user of that org opened github in Iran (causing backlashes like this that led to their CEO's interfering) and finally they lifted the sanctions for Iran. I know we don't get much volunteer contributions from Iran but I want to point to the inherit racism in their decision and the fact that being heavily dependent to corporations might lead to discrimination against our volunteers in unexpected ways (and you might not suffer from it, I don't. But it leaves a bitter taste in my mouth as an Iranian). I highly recommend pushing codes to be in gerrit or (in future) in gitlab.

+100 -- decisions like this that limit contributions without cause had a direct impact on the decisions to self-host GitLab on the community edition and is one reason among many to host as much as possible on our own Gerrit install in the interim.

thcipriani triaged this task as Low priority.

To update this ticket: I create a list of repos that are maintained on GitHub and it's posted on MediaWiki: https://www.mediawiki.org/wiki/Gerrit/GitHub#Projects_on_GitHub

The mechanism by which this list is created is https://github.com/thcipriani/wikimedia-github-projects

Summary:

  1. Get a list of all Gerrit repos https://gerrit.wikimedia.org/r/projects/?all
  2. Get a list of all GitHub repos https://api.github.com/users/wikimedia/repos
  3. Get a list of Gerrit repos mirrored to GitHub by making an HTTP HEAD request to GitHub to find canonical repo names and 301s (e.g., mediawiki-core is a 301 to mediawiki)
  4. Get a list of repos on phabricator that are mirrored to GitHub
  5. Repos only on GitHub are == GitHub repos - GitHub mirrors from Gerrit - GitHub mirrors from phabricator
  6. Add back repos from phabricator that are mirrored from GitHub to Gerrit -- there are currently 5.

Last time I ran this there are 153 repos on GitHub only.

I'm claiming this task at a low priority as we still need the list of repos that are maintained on Phabricator only, but the scripts in my repo can likely (thorough minor modifications) answer that question.

thcipriani renamed this task from Create and maintain somehow a list of repos mastered in GitHub (and in Phabricator Diffusion) to Create and maintain a list of organization repos that are maintained on Gerrit, GitHub, and Diffusion.Tue, Jan 19, 5:36 PM
thcipriani updated the task description. (Show Details)
thcipriani closed this task as Resolved.Tue, Jan 19, 11:15 PM

TIL thanks to @mmodell phab has this list built in: https://phabricator.wikimedia.org/diffusion/query/ias.VEMtSCGU

This is repos that are not mirrored from another source in diffusion. There are 310 repos by my count as of 2021-01-19. Claiming victory!