Description

(In the context of T160430)

Wikimedia has some Wikimedia's "GitHub-only" code repositories.
Find out how to differentiate (exclude) those repositories that are mirrors-only (maybe there no trivial way).
Also, open question: What about stuff that Wikimedia forked? Exclude or not? (Similar problem with measuring activity in pulled upstream repos in Gerrit)

Event Timeline

Aklapper created this task.Apr 21 2017, 7:08 PM
Aklapper claimed this task.Apr 21 2017, 7:35 PM
Aklapper updated the task description. (Show Details)May 2 2017, 11:51 AM

how to differentiate (exclude) those repositories that are mirrors-only

Link header response for WM repos on GitHub says &page=1964>; rel="last" so there are 1964 repos.
For a comparison of numbers:

$:acko\> ssh aklapper@gerrit.wikimedia.org -p 29418 gerrit ls-projects | wc -l 1736 Trying to somehow get a complete list of repos that are only mirrors there seems to be no one approach. Different spellings; only some mirrored repos have set the homepage key:$:acko\> wget -q "https://api.github.com/orgs/wikimedia/repos?page=XY&per_page=100" -O fooXY.json
$:acko\> grep -r "Github mirror" . | wc -l 1594$:acko\> grep -r "GitHub mirror" . | wc -l
2
$:acko\> grep -r "actual code is hosted" . | wc -l 1595$:acko\> grep -r "\"homepage\": \"https://gerrit.wikimedia.org" . | wc -l
575

Also some repos have no description at all:

$:acko\> grep -r "\"description\": null" . | wc -l 68$:acko\> cat fooXY.json | jq '.[] | select(.description == null) | .name'       ## list GitHub repos with empty description
Aklapper updated the task description. (Show Details)May 3 2017, 9:54 AM

Chad was kind enough to point me to:

So now there does not seem to be an "easy" way.

Qgil added a subscriber: Qgil.May 4 2017, 10:53 AM

In practical terms, to me this task should be block by another one "Create a list of Featured Projects for new developers" and see whether we have any GitHub only projects in that list.

If we have any GitHub only projects, then we can see whether having an export/mirror in Gerrit makes sense. If not, then we can check the metrics problem again.

In other words, I think putting time on this task before having Featured Projects is not a good use of time.

• Created list of GitHub repositories (output from cat foo01.json | jq -r '.[] | .full_name > github.list' after concatenation)
• Created list of Gerrit repositories (output from ssh aklapper@gerrit.wikimedia.org -p 29418 gerrit ls-projects > gerrit.list)
• Stripped wikimedia/ prefix in github.list
• Replaced / by - in gerrit.list
• Sort entries in both files alphabetically
• Ran diff and only show changed lines (via grep and nothing else)
• - means in Gerrit only, + means in GitHub only:

1 $:acko\> diff -dipu0 gerrit.list github.list | egrep '^[+-]' --- gerrit.list 2017-05-04 12:55:01.216117052 +0200 +++ github.list 2017-05-04 12:43:44.289274339 +0200 -3d2png-deploy +AFNetworking -All-Projects-In-Phabricator -All-Users -analytics-analytics.wikimedia.org +analytics-editor-geocoding +analytics-fundraising +analytics-fundraising-dashboard -analytics-kraken +analytics-quarry-puppet +analytics-query-service -analytics-websites_maintenance -analytics-wmde +analytics.wikimedia.org +android-mwlogin +AnimatedGIFImageSerialization +ansible-deploy +aosp-morelangs-ime +aosp-morelangs-ime-dictionaries +apps-android-commons -apps-ios-wikipedia +arc-lamp +arcanist +Assert +authoid +bingle +BlocksKit +bunyan-syslog-udp +camus +cassandra-codec +cassandra-metrics-collector +change-propagation +ChromeWikimediaDebug +citoid +citolytics -css-sanitizer +CocoaLumberjack +community-tech-tools +composer-merge-plugin +content-type +CopyPatrol +Cyberbot_II +data-warehouse +dClass +DeadlinkChecker +dump-scheduler-eval +dumpgrepper +eslint-config-node-services +eslint-config-wikimedia +extensions-Limn +FirefoxWikimediaDebug -gerrit +git-client-plugin +git-fat -graphs-shared -HtmlFormatter +grunt-banana-checker +grunt-stylelint +grunt-tyops +hpple +htcp-purge +html-formatter +html-metadata +html5depurate +htmldumper +hyperswitch +ifttt +incubator-cordova-android +incubator-cordova-ios +incubator-cordova-js -integration-gerrit-commit-message-validator -integration-phantomjs -integration-phpcs +java-morelangs +java-mwapi +jquery-client +jquery-tipsy +jquery.i18n +jquery.ime +jquery.uls +jquery.webfonts +jscs-preset-wikimedia +json-stable-stringify +KafkaSSE +kasocki +kraken +kraken-puppet -labs-incubator -labs-tools-quarrybot-enwiki -labs-tools-stashbot -labs-tools-Wikimedia-Emoji-Bot -labs-tools-wikiviewstats -mapdata +lcm-dashboard +limitation +limn +limn-data +limn-debugging-data +limn-deploy +limn-editor-engagement-data +limn-fundraising-data +limnpy +malu -maps-ClearTables -maps-meddo +MathJax +MathJax-node +mathoid -mediawiki-core +mediawiki-bots +mediawiki-bots-PHPWikiBot +mediawiki-containers +mediawiki-core-vendor +mediawiki-docker -mediawiki-extensions-3D +mediawiki-extensions-2ColConflict +mediawiki-extensions-AddMetaAndTitle +mediawiki-extensions-AmazonLookup -mediawiki-extensions-AutoGallery -mediawiki-extensions-BlueSpiceExtendedFilelist -mediawiki-extensions-BlueSpiceInsertTemplate -mediawiki-extensions-BlueSpiceMultiUpload -mediawiki-extensions-BlueSpiceSignHere -mediawiki-extensions-BlueSpiceSubPageTree +mediawiki-extensions-BSExtendedSearch +mediawiki-extensions-CategoryMagicWords -mediawiki-extensions-CategoryWatch +mediawiki-extensions-Censor +mediawiki-extensions-CentralNotice-BannerProxy +mediawiki-extensions-CloudSearch +mediawiki-extensions-ContextComments -mediawiki-extensions-CreatePageUw +mediawiki-extensions-CustomMagic +mediawiki-extensions-DataModel -mediawiki-extensions-DebateTree +mediawiki-extensions-DebianISOCodes +mediawiki-extensions-DetectLanguage +mediawiki-extensions-DisplayTitle +mediawiki-extensions-EImage +mediawiki-extensions-ELearnware +mediawiki-extensions-ExtensionStatus -mediawiki-extensions-FileExporter -mediawiki-extensions-FileImporter +mediawiki-extensions-FlashPlayer +mediawiki-extensions-GoogleTagManager +mediawiki-extensions-HaloTripleStoreConnector +mediawiki-extensions-HotCat -mediawiki-extensions-Ids -mediawiki-extensions-ImageRating +mediawiki-extensions-ImportBibliography +mediawiki-extensions-InterwikiExistence +mediawiki-extensions-Ipernity +mediawiki-extensions-Isbn +mediawiki-extensions-ISO3166 +mediawiki-extensions-ISO639 +mediawiki-extensions-JsonData-JsonSchema +mediawiki-extensions-KeepSearches -mediawiki-extensions-LdapGroups +mediawiki-extensions-LightboxGallery +mediawiki-extensions-ListTransclusions -mediawiki-extensions-MagicNumberedHeadings +mediawiki-extensions-MailChimpSubscription -mediawiki-extensions-MessageCommons +mediawiki-extensions-MetaDescriptionTag +mediawiki-extensions-MirrorTools +mediawiki-extensions-MobileSections -mediawiki-extensions-MoveToCommons -mediawiki-extensions-MoveToCommonsClient +mediawiki-extensions-MultiAudioVideo -mediawiki-extensions-NamespacePopups +mediawiki-extensions-Notifications +mediawiki-extensions-NoUnwrap -mediawiki-extensions-OrphanedTalkPages +mediawiki-extensions-PageCredits -mediawiki-extensions-PageLanguageApi -mediawiki-extensions-PageNameFormula -mediawiki-extensions-PagePopups +mediawiki-extensions-PDBHandler +mediawiki-extensions-PerformanceMonitor +mediawiki-extensions-PhpTagsDebugger -mediawiki-extensions-Pickle -mediawiki-extensions-PlanOut +mediawiki-extensions-PipVideoJs +mediawiki-extensions-PlaceNewSection +mediawiki-extensions-PrefixExport +mediawiki-extensions-PropertySuggester +mediawiki-extensions-ProtectedTitles +mediawiki-extensions-PubSubHubbubSubscriber +mediawiki-extensions-PurposeCentricSearch +mediawiki-extensions-QueryResult +mediawiki-extensions-RawImageHandler +mediawiki-extensions-RealNames +mediawiki-extensions-SemanticDummyEditor +mediawiki-extensions-ShareThisWidget +mediawiki-extensions-ShortUrlApi -mediawiki-extensions-ShowMe +mediawiki-extensions-ShrinkTheWeb +mediawiki-extensions-SignupAPI +mediawiki-extensions-SimpleSamlAuth +mediawiki-extensions-SMWEnrich +mediawiki-extensions-SMWHalo +mediawiki-extensions-SOLRSearch +mediawiki-extensions-StoryParagraph +mediawiki-extensions-SubpageWatchlist +mediawiki-extensions-SwedishCollation +mediawiki-extensions-TemplateDocumentation -mediawiki-extensions-TopLists +mediawiki-extensions-VersionView +mediawiki-extensions-ViewportMetrics +mediawiki-extensions-VisualWiki +mediawiki-extensions-WhatsNearby +mediawiki-extensions-WhichImageIsBetter +mediawiki-extensions-WikibaseClient +mediawiki-extensions-WikibaseLib +mediawiki-extensions-WikibaseView +mediawiki-extensions-WikiCortex +mediawiki-extensions-WikiEduDashboard +mediawiki-extensions-WikiFarm -mediawiki-extensions-WikimediaPageViewInfo +mediawiki-extensions-WikivotePageSchemas +mediawiki-extensions-WikivoyageMessages -mediawiki-libs -mediawiki-libs-Assert -mediawiki-libs-etcd -mediawiki-libs-RemexHtml -mediawiki-libs-ScopedCallback -mediawiki-libs-Timestamp -mediawiki-libs-WaitConditionLoop +mediawiki-libs-FileOgg +mediawiki-node-services -mediawiki-php-FastStringSearch -mediawiki-php-wikidiff -mediawiki-services-citoid -mediawiki-services-eventstreams-deploy -mediawiki-services-html5depurate -mediawiki-services-mathoid -mediawiki-services-ores-editquality -mediawiki-services-ores-wikiclass -mediawiki-services-parsoid +mediawiki-services-zotero -mediawiki-skins-Athena -mediawiki-skins-Material -mediawiki-skins-Poncho +mediawiki-title -mediawiki-tools-git-remote -mediawiki-tools-scap +metrics +MGSwipeTableCell -nfsd +nagf +node-rcstream +node-rdkafka +node-serviceworker +node-serviceworker-proxy +node-txstatsd +nodejs-driver +NSDate-Extensions +OAStackView +officeit-puppet -oojs-core +oojs-router -operations-calico-cni -operations-calico-k8s-policy-controller +operations-debs-apertium +operations-debs-cg3 +operations-debs-contenttranslation-apertium-af-nk +operations-debs-contenttranslation-apertium-api -operations-debs-contenttranslation-apertium-arg -operations-debs-contenttranslation-apertium-arg-cat -operations-debs-contenttranslation-apertium-cat -operations-debs-contenttranslation-apertium-fra -operations-debs-contenttranslation-apertium-fra-cat +operations-debs-contenttranslation-apertium-hi-ur -operations-debs-contenttranslation-apertium-spa -operations-debs-contenttranslation-apertium-spa-arg -operations-debs-contenttranslation-apertium-spa-cat -operations-debs-contenttranslation-apertium-swe-dan -operations-debs-contenttranslation-apertium-swe-nor -operations-debs-contenttranslation-foma -operations-debs-contenttranslation-hfst-ospell +operations-debs-dropwizard-metrics +operations-debs-ffmpeg2theorawmf -operations-debs-geckodriver -operations-debs-libav +operations-debs-lttoolbox -operations-debs-mcrouter -operations-debs-mtail -operations-debs-openssl11 -operations-debs-osmborder -operations-debs-phantomjs -operations-debs-prometheus-apache-exporter -operations-debs-prometheus-memcached-exporter -operations-debs-prometheus-redis-exporter -operations-debs-prometheus-snmp-exporter -operations-debs-pybal -operations-debs-pykube -operations-debs-python-confluent-kafka -operations-debs-python-mmh3 -operations-debs-python-sprockets -operations-debs-python-sprockets-clients-statsd +operations-debs-python-statsd -operations-debs-python-ua-parser -operations-debs-StatsD +operations-debs-stud +operations-debs-txstatsd +operations-debs-yammer-metrics +operations-deployment -operations-docker-images +operations-docker-images-toolabs-images -operations-dumps-import-tools -operations-dumps-statusapi -operations-gerrit-plugins -operations-mediawiki-config-fonts -operations-mediawiki-multiversion -operations-puppet -operations-puppet-cdh +operations-puppet-cassandra -operations-puppet-jmxtrans -operations-puppet-kafka -operations-puppet-modules -operations-puppet-zookeeper -operations-software-cumin -operations-software-etcd-mirror +operations-software-grafana -operations-software-hhvm_exporter -operations-software-hhvm-dev +operations-software-mwprof +operations-software-mwprof-reporter -operations-software-prometheus_jmx_exporter -operations-software-statsdlb -operations-software-varnish-libvmod-header -operations-software-varnish-varnishkafka -operations-software-varnish-varnishkafka-testing -operations-software-xhprof -operations-switchdc -operations-wheels-paws-internal +parsoid +parsoid-dom-utils +peformance-docroot -performance-visualmetrics-docker -performance-WebPageTest -phabricator-arcanist -phabricator-phabricator +phantomjs +phlogiston +php-ffs +php-gpglib +piwik-sdk-ios +portals +preq +puppet +puppet-cdh +puppet-jmxtrans +puppet-kafka +puppet-kafka-0.7.2 +puppet-storm +puppet-zookeeper +PyBal +python-diamond +pywikiapi +pywikibot-externals-six -rcstream +rescue-pxe -research-ores -research-ores-deploy -research-recommendation-api-scap +restbase +restbase-mod-table-cassandra +restbase-mod-table-spec +restbase-mod-table-sqlite +restevent +riemann-jmx +routeswitch +scap +ScopedCallback -search-ltr -search-MjoLniR -secrets +SelfSizingWaterfallCollectionViewLayout +service-runner +service-template-node +simplei18n -subversion-svn.wikimedia.org-mediawiki-trunk +sqoopy +SSDataSources +stashbot +stylelint-config-wikimedia +subversion +subversion-svn.wikimedia.org +subversion-svn.wikimedia.org-mediawiki +subversion-svn.wikimedia.org-trunk +swagger-js +swagger-router +swagger-ui +SWStepSlider +template-expression-compiler -testing-access-wrapper +testreduce +testrepo +texvcjs +thumbor +thumbor-base-engine +thumbor-conditional-sharpen +thumbor-djvu-engine +thumbor-exif-optimizer +thumbor-ghostscript-engine +thumbor-multi-handler +thumbor-page +thumbor-proxy-engine +thumbor-proxy-loader +thumbor-purger +thumbor-request-storage +thumbor-result-storage +thumbor-svg-engine +thumbor-tiff-engine +thumbor-video-engine +thumbor-video-loader +thumbor-vips-engine +thumbor-xcf-engine +Timestamp +tool-gridengine-status +TSMessages +TUSafariActivity +twcs +Tweaks +umapi_client +user_metrics +uxprototypes +varnishkafka +ve-dirtydiffbot +ve-needcheck-reporter-bot -VisualEditor-VisualEditor +visualmetrics-docker +WaitConditionLoop +web-html-stream +web-stream-util +Wiki-Class +WikiFont +wikimedia-analytics-wikimetrics-deploy -wikimedia-communications-WP-Victor -wikimedia-fundraising-civicrm +wikimedia-fundraising-dashboard -wikimedia-fundraising-process-control +wikimedia-IPSet +wikimedia-logo -wikimedia-portals -wikimedia-textcat-demo +wikimedia-thumbor-djvu-engine +wikimedia-thumbor-ghostscript-engine +wikimedia-thumbor-multi-handler +wikimedia-thumbor-page +wikimedia-thumbor-proxy-loader +wikimedia-thumbor-request-storage +wikimedia-thumbor-tiff-engine +wikimedia-thumbor-video-loader +wikimedia-thumbor-xcf-engine +wikimedia-whatcanidoforwikipedia -wikimedia-WikimediaShopTools -wikimedia-wlm-api +wikimedia.github.io +wikimediablog-wordpresscom +WikimediaUI-base +WikimediaUI-Style-Guide -wikipedia-gadgets +wikipedia-ios +wikipedia-iphone +WikipediaMobileJ2ME +WikisourceMobile -winter +WiktionaryMobile +WLMMobile +wpt-reporter +www.wikipedia.org +xhprof +YapDatabase +zotero$:acko\>

In other words, I think putting time on this task before having Featured Projects is not a good use of time.

True that. My curiosity was too strong though. :)

Qgil added a comment.May 4 2017, 11:26 AM

"Not a good use of time" is not an appropriate expression for this task. My apologies! :)

I just wanted to clarify how much we need this answer and under which circumstances. Regardless, I am reading with curiosity too. It is an interesting problem.

Aklapper closed this task as Declined.Jun 30 2017, 5:18 PM

For my records, Mukunda pointed out that mirrored Gerrit projects are listed on https://phabricator.wikimedia.org/r/

I'm going to decline this task for the time being. ("decline" because of the "how" in the summary. Could also be "resolved" because of the "whether" which is "no").
While I found out a few interesting things this task won't move forward due to the current setup. See the dependency tasks in T163576#3232491 which would need to get fixed first to get a basic grip here to move forward.
This task can always be reopened once it's less of a mess and requires less complex manual work to get a basic grip.

Tgr added a subscriber: Tgr.Feb 21 2018, 9:35 PM

Would T109939 really be the easy way here? It seems very fragile. Surely reading the master configuration from Phabricator or gerrit or whatever does the mirroring is superior?

Would T109939 really be the easy way here? It seems very fragile. Surely reading the master configuration from Phabricator or gerrit or whatever does the mirroring is superior?

@Tgr: "whatever does the mirroring": If I understand correctly, replication from Gerrit to Github is done by a Gerrit "replication" plugin. wikitech:Gerrit implies that plugin is com.googlesource.gerrit.plugins.replication (upstream code location). Which brought me to T109939...