Page MenuHomePhabricator

Import all gerrit.wikimedia.org repositories with Diffusion
Closed, ResolvedPublic

Description

One day, Diffusion will feature all the repositories available at https://gerrit.wikimedia.org today. There are different ways to do this, and we need to choose one.

  • When: We probably want to do this progressively as opposed to all repos at once. Starting with a selection of brave, non-mission-critical projects, and continuing moving repos until we are done.
  • How: is this task performed with manual work only or is there some software that we need to create or reuse?
  • Read-only vs Differential: do we tie new repos available with the code review migration, or is there a point in offering Diffusion to browse read-only repositories as well?

T560: Proof of concept of code review in Phabricator should feature a bunch of repos, but having all of tem is not required at all (this hasn't been discussed throughout, feedback welcome). However, by the time we deliver the PoC, we should have an agreed plan to execute this task.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Completed? I see over 800 repositories in diffusion, https://git.wikimedia.org/projects counts about 1200.

Completed? I see over 800 repositories in diffusion, https://git.wikimedia.org/projects counts about 1200.

Not quite. We've been having a minor issue importing a few, so they're on hold pending the next code pull.

Most repos are done now. Will tidy up the remaining ones.

This is basically almost all done.

1AhoCorasick
2at-ease
3integration/raita
4labs/tools/cdnjs-index
5labs/tools/crosswatch
6labs/tools/extreg-wos
7labs/tools/flow-oauth-demo
8labs/tools/forrestbot
9labs/tools/ptable
10labs/tools/translatemplate
11labs/tools/wikidipendenza
12labs/tools/wrcp
13mediawiki/core/vendor
14mediawiki/extensions/FlickrAPI
15mediawiki/extensions/ImportFreeImages
16mediawiki/extensions/LinkTarget
17mediawiki/extensions/MassMessageEmail
18mediawiki/extensions/OOJsUIAjaxLogin
19mediawiki/extensions/PhpTagsStorage
20mediawiki/extensions/Prezi
21mediawiki/extensions/SMWEnrich
22mediawiki/extensions/SemanticLinks
23mediawiki/extensions/ShoutWikiAPI
24mediawiki/extensions/SmiteSpam
25mediawiki/extensions/SpamDiffTool
26mediawiki/extensions/StaffPowers
27mediawiki/extensions/WikiPinger
28mediawiki/extensions/WikidataQualityConstraints
29mediawiki/extensions/WikidataQualityExternalValidation
30mediawiki/extensions/normal
31mediawiki/extensions/skins
32mediawiki/services/mathoid/deploy
33mediawiki/services/mobileapps
34mediawiki/services/mobileapps/deploy
35mediawiki/services/python-cache-relay
36mediawiki/tools/messdetector
37operations/debs/confd
38operations/debs/contenttranslation/apertium-ca-it
39operations/debs/contenttranslation/apertium-cy-en
40operations/debs/contenttranslation/apertium-eo-ca
41operations/debs/contenttranslation/apertium-eo-es
42operations/debs/contenttranslation/apertium-eo-fr
43operations/debs/contenttranslation/apertium-es-it
44operations/debs/contenttranslation/apertium-es-ro
45operations/debs/contenttranslation/apertium-is-sv
46operations/debs/contenttranslation/apertium-isl-eng
47operations/debs/contenttranslation/apertium-mlt-ara
48operations/debs/contenttranslation/apertium-sme-nob
49operations/debs/contenttranslation/python-toro
50operations/debs/dropwizard-metrics
51operations/debs/linux
52operations/debs/linux-meta
53operations/debs/python-etcd
54operations/debs/yammer-metrics
55operations/dumps/html
56operations/dumps/html/deploy
57operations/gerrit/plugins
58operations/puppet/mesos
59operations/software
60operations/software/blazegraph
61operations/software/brrd
62operations/software/coal
63operations/software/conftool
64operations/software/deployment/trebuchet-trigger
65operations/software/sentry
66operations/software/tendril
67operations/software/tools-checker
68operations/software/tools-manifest
69operations/software/tools-webservice
70phabricator/arcanist
71phabricator/libphutil
72phabricator/phabricator
73python-diamond
74rescue-pxe
75wikidata/query/blazegraph
76wikidata/query/rdf
77wikimedia/TransparencyReport-private
78wikimedia/bugzilla/modifications
79wikimedia/bugzilla/triagescripts
80wikimedia/bugzilla/wikibugs
81wikimedia/education
82wikimedia/fundraising/php-queue

hashar subscribed.

Seems more have been migrated since @demon last comment. Mind adding a status update?

How to keep up with new repositories?

Seems more have been migrated since @demon last comment. Mind adding a status update?

How to keep up with new repositories?

@demon: Could you answer that (if you know)? Or do you know who could answer that?

Could we please please have an update on this? The extension I'm currently writing as a GSoC project (phab: MediaWiki-extensions-SmiteSpam, gerrit: mediawiki/extensions/SmiteSpam) is on the list of pending imports to Diffusion and this makes writing things like T101095: Weekly reports for Extension:SmiteSpam very very annoying because I have to link commit hashes to gerrit manually. 😢

How to keep up with new repositories?

Change https://www.mediawiki.org/wiki/Gerrit/New_repositories to require this when creating new ones.

Our callsign naming conventions make no sense and limiting callsigns to 4 characters serves no purpose at all.

Our callsign naming conventions make no sense and limiting callsigns to 4 characters serves no purpose at all.

Related and maybe a better place for general discussions: T101358: Update {{git file}} to link to diffusion, T106130: Phund Phacility to rid us of callsigns

Our callsign naming conventions make no sense and limiting callsigns to 4 characters serves no purpose at all.

We're not debating it again. Please stick to the conventions no matter how dumb you may think them.

In T616#1481914, @demon wrote:

Our callsign naming conventions make no sense and limiting callsigns to 4 characters serves no purpose at all.

We're not debating it again. Please stick to the conventions no matter how dumb you may think them.

I wasn't included in the original debate about callsigns. I did stick to the 4 character rule, even though I disagree with it. I wasn't aware of the prefix standards, I actually can see some value in standardized prefixes.

@mmodell, thanks a lot for importing! You're awesome. :D

Any repo created after I mass-created them awhile back is likely to be missing. Individual tasks to import missing ones are...less than useful...as I prefer to do them in batches.

Is there a task to make diffusion automatically sync from gerrit/gitblit, or is that considered obvious as part of this task?

That doesn't block this. Repos can be imported to Diffusion without the template updates. In fact, the vast majority already have. The dependency should probably be between T108864 and T110607.

Any repo created after I mass-created them awhile back is likely to be missing. Individual tasks to import missing ones are...less than useful...as I prefer to do them in batches.

Is there a task to make diffusion automatically sync from gerrit/gitblit, or is that considered obvious as part of this task?

Obvious, I guess.

Any repo created after I mass-created them awhile back is likely to be missing. Individual tasks to import missing ones are...less than useful...as I prefer to do them in batches.

So you want endless additions to this page? Here's another:

Is there a task to make diffusion automatically sync from gerrit/gitblit, or is that considered obvious as part of this task?

??

How to keep up with new repositories?

Change https://www.mediawiki.org/wiki/Gerrit/New_repositories to require this when creating new ones.

Those are instructions to the owner of a project. Whoever responds to their request should do this diffusion and callsign work. I added https://www.mediawiki.org/wiki/Gerrit/New_repositories#Callsign , but it seems silly to have a step "After you request a repository with this form, make another request for it to be set up in diffusion as well."

In T616#1661452, @Spage wrote:

Any repo created after I mass-created them awhile back is likely to be missing. Individual tasks to import missing ones are...less than useful...as I prefer to do them in batches.

So you want endless additions to this page? Here's another:

No. I'm working on a script that will handle these automatically so nobody has to remind anyone about it.

Is there a task to make diffusion automatically sync from gerrit/gitblit, or is that considered obvious as part of this task?

??

See above.

How to keep up with new repositories?

Change https://www.mediawiki.org/wiki/Gerrit/New_repositories to require this when creating new ones.

Those are instructions to the owner of a project. Whoever responds to their request should do this diffusion and callsign work. I added https://www.mediawiki.org/wiki/Gerrit/New_repositories#Callsign , but it seems silly to have a step "After you request a repository with this form, make another request for it to be set up in diffusion as well."

The whole process on that page is already broke anyway. We need to rethink it.

@demon: it looks like upstream has a solution for us:

@epriestley wrote:
"Yeah, I'm planning to convert the "Clone As" setting to "optional, unique repository name" and use it as an identifier in URIs if it's set."

Maybe a note should be added to https://www.mediawiki.org/wiki/Git/New_repositories/Requests so that an admin knows to create the project in gerrit and import to phabricator.

Maybe a note should be added to https://www.mediawiki.org/wiki/Git/New_repositories/Requests so that an admin knows to create the project in gerrit and import to phabricator.

If you want, but @demon gets them whenever he does his latest batch of new imports (he does them in batches to save his time).

Seems that there only needs to be a few repos created in phabricator. I'm not sure how many and from now on when a repo is created in gerrit it should also be created in phabricator.

They're all imported.

If we create new repositories in Gerrit are they going to be automatically imported in Diffusion?

@hashar the admin who creates the repo has been asked to also create it in diffusion.

In T616#1604430, @demon wrote:

Is there a task to make diffusion automatically sync from gerrit/gitblit, or is that considered obvious as part of this task?

Obvious, I guess.

Hmm.