Page MenuHomePhabricator

Update {{git file}} to link to diffusion
Closed, ResolvedPublic

Event Timeline

Spage created this task.Jun 4 2015, 4:58 AM
Spage claimed this task.
Spage raised the priority of this task from to Normal.
Spage updated the task description. (Show Details)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 4 2015, 4:58 AM

@Negative24 started Template:Phab_file, once we get it working well enough it can become the new Template:Git_file on mediawiki.org and wikitech. It looks doable, just hundreds of lines in a {{#switch: {{{project}}} | ... }} ParserFunction to map project= paths to diffusion callsigns.

mmodell added a subscriber: mmodell.Jun 4 2015, 5:12 AM

It looks doable, just hundreds thousands of lines in a {{#switch: {{{project}}} | ... }} ParserFunction to map project= paths to diffusion callsigns.

ftfy

mmodell added a comment.EditedJun 4 2015, 5:20 AM

Because we have
Macro over9000: over 9000 git repositories

{P726, lines=15}
...
(obviously)

hashar added a subscriber: hashar.Jun 4 2015, 10:17 AM

@Negative24 started Template:Phab_file, once we get it working well enough it can become the new Template:Git_file on mediawiki.org and wikitech. It looks doable, just hundreds of lines in a {{#switch: {{{project}}} | ... }} ParserFunction to map project= paths to diffusion callsigns.

#switch is a terrible idea to store such map:

  • it is going through the Parser and its is quite slow
  • its going to be a pain in the ass to keep up-to-date
  • LUA module!!!

The Gerrit project name and Diffusion names do not match and are inconsistent (the last created project names in Diffusion have / as separator whereas it was a dash previously). Might have an impact as well.

What about having the Phabfile template to use the callsign? It is more or less meant to be an immutable human friendly ID and saves the crazy #switch.

So instead of:

{{phab file |project=mediawiki/extensions/Flow |file=composer.json }}

Do:

{{phab file |callsign=EFLW |file=composer.json }}

And the lookup map is gone.

Can we instead please teach diffusion to lookup repositories by name?

demon added a comment.Jun 4 2015, 2:50 PM

Can we instead please teach diffusion to lookup repositories by name?

I'd like that too, but two caveats:

  1. Old Gerrit names don't have an easy way to tell the corresponding Phab name...repo names are far more flexible.
  2. People can change their repo name in Phab (see #1), whereas callsigns are set in stone.

So I think having a way to find a repo based on repo name is nice, but can't be relied on 100% of the time. Getting things to speak callsigns makes that problem go away.

What about having the Phabfile template to use the callsign? It is more or less meant to be an immutable human friendly ID and saves the crazy #switch.

Callsigns are Phabricator-specific and must die.

Nemo_bis added a comment.EditedJun 4 2015, 5:03 PM

Getting things to speak callsigns

What things? Callsigns are not meant to be usable or used. Deprecating the "project" parameter of https://www.mediawiki.org/wiki/Template:Git_file would invalidate thousands of usages, it's way better (albeit still horrible) to have the template do the conversion.

(And yes, eventually as Ricordisamoa says Phabricator is supposed to become sane and get rid of callsigns.)

If we stored the mapping in phabricator then a bit of custom code could look up the phabricator callsign based on the gerrit project. We did something sorta similar for redirecting bugzilla urls to phabricator urls. We'd have to store the mapping _somewhere_ though and I don't know where exactly to put it. Also, over 9000 repositories.

demon added a comment.Jun 4 2015, 5:17 PM

If we stored the mapping in phabricator then a bit of custom code could look up the phabricator callsign based on the gerrit project. We did something sorta similar for redirecting bugzilla urls to phabricator urls. We'd have to store the mapping _somewhere_ though and I don't know where exactly to put it. Also, over 9000 repositories.

I was thinking of doing something along those lines. But remember, it'll only be for historical repos coming from Gerrit (of which we have 1320, lol). Since we have the upstream URLs already at repo creation time, it's easy to infer the repo name from there and store it. Then building redirector on top of that is super easy.

@demon: so we could make a url mapping in phabricator that was essentially https://phabricator.wikimedia.org/r/old/gerrit/path and route that to diffusion, then the wiki parser function could just replace gerrit.wikimedia.org/ with phabricator.wikimedia.org/ and be done with it.

demon added a comment.Jun 4 2015, 5:29 PM

What about having the Phabfile template to use the callsign? It is more or less meant to be an immutable human friendly ID and saves the crazy #switch.

Callsigns are Phabricator-specific and must die.

Immutable repo names are Gerrit specific and must die.

Seriously, repos have to have a way to be referred to that is immutable. Phab uses callsigns for this, not the repo name (which is more Github-esque, you can always rename a repo if the project changes). If we had been using callsigns all this time and were talking about changing to Gerrit, we'd be having the same argument in reverse.

As long as we have the proper tooling around redirecting old repo usages to the new ones, I fail to see the problem.

Getting things to speak callsigns

What things? Callsigns are not meant to be usable or used.

{{cn}} on not being usable or used.

Deprecating the "project" parameter of https://www.mediawiki.org/wiki/Template:Git_file would invalidate thousands of usages, it's way better (albeit still horrible) to have the template do the conversion.

I don't really care where the links get fixed, as long as they eventually end up in the right place (Diffusion). If we have them go through a redirection, that's cool, we need the redirection for Gitblit deprecation anyway.

(And yes, eventually as Ricordisamoa says Phabricator is supposed to become sane and get rid of callsigns.)

AIUI, they're going to be optional. For migration an existing system to Phabricator we'd still need them as we require a sane way to know a 1:1 mapping from old repo to new repo (and since repo names can change....)

demon added a comment.Jun 4 2015, 5:30 PM

@demon: so we could make a url mapping in phabricator that was essentially https://phabricator.wikimedia.org/r/old/gerrit/path and route that to diffusion, then the wiki parser function could just replace gerrit.wikimedia.org/ with phabricator.wikimedia.org/ and be done with it.

Something like that yeah. We'd need the same ability for deprecating Gitblit too.

{{cn}} on not being usable or used.

There was a wide consensus on http://markmail.org/message/jtdipertspvwdgpd

demon added a comment.Jun 4 2015, 6:24 PM

{{cn}} on not being usable or used.

There was a wide consensus on http://markmail.org/message/jtdipertspvwdgpd

Ok, on the usable bit. They are very much used.

The alternative to callsigns in phabricator will be to use an integer. R1, R2, ... R9001 see https://secure.phabricator.com/T4245#53345

What about having the Phabfile template to use the callsign?

Could do. I assume it's possible to use a bot to go through the ~1600 pages on mw.org (most of which are translations) to update {{git file}}.

If we maintain a mapping, then it should be output by a Lua module. Some other templates and bots will need it.

The key issue is what's the most "correct" and future-proof way to specify a repo for {{git file}}:

  1. project=mediawiki/extensions/Flow (current, a gerrit path)
  2. callsign=EFLW
  3. name=extension-Flow (the diffusion name in https://phabricator.wikimedia.org/diffusion/query/all/)

I don't know enough about diffusion and arcanist (but I'll blabber anyway :) )

(2) is obviously Phabricator-specific, but note (1) is not simply the project's git repo name, it's path/to/reponame. If and when we transition to arcanist and diffusion, will there still be a canonical path/to for each project, and where will it be shown? As hashar noted

The Gerrit project name and Diffusion names do not match and are inconsistent

In some cases you need to clone a repo in a certain directory structure (e.g. extensions/Flow or the old integration/jenkins/jjb/zuul ). But if the path/to in project= is arbitrary then it'll force people to invent arbitrary project path names that we have to add to a 1276-line mapping file, that others trying to make a {{git file}} link won't know.

So maybe (3) name=diffusion-name is the future, a loose mapping. Can you use that in a link to a git file, or will we have to maintain a mapping extension-Flow -> EFLW for it?

Ultimately everything is a naming problem :)

Spage removed Spage as the assignee of this task.Aug 3 2015, 9:45 PM

Unassigning myself. The Developer tools developers need to decide

what's the most "correct" and future-proof way to specify a repo for {{git file}}

and then it'll be apparent what to do with the template.

hashar reopened this task as Open.Aug 24 2015, 10:34 AM

Reopening.

A manually maintained list of call signs / project name at https://www.mediawiki.org/wiki/Module:Callsigns is prone to failure and maintenance nightmare.

See my comment above at T101358#1336865

What do you propose instead?

Your suggested syntax {{phab file |callsign=EFLW |file=composer.json }} is unwanted.

I have added the point to the Release-Engineering-Team weekly meeting agenda for August 25th.

We can generate the callsign mapping in phabricator. One option that @demon and I came up with is to offer a static url, something like https://phabricator.wikimedia.org/r/mediawiki/extensions/Flow which would simply redirect to https://phabricator.wikimedia.org/diffusion/EFLW

This way the mapping to callsigns is maintained in the repository configuration within phabricator and there wouldn't be a maintenance issue. Also this way existing templates would continue to work with minimal modification.

We can generate the callsign mapping in phabricator. One option that @demon and I came up with is to offer a static url, something like https://phabricator.wikimedia.org/r/mediawiki/extensions/Flow which would simply redirect to https://phabricator.wikimedia.org/diffusion/EFLW

I created T110607: redirect gerrit repo paths to diffusion callsigns. Great, so project=path/in/gerrit/to/reponame will continue as the way we specify a repo in templates. I think a project path makes sense, at least until we no longer use gerrit.

greg removed Nemo_bis as the assignee of this task.Sep 24 2015, 1:23 AM
greg removed a project: Release-Engineering-Team.
greg set Security to None.
Qgil removed a subscriber: Qgil.Sep 24 2015, 7:46 AM
greg moved this task from To Triage to Tooling on the Gerrit-Migration board.Dec 2 2015, 12:29 AM
greg moved this task from To Triage to Backlog on the Gitblit-Deprecate board.Dec 2 2015, 6:32 PM
Aklapper updated the task description. (Show Details)Feb 16 2016, 12:27 PM
Aklapper updated the task description. (Show Details)
demon added a comment.Mar 14 2016, 4:40 PM

{{git file}} is already using Diffusion, all the callsigns were updated in D139: Full dump of all existing callsign to repository mappings. What's left?

demon moved this task from Backlog to In Progress on the Gitblit-Deprecate board.Mar 14 2016, 4:41 PM
mmodell closed this task as Resolved.Mar 14 2016, 7:47 PM
mmodell claimed this task.
mmodell removed mmodell as the assignee of this task.

not sure who should claim this one

demon moved this task from In Progress to Done on the Gitblit-Deprecate board.Mar 14 2016, 8:13 PM
greg moved this task from Tooling to Done/Archive on the Gerrit-Migration board.Mar 15 2016, 9:28 PM
hashar reopened this task as Open.Mar 17 2016, 12:11 PM

https://www.mediawiki.org/wiki/Template:Git_file still invokes the the manually maintained mapping Module::Callsigns

https://phabricator.wikimedia.org/diffusion/{{#invoke:callsigns|repoToCallsign|{{{project|mediawiki/core}}}}}/

demon added a comment.Mar 17 2016, 2:23 PM

Then someone can copy over all the callsigns :)

Nemo_bis closed this task as Resolved.Mar 21 2016, 7:47 AM
Nemo_bis claimed this task.

The complete solution is tracked at T129447 / T108864. For {{git file}}, Module:Callsigns is a sufficient solution as the template is used with few repositories and usually always the same ones.