Page MenuHomePhabricator

Diffusion replacement for tarfile download from git.wikimedia.org
Closed, ResolvedPublic

Description

You can request from gitblit, e.g. https://git.wikimedia.org/zip/?r=mediawiki/extensions/SyntaxHighlight_GeSHi&h=refs/heads/master&format=gz . Template:WikimediaDownload uses this to provide a (Git master) link, and via {{Extension}} it's used in hundreds of extensions.

It seems Diffusion doesn't offer an equivalent, I couldn't see it at https://phabricator.wikimedia.org/diffusion/ESHG/browse/master/ .

Event Timeline

Spage created this task.Sep 9 2015, 1:28 AM
Spage raised the priority of this task from to Needs Triage.
Spage updated the task description. (Show Details)
Legoktm added a subscriber: Legoktm.Sep 9 2015, 1:40 AM

If these are extensions, can we link to ExtensionDistributor?

hashar set Security to None.
hashar removed a subscriber: hashar.
Restricted Application added a subscriber: scfc. · View Herald TranscriptSep 9 2015, 10:05 AM
hashar added a subscriber: hashar.Sep 9 2015, 10:06 AM

Moved out of Release-Engineering-Team since it is already in our project Gitblit-Deprecate. Added Phabricator since the request is related to Diffusion.

hashar removed a subscriber: hashar.Sep 9 2015, 10:06 AM
greg added a comment.Sep 9 2015, 5:14 PM

If these are extensions, can we link to ExtensionDistributor?

I don't see why not. Does that cover all of the use-case here? I'm explicitly not thinking about MW or the Android Wikipedia app, for instance, which have their own release/tarball process.

Does that cover all of the use-case here?

Doesn't gitblit allow to download any revision, branch or tag? ExtensionDistributor only does the RELX_Y branches.

Does that cover all of the use-case here?

Doesn't gitblit allow to download any revision, branch or tag? ExtensionDistributor only does the RELX_Y branches.

ED does REL* branches and master. I was only thinking of the "Git master" download link in the WikimediaDownload template.

greg added a comment.Sep 10 2015, 6:05 PM

Does that cover all of the use-case here?

Doesn't gitblit allow to download any revision, branch or tag? ExtensionDistributor only does the RELX_Y branches.

The question is: is that feature absolutely needed in Diffusion on day 1 (after deprecating gitblit)? It seems like ED addresses the main use case.

greg added a comment.Sep 10 2015, 6:06 PM

The bit from the duplicate task that isn't here is: apparently composer depends on this feature?

That was me being unspecific. No, composer does not currently depend on this being available from git.w.o. From my side this is not required on day 1.

greg triaged this task as Normal priority.Sep 10 2015, 8:14 PM
greg removed a project: Gitblit-Deprecate.

There are heaps of extensions around with individual versioning which prove ED being not a good choice for downloads. If it is not necessary from day one people should at least be somehow made aware that they should use the GitHub mirror for this

There are heaps of extensions around with individual versioning which prove ED being not a good choice for downloads. If it is not necessary from day one people should at least be somehow made aware that they should use the GitHub mirror for this

And the links should not break, so we need a task to redirect all /zip/ URLs to a sensible location from day 1.

greg added a comment.Sep 11 2015, 4:17 PM

There are heaps of extensions around with individual versioning which prove ED being not a good choice for downloads. If it is not necessary from day one people should at least be somehow made aware that they should use the GitHub mirror for this

what is the use case? which extensions?

There is no support for this in phabricator.

This needs to be filed as a feature request/bug at secure.phabricator.com

demon added a comment.Dec 18 2015, 5:43 PM

There is no support for this in phabricator.

Good. Let's keep it that way.

This needs to be filed as a feature request/bug at secure.phabricator.com

I've said it before and I'll say it again. Allowing users to generate zip/tar files for completely arbitrary sha1s is a bad idea and an obvious DOS vector that we've been hit with several times over the years. If Github wants to spend CPU cycles on such a feature then they're more than welcome. We should not be wasting upstream's time on implementing it or our servers serving them.

Ok well could there be at least a redirect script for when click a zip file it will download it from GitHub.

Ok well could there be at least a redirect script for when click a zip file it will download it from GitHub.

No we should not refer to GitHub for things that we are hosting on our own platform, that would be even worse.

Allowing users to generate zip/tar files for completely arbitrary sha1s is a bad idea and an obvious DOS vector

So is a red herring and rendering Wiki pages.

What is the exact feature required? (Nobody asked for completely aritrary sha1s.)

what is the use case? which extensions?

greg added a comment.Mar 4 2016, 8:36 PM

There are heaps of extensions around with individual versioning which prove ED being not a good choice for downloads. If it is not necessary from day one people should at least be somehow made aware that they should use the GitHub mirror for this

We've been asking a few times in this task for a use case, can you please give us a clear usecase from what you say here? A clear use case (that is in use ans sensible) for when ExtensionDistributor is not a good choice, that is. Thanks.

Maybe phabricator upstream could possible add support for either plugins or extensions which means that phabricator would be able to extend certain functionality without needing to maintain the plugin or extension. It will also allow us to develop features quicker.

There are heaps of extensions around with individual versioning which prove ED being not a good choice for downloads. If it is not necessary from day one people should at least be somehow made aware that they should use the GitHub mirror for this

We've been asking a few times in this task for a use case, can you please give us a clear usecase from what you say here? A clear use case (that is in use ans sensible) for when ExtensionDistributor is not a good choice, that is. Thanks.

User case use can be that not everyone wants to download from GitHub or use a git client to download. Instead being able to download from diffusion here, GitHub or through a git client will make everything easy. But also even going through GitHub, you have to search for it and there are probably billions of repos. Whereas in diffusion on Wikimedia we only have repos to do with Wikimedia so everything will be easier.

greg added a comment.Mar 4 2016, 8:45 PM

Paladox: you're missing the point, please read the whole task first. 1) Phabricator can be extended, that's not the point 2) of course people could want to do anything/download whatever, but I'm/we're asking for a use case for WHY and WHAT THEY GAIN that they don't from Extension Distributor.

@greg well what they wont get from that extension is that some repos are not an extension so that extension will not allow you to download those, and not all branches are shown.

greg added a comment.Mar 4 2016, 9:17 PM

@Paladox: I understand all of this but you aren't telling me anything unique and the question still stands: WHY?

Please, if you don't have a use case that you are actively using to get work done/administer a MW install, let those who might have a use case reply.

Krinkle removed a subscriber: Krinkle.Mar 5 2016, 12:59 AM
Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptMay 13 2016, 9:46 PM
Paladox added a comment.EditedMay 13 2016, 9:46 PM

The above revision uses zip instead of tar. It uses GitHub mirror to download from.

But it can only download branches.

Commits and tags are not supported yet but maybe in the future they will.

Ive updated https://phabricator.wikimedia.org/D233 now. It supports both zip and .tar.gz (gz) formats.

It also supports downloading commits which means it downloads the whole repo with the commit like you can at git.wikimedia.org.

Ive also added support for downloading tags.

The open changes will not work unless we switch to phabricator to do the mirroring since gerrit only mirrors refs/heads/ and refs/tags/ whereas phabricator mirrors refs/*

Nemo_bis removed a subscriber: Nemo_bis.May 14 2016, 3:18 PM
Paladox added a comment.EditedMay 15 2016, 1:45 PM

Ive added downloading support in https://phabricator.wikimedia.org/D233 but one problem with that is that when we switch to differential, differential only creates diffs, it dosent create the ref until it is merged which means downloading will not be supported on diffs created there.

  1. I think upstream should add some kind of support like gerrit to create the ref but not merge it into refs/heads so we can view the ref in diffusion and also view it on GitHub and download.
  1. Or upstream will need to add inbuilt downloading support in diffusion and differential.

I think we should go with 1 due to the fact it would make viewing commits easy since it will also be pushed to our GitHub mirror. And then over time go with 2. since 1 would have be implanted.

Currently we are using gerrit so this is not a problem yet but when we do switch to differential it will.

With https://phabricator.wikimedia.org/D234 it drops the requirement of renaming repos. All you have to do now is add a mirror and a download button will appear. This only supports GitHub links.

Paladox added a comment.EditedMay 24 2016, 7:21 AM

Hi What do people think of linking to GitHub for our downloads for now please. I've used GitHub to do https://phabricator.wikimedia.org/D234 which would make it easy for people to download without needing arc or without needing to go to GitHub and then download. It links directly to GitHub for downloads so we would like to know what everyone feels about us doing it this way please.

mmodell closed this task as Resolved.Jun 9 2016, 12:28 AM
mmodell assigned this task to Paladox.

Not quite fixed, we're still forcing people to use GitHub (advertised with links at the top right of e.g. https://phabricator.wikimedia.org/source/mediawiki/ ).