Page MenuHomePhabricator

ORES deployment submodules should point to phabricator HTTPS repos.
Closed, DeclinedPublic

Description

Currently, we point to git-ssh, which is overkill since we only read from these repos, and furthermore our team doesn't work on submodules in place, but in isolation.

We should change to use submodule URLs like https://phabricator.wikimedia.org/source/draftquality.git

Details

Related Gerrit Patches:
mediawiki/services/ores/deploy : masterSwitch to using HTTPS for submodules

Event Timeline

awight created this task.Oct 25 2017, 3:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 25 2017, 3:16 PM
awight added a subscriber: demon.Oct 25 2017, 3:22 PM

@demon If you would like to bless this change, it would give us more confidence ;-)

demon awarded a token.Oct 25 2017, 3:23 PM

Change 386407 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] Switch to using HTTPS for submodules

https://gerrit.wikimedia.org/r/386407

Change 386407 merged by Ladsgroup:
[mediawiki/services/ores/deploy@master] Switch to using HTTPS for submodules

https://gerrit.wikimedia.org/r/386407

Halfak added a subscriber: Halfak.Oct 26 2017, 9:49 PM

We used to point to the https in phabricator explodes when you try to pull from our giant repos.

We need to fix that. SSH doesn't scale indefinitely.

mmodell added a subscriber: mmodell.EditedOct 27 2017, 6:06 PM

@demon: fixing phabricator is going to be difficult. Git was never meant to be so huge, and phabricator's git-http functionality is badly broken for huge repos in ways which are complex (at best) to address.

demon added a comment.Oct 27 2017, 8:35 PM

I mean we need to fix deploying large blobs (cf: git-lfs, git-fat, etc tasks)

Now I'm wondering if Phabricator has the same bug with it's own git-lfs support (file size limit due to large HTTP transactions buffering in the web server and/or PHP)

This may have been a bad decision. I get a lot of timeouts trying to clone our submodules onto deployment-tin and locally.

Yep, this became an issue again. I can't make fresh checkouts of the ores-deploy repo because of timeouts:

Cloning into '/home/awight/ores-prod-deploy/submodules/articlequality'...
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly
fatal: clone of 'https://phabricator.wikimedia.org/source/wikiclass.git' into submodule path '/home/awight/ores-prod-deploy/submodules/articlequality' failed
Failed to clone 'submodules/articlequality'. Retry scheduled
Cloning into '/home/awight/ores-prod-deploy/submodules/draftquality'...
Cloning into '/home/awight/ores-prod-deploy/submodules/editquality'...
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly
fatal: clone of 'https://phabricator.wikimedia.org/source/editquality.git' into submodule path '/home/awight/ores-prod-deploy/submodules/editquality' failed

Just noticed that we've already rolled this back for git-lfs work, we're now using gerrit again. Since I created the task originally, and my motivation was murky, this might be fine.

awight closed this task as Declined.Apr 30 2018, 9:22 PM

So, we're using Gerrit's git-lfs now?

@demon: At least for now, since the swift backend I've been working on for phabricator's git-lfs storage still isn't finished.