The submodule WikibaseLexeme included into the repository mediawiki/extensions is not available, making it unclonable (with submodule recursion).
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T84 Make sure anti-vandalism features are up to snuff | |||
| Restricted Task | |||||
| Restricted Task | |||||
| Stalled | BUG REPORT | None | T409519 WikibaseLexeme submodule is not available | ||
| Open | None | T374926 [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host |
Event Timeline
Comment Actions
What do you mean it's not available? Temporarily it failed? Permanently?
What error message?
What branch are you using?
[submodule "WikibaseLexeme"] path = WikibaseLexeme url = https://gerrit.wikimedia.org/r/mediawiki/extensions/WikibaseLexeme branch = .
Comment Actions
https://gerrit.wikimedia.org/r/mediawiki/extensions/WikibaseLexeme is not found.
/usr/bin/git pull --recurse-submodules is aborted on master branch.
Comment Actions
$ git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/WikibaseLexeme Cloning into 'WikibaseLexeme'... remote: Counting objects: 1221, done remote: Finding sources: 100% (18/18) remote: Getting sizes: 100% (13/13) remote: Compressing objects: 100% (667753/667753) remote: Total 40375 (delta 5), reused 40362 (delta 1) Receiving objects: 100% (40375/40375), 11.34 MiB | 9.22 MiB/s, done. Resolving deltas: 100% (31124/31124), done.
Comment Actions
[acko@foo ~]$ git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/WikibaseLexeme Cloning into 'WikibaseLexeme'... remote: Counting objects: 1221, done remote: Finding sources: 100% (18/18) remote: Getting sizes: 100% (13/13) remote: Compressing objects: 100% (667753/667753) remote: Total 40375 (delta 5), reused 40362 (delta 1) Receiving objects: 100% (40375/40375), 11.34 MiB | 6.45 MiB/s, done. Resolving deltas: 100% (31124/31124), done. [acko@foo ~]$
For future reference, please include clear steps to reproduce something somewhere with no space left for interpretation. Thanks.
Comment Actions
/usr/bin/git clone --recurse-submodules https://gerrit.wikimedia.org/r/mediawiki/extensions
… Cloning into '/var/mediawiki/extensions/WikibaseLexeme/resources/special/new-lexeme'... fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/': The requested URL returned error: 403 fatal: clone of 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git' into submodule path '/var/mediawiki/extensions/WikibaseLexeme/resources/special/new-lexeme' failed Failed to clone 'resources/special/new-lexeme'. Retry scheduled Cloning into '/var/mediawiki/extensions/WikibaseLexeme/resources/special/new-lexeme'... fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/': The requested URL returned error: 403 fatal: clone of 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git' into submodule path '/var/mediawiki/extensions/WikibaseLexeme/resources/special/new-lexeme' failed Failed to clone 'resources/special/new-lexeme' a second time, aborting fatal: Failed to recurse into submodule path 'WikibaseLexeme'
Comment Actions
https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git is not cloneable, because it requires user login to Phabricator to access:
Login Required You must log in to take this action.
Should a bug with WikibaseLexeme be registered instead?
Comment Actions
Ah, so ext:WikibaseLexeme's .gitmodules points to Diffusion:
[submodule "resources/special/new-lexeme"] path = resources/special/new-lexeme url = https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git
This feels like maybe T374926: [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host, but combined with the effects of the latter part of T408956#11341971 (non-public) or something similar.
Comment Actions
Ah, thanks.
[acko@foo ~]$ git clone https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git Cloning into 'new-lexeme-special-page'... fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/': The requested URL returned error: 403
That is interesting, because the Phabricator Diffusion management UI says that this URI should allow read only access:
Comment Actions
That is a side effect of T409175: Diffusion code view: Hide last change date for anonymous users && display login dialog on every page, and I would naïvely expect that log-in requirement to only affect the web interface (maybe I am wrong though).
Are you sure that there is a relation between git clone failing on the command line for code checkout, and the web interface in a browser asking you to log in for code browsing?
Comment Actions
Are you sure that there is a relation between git clone failing on the command line for code checkout, and the web interface in a browser asking you to log in for code browsing?
I would naïvely presume that code checkout is a kind of code browsing. But the question is, rather, to the admins of that server.
Comment Actions
(Adding tags for visibility, though I assume this needs to be either addressed on the Phabricator side, or resolved as part of T372458 / T374926 like @A_smart_kitten mentioned.)
Comment Actions
This seems to depend on the path. /source/ works (that's a 404 on web, always been), /diffusion/ does not (that's the new login dialog on web)?:
[acko@foo ~]$ git clone https://phabricator.wikimedia.org/diffusion/MW/mediawiki.git Cloning into 'mediawiki'... fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/MW/mediawiki.git/': The requested URL returned error: 403 [acko@foo ~]$ git clone https://phabricator.wikimedia.org/source/mediawiki.git Cloning into 'mediawiki'... [...] Updating files: 100% (12077/12077), done. [acko@foo ~]$
My theory is that this could be related to the path handling in PhabricatorDiffusionApplication::getRoutes(). Which I hadn't expected.
I'd prefer to revert T409175: Diffusion code view: Hide last change date for anonymous users && display login dialog on every page for now in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/107
Comment Actions
$ GIT_TRACE_CURL=1 git clone https://phabricator.wikimedia.org/diffusion/MW/mediawiki.git Cloning into 'mediawiki'... [...] Recv data: <!DOCTYPE html>.<html lang="en">.<meta charset="utf-8">.<tit Recv data: le>Wikimedia Error</title>.<style>.* { margin: 0; padding: 0 [...] Recv data: class="content-text">.<h1>Error</h1>..<p>Our servers are cu Recv data: rrently under maintenance or experiencing a technical issue< Recv data: /p>.</div>.</div>.<div class="footer"><p>If you report this Recv data: error to the Wikimedia System Administrators, please include Recv data: the details below.</p><p class="text-muted"><code>Request s [...] fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/MW/mediawiki.git/': The requested URL returned error: 403
Maybe I'm missing something, but this seems like a block from the SRE side of things (rather than from the Phab code)? That's why I assumed it might be related to the change mentioned in the latter part of T408956#11341971.
Comment Actions
Uh, thanks! Let's CC collaboration-services for potential input. I'm especially curious about the /diffusion/ vs /source/ URI differences in behavior.
Comment Actions
That is caused by a system in front of Phabricator which prevents crawlers browsing Diffusion (as I described at T408956#11335166). That was put in place early this week. I missed we had some repositories being cloned from Phabricator and those should be exempt, notably git-upload-pack service should always be allowed:
GIT_CURL_VERBOSE=2 git clone https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/ |& grep GET 18:13:39.378185 http.c:913 == Info: [HTTP/2] [1] [:method: GET] 18:13:39.378235 http.c:872 => Send header: GET /diffusion/NLSP/new-lexeme-special-page.git/info/refs?service=git-upload-pack HTTP/2
Comment Actions
Indeed there is nothing to fix or change on a configuration level of Diffusion or Phabricator itself.
Maybe maybe some day anti-abuse measures in requestctl could be loosened again, however currently I consider overall availability of a service like Phabricator more important than individual repository cloning via Diffusion.
The proper fix for the problem described in this task is to have the Wikibase team fix T374926: [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host.
Comment Actions
Stalling this task until we get a response to T374926: [EPIC][Infra] Move Wikibase and WikibaseLexeme Git submodules to suitable Git host
Comment Actions
I'm also getting people on mediawiki.org support complaining about this btw.
If you run into this problem and need an immediate workaround, you can probably configure your git client to remap the url with something like:
git config --global url.https://github.com/wmde/WikibaseDataModel.git.insteadof https://phabricator.wikimedia.org/source/wikibase-data-model.git git config --global url.https://github.com/wmde/DataValuesJavaScript.git.insteadof https://phabricator.wikimedia.org/source/datavalues-javascript.git git config --global url.https://github.com/wmde/WikibaseSerializationJavaScript.git.insteadof https://phabricator.wikimedia.org/source/wikibase-serialization.git
Which will be persisted in your .gitconfig as
[url "https://github.com/wmde/WikibaseDataModelJavascript.git"]
insteadof = https://phabricator.wikimedia.org/source/wikibase-data-model.git
[url "https://github.com/wmde/DataValuesJavaScript.git"]
insteadof = https://phabricator.wikimedia.org/source/datavalues-javascript.git
[url "https://github.com/wmde/WikibaseSerializationJavaScript.git"]
insteadof = https://phabricator.wikimedia.org/source/wikibase-serialization.gitThis begs the question.. maybe we should do this in the opposite direction for wikimedia production, thus immediately having everyone on the GitHub urls, and only WMF replacing the urls for their situation ? Should be easy to have that controlled in puppet I think ?
Comment Actions
I'm currently blocked from updating my train-dev development environment because I can't clone https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git from offsite.
dancy@base:/train-dev/scap$ git clone https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git /tmp/xxx Cloning into '/tmp/xxx'... fatal: unable to access 'https://phabricator.wikimedia.org/diffusion/NLSP/new-lexeme-special-page.git/': The requested URL returned error: 403