scap git LFS only works for submodules
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Dec 7 2022, 1:17 PM

Description

Found when working with @jnuche on adding git LFS to a repository deployed by scap. It does not quite work:

P42441 scap3-dev jenkins-deploy and LFS

1	deploy-local failed: <FailedCommand> {'exitcode': 128, 'stdout': '', 'stderr': 'Cloning into \'/srv/deployment/jenkins-deploy-cache/revs/8f082e7856866ec760882f8ac3021da73fe37f1a\'...
2	done.
3
4	hint: The remote resolves to a file:// URL, which can only work with a
5	hint: standalone transfer agent. See section "Using a Custom Transfer Type
6	hint: without the API server" in custom-transfers.md for details.
7	Downloading jenkins-plugin-manager-2.12.9.jar (6.3 MB)
8
9	hint: The remote resolves to a file:// URL, which can only work with a
10	hint: standalone transfer agent. See section "Using a Custom Transfer Type
11	hint: without the API server" in custom-transfers.md for details.
12	Error downloading object: jenkins-plugin-manager-2.12.9.jar (5fa0751): Smudge error: Error downloading jenkins-plugin-manager-2.12.9.jar (5fa0751bfe9d3a4b7f6b25364670eeab990aa65c1e720d08745f09b73b5ccdba): batch request: missing protocol: "file:///srv/deployment/jenkins-deploy-cache/cache/.git/info/lfs"
13
14	Errors logged to /srv/deployment/jenkins-deploy-cache/revs/8f082e7856866ec760882f8ac3021da73fe37f1a/.git/lfs/logs/20221207T100058.363353208.log
15	Use `git lfs logs last` to view the log.
16	error: external filter \'git-lfs filter-process\' failed
17	fatal: jenkins-plugin-manager-2.12.9.jar: smudge filter lfs failed
18	warning: Clone succeeded, but checkout failed.
19	You can inspect what was checked out with \'git status\'
20	and retry the checkout with \'git checkout -f HEAD\'
21
22	'}

The reason is that on the targets the cached revision remote is set to point to the local cache using file://. That is not supported by Git LFS 2.7.1 (but is by 2.10).

The cache itself points back to the deployment server using http:// and it is unclear to me whether LFS would work this way.

Related Objects

Mentioned In: T279509: git-fat replacement/removal
Mentioned Here: P42441 scap3-dev jenkins-deploy and LFS

Event Timeline

hashar created this task.Dec 7 2022, 1:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 7 2022, 1:17 PM

For ORES.

On the deployment server

There is no .lsconfig. Git LFS is 2.7.1

$ git -C /srv/deployment/ores/deploy lfs env|grep Endpoint
Endpoint=https://gerrit.wikimedia.org/r/p/mediawiki/services/ores/deploy.git/info/lfs (auth=none)

It comes from the remote URL https://gerrit.wikimedia.org/r/p/mediawiki/services/ores/deploy.git .

On one of the target (ores1001.eqiad.wmnet)

$ git -C /srv/deployment/ores/deploy lfs env|grep End
Endpoint=file:///srv/deployment/ores/deploy-cache/cache/.git/info/lfs (auth=none)

$ git -C /srv/deployment/ores/deploy-cache/cache lfs env|grep End
Endpoint=http://deploy1002.eqiad.wmnet/ores/deploy/.git/info/lfs (auth=none)

The file:// scheme in the end point of the first repository is not recognized by Git LFS 2.7.1. And even if it was the cache would need the LFS object to be fetched in.

Why does it work with ORES? The trick is LFS tracked files are all in submodules which always have their remote pointing to Gerrit. Be it in:

the deploy directory (should be a file:// pointing to the cache)
the cache directory (should be http:// to the deployment server)

The issue is scap does not rewrite submodules URL to point to the deployment server. As a side effect that prevents breaking git lfs entirely.

When deploying a repository (without submodule) we thus need to set the end point .lsconfig:

[lfs]
  url = https://<path to Gerrit or Gitlab>

Or alternatively teach scap to pass the config when running git commands (git -c lfs.url=<path to Gerrit or Gitlab>).

hashar mentioned this in T279509: git-fat replacement/removal.Mar 29 2023, 7:37 PM

thcipriani moved this task from INBOX to Onboarding 🚀 on the Release-Engineering-Team board.Nov 15 2023, 4:43 PM

thcipriani edited projects, added Release-Engineering-Team (Onboarding 🚀); removed Release-Engineering-Team.

I guess scap can be made to use the canonical git URL on the target host instead of the deployment server git url, the workaround is to craft a .lfsconfig to bypass the autodetection. It is simple enough that I am claiming that workaround to fullfill the need.

hashar added a project: git-lfs.Thu, Apr 4, 8:25 AM

See also https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/226 deploy: Fix git-lfs support (submodules)

scap git LFS only works for submodulesClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

scap git LFS only works for submodules
Closed, ResolvedPublic
Actions