Migrate Gerrit deployment from git-fat to git-lfs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Mar 29 2023, 4:11 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

• demon

hashar

Description

To ship the gerrit.war and plugins.jar, the Gerrit deployment repository is using git-fat and git lfs offers a slightly more native experience.

Gerrit is deployed from a ~ 80MB war files + several jar files for plugins. The repository is https://gerrit.wikimedia.org/g/operations/software/gerrit.git and the artifacts are held in |deploy/wmf/stable-3.4` branch.

When updating Gerrit or its plugin, we commit the artifacts with git-fat and upload them using a Python script deploy_artifacts.py which wraps mvn deploy:deploy-file (see Gerrit/Upgrade#Upload_artifacts.

The process has a few issues:

one need an account on archiva.wikimedia.org
maven requires a user XML configuration with a locally encrypted credential
artifacts are uploaded with an arbitrary version and a reupload in case of mistake requires a different version to be passed
the python wrapper adds complexity
Archiva is rather slow T273086

Since LFS storage is provided by Gerrit itself, deploying Gerrit with git-lfs would depend on Gerrit running. The upgrade process is managed by scap with a few manual tasks (Gerrit/Upgrade#Deploying) and it boils down to:

deployment server: git pull
deployment server: scap deploy
- on the deployment target hosts scap invokes git fat to pull the artifact from Archiva
We manually stop Gerrit and refresh artifacts, refresh plugins then start Gerrit.

With git-lfs when scap fetch the artifacts Gerrit would be up and running and there will be no issue.

If the upgrade ever goes wrong and Gerrit is not running anymore or not serving LFS (for example if the LFS service itself is broken due to the upgrade) we would not be able to ship new artifacts. However given the artifacts are kept on the deployment targets scap cache, we should be able to rollback. If Gerrit is down, we would need to ensure we can indeed rollback to a previously deployed state and that git lfs being run by scap does not attempt to reach Gerrit but works entirely from the local disk.

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
Remove archiva.wikimedia.org	Hashar	operations/software/gerrit	deploy/wmf/stable-3.5	+0 -48
Migrate from git fat to git lfs	Hashar	operations/software/gerrit	deploy/wmf/stable-3.5	+41 -107
Remove gerrit-git-fat-pull job	Hashar	integration/config	master	+0 -31
Enable lfs for operations/software/gerrit	Hashar	All-Projects	refs/meta/config	+6 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T388129 remove git-fat objects from deployment server
Resolved	dancy	T279509 git-fat replacement/removal
Resolved	hashar	T333465 Migrate Gerrit deployment from git-fat to git-lfs

Event Timeline

hashar created this task.Mar 29 2023, 4:11 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 29 2023, 4:11 PM

Change 904230 had a related patch set uploaded (by Hashar; author: Hashar):

[All-Projects@refs/meta/config] Enable lfs for operations/software/gerrit

https://gerrit.wikimedia.org/r/904230

Change 904230 merged by Hashar:

[All-Projects@refs/meta/config] Enable lfs for operations/software/gerrit

https://gerrit.wikimedia.org/r/904230

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2023, 5:10 PM

Change 904239 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/software/gerrit@deploy/wmf/stable-3.5] Migrate from git fat to git lfs

https://gerrit.wikimedia.org/r/904239

gerritbot added a project: Patch-For-Review.Mar 29 2023, 5:30 PM

$ git push origin HEAD:refs/for/deploy/wmf/stable-3.5
Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://gerrit.wikimedia.org/operations/software/gerrit.git.locksverify true
Uploading LFS objects:  90% (9/10), 140 MB | 453 KB/s                                                                                                                                         
Uploading LFS objects: 100% (10/10), 158 MB | 448 KB/s, done.                                                                                                                                 
Enumerating objects: 36, done.
Counting objects: 100% (36/36), done.
Delta compression using up to 8 threads
Compressing objects: 100% (19/19), done.
Writing objects: 100% (19/19), 2.69 KiB | 306.00 KiB/s, done.
Total 19 (delta 5), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (5/5)
remote: Processing changes: refs: 1, new: 1, done    
remote: 
remote: SUCCESS
remote: 
remote:   https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/904239 Migrate from git fat to git lfs [NEW]
remote: 
To ssh://gerrit.wikimedia.org:29418/operations/software/gerrit.git
 * [new reference]         HEAD -> refs/for/deploy/wmf/stable-3.5

hashar updated the task description. (Show Details)Mar 29 2023, 7:26 PM

hashar merged a task: T316878: gerrit: stop using git-fat.

hashar added a subscriber: • demon.

hashar added a parent task: T279509: git-fat replacement/removal.Mar 29 2023, 7:28 PM

Change 904551 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Remove gerrit-git-fat-pull job

https://gerrit.wikimedia.org/r/904551

Change 904551 merged by jenkins-bot:

[integration/config@master] Remove gerrit-git-fat-pull job

https://gerrit.wikimedia.org/r/904551

Change 904239 merged by jenkins-bot:

[operations/software/gerrit@deploy/wmf/stable-3.5] Migrate from git fat to git lfs

https://gerrit.wikimedia.org/r/904239

Maintenance_bot removed a project: Patch-For-Review.Apr 4 2023, 7:30 AM

I have successfully deployed with git-fat this morning. With the upstream gerrit.war plugins bundled in our git repository, we no more have to run gerrit init and restart gerrit, the new versions of plugins are automatically picked up when Gerrit plugin loader inspects the directory (once per minute). The only case we need to restart it would be when Gerrit itself is upgraded.

I have to update the deployment documentation https://wikitech.wikimedia.org/wiki/Gerrit/Upgrade#Upload_artifacts