Page MenuHomePhabricator

Create GitLab mirror
Open, LowPublic

Description

As discussed we want to have some kind of repository mirror for production GitLab which has a lower delay than the replica (currently 24h). I see multiple options to implement this.

Restore replica more often

Currently the GitLab replica is synced with production GitLab every 24h. We could increase the backup and restore frequency to something like once every 2h. I don't see any technical constrains here. The resource usage on production GitLab looks fine while doing backups.

We cloud also think about introducing partial backup and restore. So we could do a full backup and restore cycle on the replica every 24hs. For partial backups we would skip certain folders like the database and only sync the repositories.

Sync git files to replica/mirror machine

We could also try to just sync the repositories using rsync. So we could setup a timer to sync /var/opt/gitlab/git-data/repositories/ to the replica or another mirror machine.

However I don't know what happens to the gitaly server if we update the repositories on disk. The documentation discourages this.

Mirror repositories using repository mirroring

Repository mirroring would be GitLabs build-in feature to create mirrors. We could mirror repositories to:

  • GitLab.com
  • replica(s)
  • Github.com
  • a dedicated mirror machine

Mirrors have to be configured per project. So we would have to create some kind of script or hook which sets up the mirror. Mirrors can be updated via api. Mirrors also support push-updates. So the mirror would be in sync with production GitLab.

I assume to implement mirroring we have to write some code to add mirroring to all projects. If we want to mirror to gitlab.com we have to think about what projects we want to mirror (only "our" official projects or user projects as well?). If we want to mirror to the replica we have to make sure to disable mirroring when we switch or fallback to the replica.

Use gitaly in cluster mode

Another option would be to use gitalys cluster feature. It is possible to run multiple instances of the git server and replicate the data. However this is quite an advanced feature. We would have to run a dedicated gitlay server and would start to separate some components out of the omnibus package.

Conclusion

For the current architecture and usage I would like to use the first solution. Most of the code is in place and we just have to increase the backup and restore frequency. We also wouldn't need a additional mirror machine or setup additional hosts.

Details

Event Timeline

brennen edited projects, added: GitLab (Infrastructure); removed: GitLab.

Noting here the existence of Replicant, which KDE are using to handle realtime backups of Git activity. Probably worth looking at.

It sounds like there are three open options here and the next step is to decide on the path forward, do I have that right?

Change 774418 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: run backup and restore twice daily

https://gerrit.wikimedia.org/r/774418

Change 774418 merged by Dzahn:

[operations/puppet@production] gitlab: run backup and restore twice daily

https://gerrit.wikimedia.org/r/774418

Jelto removed Jelto as the assignee of this task.Sep 20 2022, 3:37 PM
Jelto added a project: collaboration-services.
LSobanski updated the task description. (Show Details)

Given the age of this task I'd like to understand the use cases it is supposed to address before choosing a solution. If you are an interested party, please respond with your use case.

When opening the task disaster recovery was the main use-case. If production GitLab is down, we may have lost up to 24 hours of repository changes (during the time of ticket creation, now it's six hours). So the idea of that mirror is to reduce the time which we may loose.

For that use case we most probably don't need a fully working git frontend like the replicas or another repo hosted somewhere else. In case of an incident with data loss I'd not except end-users to manually pull their changes from the mirror and re-applying them and SRE team would be involved anyways. So we can also create more frequent git repository backups and store them somewhere else (replicas, bacula).

According to some tests earlier this year (T316935#8805128) a backup of the repositories takes around two to three minutes. So if we do repository backups every few hours and sync the backup to another host, we should cover this use-case mostly.