A second replica of GitLab in codfw would help to validate the setup of production Gitlab gitlab1001 in eqiad. This replica should be passive and we want to use it to test backup and restore of gitlab1001 backups.
In the future this instance can also be used for failover and automated restore scenarios.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Jelto | T285867 GitLab replica in codfw | |||
| Resolved | None | T285456 codfw: 1 of VMs requested for gitlab | |||
| Resolved | Jelto | T285870 request service IP / DNS name for gitlab-replica, apply puppet role on gitlab2001 |
https://gerrit.wikimedia.org/r/c/operations/puppet/+/702126 prevents that backups will be created twice. Bacula will only backup the active server, not the replica.
Change 707350 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/gitlab-ansible@master] add gitlab2001 to host_vars and variables
I modified and ran the install script using ansibles --check (dry-run) flag against gitlab2001. Looks good, there are two errors due to check mode usage.
I would like to roll out the ansible playbook on gitlab2001 without the --check flag to setup the machine completely.
Complete change: https://gerrit.wikimedia.org/r/c/operations/gitlab-ansible/+/707350
Output:
./install-gitlab-server.sh
PLAY [Install Gitlab Server] ***************************************************************************************************************************************************************************
TASK [Gathering Facts] *********************************************************************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Include OS-specific variables] ***************************************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Check if GitLab configuration file already exists] *******************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Check if GitLab is already installed] ********************************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Gather package facts] ************************************************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab dependencies] *****************************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab dependencies (Debian)] ********************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Check GitLab dependencies] *******************************************************************************************************************************************************
skipping: [gitlab-server-replica] => (item=openssh-server)
skipping: [gitlab-server-replica] => (item=curl)
skipping: [gitlab-server-replica] => (item=openssl)
skipping: [gitlab-server-replica] => (item=tzdata)
TASK [gitlab_server : Check GitLab dependencies (Debian)] **********************************************************************************************************************************************
skipping: [gitlab-server-replica] => (item=gnupg)
TASK [gitlab_server : Download GitLab repository installation script] **********************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab repository] *******************************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Define the Gitlab package name] **************************************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Generate random password] ********************************************************************************************************************************************************
ok: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab] ******************************************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Reconfigure GitLab (first run)] **************************************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Create GitLab SSL configuration folder] ******************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Create self-signed certificate] **************************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Copy GitLab configuration file] **************************************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Create GitLab crontab] ***********************************************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Setup GitLab user] ***************************************************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab sshd - create config directory] ***********************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab sshd - create config file] ****************************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab sshd - create moduli file] ****************************************************************************************************************************************
fatal: [gitlab-server-replica]: FAILED! => {"changed": false, "msg": "Destination directory /etc/ssh-gitlab does not exist"}
...ignoring
TASK [gitlab_server : Install GitLab sshd - create host keys] ******************************************************************************************************************************************
changed: [gitlab-server-replica] => (item=/etc/ssh-gitlab/ssh_host_rsa_key)
changed: [gitlab-server-replica] => (item=/etc/ssh-gitlab/ssh_host_ecdsa_key)
changed: [gitlab-server-replica] => (item=/etc/ssh-gitlab/ssh_host_ed25519_key)
TASK [gitlab_server : Install GitLab sshd - create service] ********************************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Install GitLab sshd - enable service] ********************************************************************************************************************************************
fatal: [gitlab-server-replica]: FAILED! => {"changed": false, "msg": "Could not find the requested service ssh-gitlab: host"}
...ignoring
TASK [gitlab_server : Remove GitLab sshd - stop service] ***********************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Remove GitLab sshd - remove config file] *****************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Remove GitLab sshd - remove config directory] ************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Remove GitLab sshd - remove service] *********************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Remove GitLab sshd - reload systemd] *********************************************************************************************************************************************
skipping: [gitlab-server-replica]
TASK [gitlab_server : Create bacula backup directory for GitLab data backup] ***************************************************************************************************************************
changed: [gitlab-server-replica]
TASK [gitlab_server : Create bacula backup directory for GitLab config backup] *************************************************************************************************************************
changed: [gitlab-server-replica]
RUNNING HANDLER [gitlab_server : restart gitlab] *******************************************************************************************************************************************************
skipping: [gitlab-server-replica]
PLAY RECAP *********************************************************************************************************************************************************************************************
gitlab-server-replica : ok=19 changed=10 unreachable=0 failed=0 skipped=15 rescued=0 ignored=2Change 707350 merged by Brennen Bearnes:
[operations/gitlab-ansible@master] add gitlab2001 to host_vars and variables
Change 708275 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/gitlab-ansible@master] disable backup cronjobs for gitlab2001
Change 708275 merged by Brennen Bearnes:
[operations/gitlab-ansible@master] disable backup cronjobs for gitlab2001
Change 708767 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] hiera::role::common::acme_chief add gitlab-replica SNI
Change 708767 merged by Jelto:
[operations/puppet@production] hiera::role::common::acme_chief add gitlab-replica SNI
Change 709383 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] hiera::role::common::idp add gitlab-replica to production idp
Change 709383 merged by Jelto:
[operations/puppet@production] hiera::role::common::idp add gitlab-replica to production idp
I restored the backup of gitlab1001 to gitlab2001 using the restore instructions of S&F.
I enhanced the guide and moved it to wikitech: https://wikitech.wikimedia.org/wiki/GitLab/Backup_and_Restore#Restore
I also documented some first information about the replica in a dedicated page: https://wikitech.wikimedia.org/wiki/GitLab/Replica
The next step is to evaluate options to automate this process and to run the restore on gitlab-replica every 24h.
Change 710948 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] profile::gitlab rsync latest backup to passive host
Change 710948 merged by Jelto:
[operations/puppet@production] profile::gitlab rsync latest backup to passive host
Change 711348 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] profile::gitlab rsync fix rsync backup command
Change 711348 merged by Jelto:
[operations/puppet@production] profile::gitlab rsync fix rsync backup command
Change 713635 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] profile::gitlab load rsync::server only on passive GitLab
Change 713635 merged by Jelto:
[operations/puppet@production] profile::gitlab load rsync::server only on passive GitLab
@Arnoldokoth some thoughts on the wrong settings on the replica after restore:
When we import the dump of production GitLab to the repica, the setting for home_page_url doesn't match the replica. So redirects and a click on the top left GitLab icon result in redirects to production GitLab. Which may confuse people especially when doing tests on the replica. So we have to find a way to adjust this setting.
One solution would be to use gitlab-settings and run the Python script to adjust all settings after doing the restore. This would mean we have to clone the script and create API keys for access. For me this sound like a lot of moving parts.
A simple solution would be to set the home_page_url using the gitlab-rails console. The console is available on the replica and doesn't require additional credentials (apart from being root).
So for the first version of the restore script we could add the following line to make sure the home_page_url matches the replica:
echo "ApplicationSetting.last.update(home_page_url: 'https://gitlab-replica.wikimedia.org/explore')" | gitlab-rails console
This solution is not optimal. As soon as we also change other settings on production GitLab we have to add them as well. So we might need to use gitlab-settings at some point. But for now I'm fine with just setting this single value in the console.
Change 725340 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] gitlab: install backup restore script
Change 725340 merged by Dzahn:
[operations/puppet@production] gitlab: install backup restore script
Change 730605 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] gitlab: add gitlab restore systemd timer
Change 730605 merged by Dzahn:
[operations/puppet@production] gitlab: add gitlab restore systemd timer
Mentioned in SAL (#wikimedia-operations) [2021-10-13T19:16:00Z] <mutante> gitlab2001 - status before was that "gitlab-ctl status" showed components "gitlab-workhorse" and "postgres-exporter" as "down". this was either pre-broken or caused by the restore process. after manually 'gitlab-ctl start gitlab-workhorse' all of the components are in "run" and https://gitlab-replica.wikimedia.org is up ( T285867)
Change 730630 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] gitlab: remove unnecessary comments from restore script
Change 730632 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gitlab: deactivate new backup-restore unit for now
Change 730632 merged by Dzahn:
[operations/puppet@production] gitlab: deactivate new backup-restore unit for now
Change 730630 merged by Dzahn:
[operations/puppet@production] gitlab: remove unnecessary comments from restore script
Change 730641 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gitlab: allow installing the restore script while NOT enabling the timer
Change 730641 merged by Dzahn:
[operations/puppet@production] gitlab: allow installing the restore script while NOT enabling the timer
Change 731154 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] gitlab: redirect out to logfile in restore script
Change 731154 merged by Dzahn:
[operations/puppet@production] gitlab: redirect out to logfile in restore script
Change 731155 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] gitlab: re-enable timer for backup-restore script
Change 731155 merged by Dzahn:
[operations/puppet@production] gitlab: re-enable timer for backup-restore script
Change 734741 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] gitlab: disable puppet in script and change timer
Change 734741 merged by Dzahn:
[operations/puppet@production] gitlab: disable puppet in script and change timer
Change 735435 had a related patch set uploaded (by AOkoth; author: AOkoth):
[operations/puppet@production] gitlab: enable backup restore timer
Mentioned in SAL (#wikimedia-operations) [2021-10-28T21:46:53Z] <arnoldokoth> restoreccccccvkvhgbvtklgce kkbeuvvuskljihickdbgcunljcr scheduled to run on gitlab2001 (T285867)
Mentioned in SAL (#wikimedia-operations) [2021-10-28T21:48:18Z] <arnoldokoth> restore script scheduled to run on gitlab2001 (T285867)
We have spread the updates related to gitlab-backup-restore / sync across multiple tickets. also see T283076#7466960
Change 735435 abandoned by AOkoth:
[operations/puppet@production] gitlab: enable backup restore timer
Reason:
merged through different change
Change 773783 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gitlab: add version check to restore script
Change 774418 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gitlab: run backup and restore twice daily
Change 773783 merged by Jelto:
[operations/puppet@production] gitlab: add version check to restore script
Change 774418 merged by Dzahn:
[operations/puppet@production] gitlab: run backup and restore twice daily
Change 776198 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gitlab: pass restore interval to gitlab::restore module
Change 776198 merged by Dzahn:
[operations/puppet@production] gitlab: pass restore interval to gitlab::restore module