So far all gitlab* machines have been virtual.
Now we just got the first dedicated hardware, physical servers for gitlab.
There is one ticket for codfw and one for eqiad, both include gitlab* and gitlab-runner* machines.
codfw - T301183
eqiad - T301177
Both are now ready for us to take over.
### GitLab Runner migration
`gitlab-runner` hosts can be integrated independently from GitLab migration. So the following machines need `role(gitlab_runner)`:
[x] gitlab-runner1002.eqiad.wmnet (paused)
[x] gitlab-runner1003.eqiad.wmnet (paused)
[x] gitlab-runner1004.eqiad.wmnet (paused)
[x] gitlab-runner2002.codfw.wmnet (paused)
[x] gitlab-runner2003.codfw.wmnet (paused)
[x] gitlab-runner2004.codfw.wmnet (paused)
If the above Runners are configured and ready, ganeti VMs `gitlab-runner1001.eqiad.wmnet` and `gitlab-runner2001.codfw.wmnet` can be unregistered and destroyed.
[] decommission gitlab-runner1001.eqiad.wmnet
[] decommission gitlab-runner2001.codfw.wmnet
### GitLab migration
GitLab migration needs some additional preparation.
**Preparations before downtime:**
[x] register second service IPs for `gitlab1003`
[x] validate puppet code and GitLab configuration with a physical replica on `gitlab1003` (also bullseye)
[] ~~evaluate additional configuration changes for potential HA setups~~
[x] create custom partman config for GitLab [793534](https://gerrit.wikimedia.org/r/c/operations/puppet/+/793534)
* bigger `/` root volume
* dedicated `/srv` volume (and move backups back to this folder instead of `/mnt`)
* ~~dedicated Docker volume~~ not needed
* dedicated Registry volume (see `gitlab_rails['registry_path']`)
[x] register second service IPs for `gitlab1004`
[x] move `gitlab-replica.wikimedia.org` entry from netbox to dns repo [800709](https://gerrit.wikimedia.org/r/c/operations/dns/+/800709)
[x] move `gitlab.wikimedia.org` entry from netbox to dns repo [800719](https://gerrit.wikimedia.org/r/c/operations/dns/+/800719)
[x] apply `role(gitlab)` to `gitlab1004` and verify installation [800728](https://gerrit.wikimedia.org/r/c/operations/puppet/+/800728)
[] copy ssh host keys for `ssh-gitlab` daemon from `gitlab1001` to `gitlab1004`
[] configure `gitlab1004` with `profile::gitlab::service_name: 'gitlab.wikimedia.org'`
[] configure `gitlab1004` as `profile::gitlab::active_host`
[] apply [gitlab-settings](https://gitlab.wikimedia.org/repos/releng/gitlab-settings) to `gitlab1004`
[] announce downtime some days ahead on ops/releng list?
**Scheduled downtime**:
[] Announce downtime in `#wikimedia-gitlab`
[] pause all GitLab Runners
[] stop GitLab on `gitlab1001` with `gitlab-ctl stop`
[] create backup on `gitlab1001` with `sudo /mnt/gitlab-backup/gitlab-backup.sh full` and `sudo /mnt/gitlab-backup/gitlab-backup.sh config`
[] sync backup, on `gitlab1001` run `/usr/bin/rsync -avp /mnt/gitlab-backup/latest/ rsync://gitlab1004.wikimedia.org/data-backup`
[] trigger restore on **`gitlab1004`** run `/srv/gitlab-backup/gitlab-restore.sh`
[] overwrite home_page_url. on `gitlab1004` run `echo "ApplicationSetting.last.update(home_page_url: 'https://gitlab.wikimedia.org/explore')" | /usr/bin/gitlab-rails console`
[] Point DNS entry for `gitlab.wikimedia.org` to gitlab1004 (TODO: prepare change) and run `authdns-update`
[] verify installation
[] unpause all GitLab Runners
[] announce end of downtime
**Tasks after downtime:**
[] switch fileset for gitlab from /mnt to /srv [800357](https://gerrit.wikimedia.org/r/c/operations/puppet/+/800357)
[] migrate additional hosts
[] gitlab2002.wikimedia.org as replica
[] gitlab2003.wikimedia.org as replica
[] decommission old hosts
[] gitlab2001.codfw.wmnet
[] gitlab1001.eqiad.wmnet