In T365259 it was discussed to move Gerrit behind the CDN/loadbalancer. It was discussed that GitLab may be a better first candidate because the setup is quite similar (one web service and one ssh service) but GitLab is less production-critical and has more test instances available. Ultimately putting those instance behind the CDN would mean they no longer need public IP addresses, so this would also cover T310265. However this is a second step, because this requires reimaging the host after successful tests.
Most discussion already happened in T365259 which also holds up for GitLab. So this task is mostly for discussing and tracking the actual technical implementation of putting GitLab behind the CDN.
GitLab consits of multiple machines and services:
- GitLab production
- web service at https://gitlab.wikimedia.org
- ssh service at gitlab.wikimedia.org:22
- GitLab replica
- web service at https://gitlab-replica.wikimedia.org
- ssh service at gitlab-replica.wikimedia.org:22
- GitLab replica old (2nd replica)
- web service at https://gitlab-replica-old.wikimedia.org
- ssh service at gitlab-replica-old.wikimedia.org:22
This services are not related or distributed in any way. The replicas are standby machines which can be used for emergency switchovers and testing. They run an actual GitLab instance with old (12h) data but this instances are not used for the production GitLab.
We will start by mostly following https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service, which consists of roughly the following steps:
- Ensure the service is running on all the backend servers
- Add relevant data in etcd: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1040094
- Add DNS records, allocate service IPs in all datacenters where the service is running
- Create an entry in the service::catalog
- Add this IP to the loopback interface on all the servers where the service is present
- Configure the load balancers to provide balancing across those backends
- Add the puppet-generated discovery DNS resources, start sending network probes/monitoring
- Make the service page
- Add discovery DNS records for the service