My current plan for cluster-wise migration and deploying services with helm3 is:

The warning was removed in the GitLab upgrade to 14.x (https://phabricator.wikimedia.org/T289802.) The grep was removed in https://gerrit.wikimedia.org/r/719930. So I'm closing this issue.
Feel free to open again.

Thanks for looking this up!

@Arnoldokoth some thoughts on the wrong settings on the replica after restore:

Some additional RBAC requirements:

@dancy Thanks for finding this issue!

I updated gitlab-ce to 14.2.3-ce.0 on apt1001

I updated gitlab-runner to 14.2.0 and gitlab-ce to 14.1.5-ce.0 on apt1001.

@Jelto I'll coordinate with you tomorrow, but if you want to go ahead with the upgrade on gitlab2001 before I'm online, feel free.

In T251305#6886431, @JMeybohm wrote:

helm test annotations changed a bit:

Note that until Helm v3, the job definition needed to contain one of these helm test hook annotations: helm.sh/hook: test-success or helm.sh/hook: test-failure. helm.sh/hook: test-success is still accepted as a backwards-compatible alternative to helm.sh/hook: test.

We might also want to add "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded as of https://helm.sh/docs/topics/charts_hooks/#hook-deletion-policies

I started to collect some thoughts for the long-term GitLab Runner setups here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#Future_Gitlab_Runner_setup_(T286958)

In T251305#7319375, @elukey wrote:

Adding a comment in here since I am trying to figure out a similar thing (although I have way less context) for what we'll probably call ml-services dir under helmfile.d (see T286791). I would love to start with helm3 straight away, and we could think about using the ml-serve-eqiad cluster as testbed in case it fits (we are testing an MVP, nothing in production yet). Let me know!

I tested the behavior of gitlab2001 during a rolling restart of puma workers:

There is a ClusterRole named deploy already for the aggregation of view and pods/portForward permissions. So I would prefer using the names <service-name> and <service-name>-rw for the two users. If we use the deploy in the name of the new user this may be confused with the read-only user and the deploy ClusterRole.

And some more input regarding RBAC and the replacement of Tiller service account:

In T251305#7304773, @JMeybohm wrote:

That is pretty cool, thanks! Did you actually deploy something using helm3 with tillerNamespace: still set? Is it just ignored in that case (helmfile can be really picky :))?

The helm binary in helmfile can be set using the --helm-binary option or by setting helmBinary in the helmfile.yaml.
It can be set globally (like in admin-ng). But it is also possible to to change the helm binary depending on the environment.

gitlab1001 just had another rolling restart of puma workers:

I created a dedicated task for the reduced availability for the puma worker/exporter: https://phabricator.wikimedia.org/T289454

I like the idea of having a semi-formal update window for GitLab as well.

I would like to either finish this task or add additional requirements. Currently we are collecting metrics of all GitLab components on gitlab1001 and gitlab2001. We have Grafana dashboards and basic alerts in icinga.

From my side everything is done. Thanks everyone. I'm going to close this ticket. Feel free to open it in case I missed anything.

The puma workers get killed either if they hit memory limit (which is puma['per_worker_max_memory_mb'] = 1024 ~1gb by default) or automatically after 12 hours. I can't see one of the workers hitting the limit in the past. They are getting killed because the uptime is above 12 hours and during the restart the service availability is reduced. My assumption is that some parts of the puma webserver/GitLab have memory leaks. This is the reason for the filling memory and why GitLab uses tools like puma_worker_killer. I try to debug and troubleshoot this problem on gitlab2001 a little bit further. I would like to have a smooth restart of the puma workers without reduced availability.

Should be fixed with https://gerrit.wikimedia.org/r/710676. No new error message from root@gitlab1001

I restored the backup of gitlab1001 to gitlab2001 using the restore instructions of S&F.
I enhanced the guide and moved it to wikitech: https://wikitech.wikimedia.org/wiki/GitLab/Backup_and_Restore#Restore

GitLab rails service has reduced availability roughly every ~24 hours (plus some offsets). See this Grafana dashboard.

Basic Icinga alerts for the public https and SSH endpoints of GitLab are in place now: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=gitlab.wikimedia.org

The scrape configuration for GitLab is in place and Prometheus collects metrics.

I modified and ran the install script using ansibles --check (dry-run) flag against gitlab2001. Looks good, there are two errors due to check mode usage.

During the refresh of old mw app servers in eqiad we noticed that thumbor machines thumbor1001 and thumbor1002 are renamed/reimaged mw hosts. As mentioned in T280203 and T233196 these machines are end of life and have to be refreshed.

I merged and rolled out the additional parameter for intra-service dependencies in systemd::timer::job and the change for a weekly rebuild of production-images.

In T274463#7159798, @jbond wrote:

I can explain this, I have created a gitlab module which is not used in production, but is running in a cloud project (ping me if you want access). The modules is a first pass at converting the ansible code into puppet code. I think its about 80-90% complete. I'm not sure what the future plans are here in relation to if we are integrating gitlab further into puppet or moving it out of puppet and into a k8s cluster, so for now i have mostly stalled on the production aspects of the code (for my needs i just needs a basic instance up to test api's); however happy to revive things or go through the code if its useful

@Dzahn I think https://gerrit.wikimedia.org/r/c/operations/puppet/+/697850 is not merged and deployed, so the fileset for GitLab doesn't exist.

So we have two options for a rebuild:

@Joe I think a periodic job rebuilding the images is implemented already.
See modules/docker/manifests/baseimages.pp#65

I checked the current exporter configuration.
Host/node metrics are available by node_exporter. The node_exporter in GitLab is disabled because we are using the stand-alone node_exporter. The host metrics are available in Grafana already. So host/node metrics should be fine.

@Dzahn and I checked the backups on gitlab1001.