I run the script in repos/releng/gitlab-trusted-runner/ manually:

In T311241#8042109, @Dzahn wrote:

currently the issue here is not DNS anymore.

but it is now: 'This job is stuck because you don't have any active runners online or available with any of these tags assigned to them: protected '

even though we see the de-registered and re-registered runner tagged as protected

In T274463#8045578, @Dzahn wrote:

unfortunately just noticed an Icinga alert for gitlab1003 (nothing mails us about this, that's just if you happen to log at web UI for some reason)

gitlab1001 and gitlab2001 will be decommissioned soon in T307142. So regarding GitLab this should be resolved soon.

Docker cache is cleaned every 24h on GitLab Runner nodes now. So failing jobs due to full docker volume should happen less frequent.

I can confirm, pipeline works again for me. Re-configuring the Cloud Runners was block because of a failing pipeline. But this works again and Cloud Runner are using the new double star config now.

@Arnoldokoth @Dzahn what's missing to also check

In T310265#7993198, @Dzahn wrote:

First and foremost though, the reason why gitlab has all public IPs is because we were trying to emulate the gerrit setup. And gerrit has public IPs and is not behind LVS because we wanted it that way. We wanted to be able to still use Gerrit and merge changes even if the caching layer is down for some reason.

We gathered some experience regarding failover when migrating GitLab to the new physical hosts in T307142.

puppet runs on the test instance gitlab-prod-1001 fail with

In T307142#7978386, @Jelto wrote:

...
One rsync job failed on the old production host gitlab1001. I'll take a closer look. I assume that's a timer/resource which is not managed by puppet anymore.

After migrating to new hosts (T307142) we got a bacula alert about backups on gitlab1001 (the old production machine):

In T307142#7976964, @Jelto wrote:

...
I'll check bacula tomorrow for new backups of gitlab1004 and make sure the replicas synced properly.

Migration of production GitLab from gitlab1001 to gitlab1004 was successful. Downtime was around 65 minutes.

Checklist for gitlab migation from gitlab1001 to gitlab1004:

Checklist for todays gitlab-replica migation from gitlab2001 to gitlab1003:

gitlab1003 and gitlab1004 are configured as GitLab replicas now and are serving https://gitlab-replica-new.wikimedia.org/ and https://gitlab-new.wikimedia.org/.

Backup size decreased after cleanup of big projects. Thanks again to @brennen and @Dzahn for finding and coordinating this!
We are down from 50GB to 10GB for one backup. That also means disk pressure on the backup volume decreased a lot (see disk usage over time dashboard).

So it did succeed, good! Not sure about that internal API error though.

@Dzahn thanks for testing the partman config! I'm happy it worked first time!

In T308380#7944179, @jcrespo wrote:

Queries involved in the issue during the first phase (wikidata): https://logstash.wikimedia.org/goto/feefd185271bdccfda06da231c09bfe1
Queries involved in the issue during the second phase (ruwiki): https://logstash.wikimedia.org/goto/5d3c2846f8ca3c41748e067d00260ba9

I solved the installation/puppet issues with gitlab1003. The gitlab-ce package was installed and login using CAS/IDP worked. Synced backups for the backup-restore cycle were also present already.

1. Trusted Runner automation and access request

Reopening, puma still fails to stop:

Thanks for opening the task!

@thcipriani I added some more open topics to the description. Can you take a look? I would like to know what is needed from your perspective until Cloud Runners can be available instance wide.

After yesterdays incident mw2412 got depooled again to restore the state before the incident (see SAL). I'm going to adjust this and pool mw2412 again. This host is ready for production similar to the other hosts of mw241[2-9].

In T307255#7913372, @Joe wrote:

Maybe you got confused by the stale files there that we should remove for the non-https LVSes?

mw241[2-9] where pooled in an incident this morning (accidentally depool and pool of codfw datacenter) . I run a scap pull on all machines to make sure they are up to date.

I added more restrictive CPU and memory limits to the Cloud Runner configuration (0.1 CPU and 200Mi Memory). I also set the timeout for jobs to 300s which is the minimum.

That's related to T295481.

https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner has CI for provisioning the managed Kubernetes cluster and setup of Kubernetes Runner now. Thats mostly done using Terraform and Helm. So we have working Cloud Runners with autoscaling (min 1 and max 2 nodes).

I would suggest to treat gitlab-runner hosts a little different. For the Runner hosts we can basically can apply the puppet role, add data to Hiera and remove the old ganeti VMs.

We have a k8s cluster on Digital Ocean that we're using to prove the viability of ^ model. We talked it over with ServiceOps and WMCS and that's a good path for the time-being if everything seems to work correctly. In future, we'll continually evaluate whether a third party cloud is the right place to run this.

This has been implemented in https://gerrit.wikimedia.org/r/732093, I'm closing this task.

In T297659#7862187, @Dzahn wrote:

@Jelto All the (non-protected) prod runners are upgraded. Now I was just wondering about the 2 protected runners. They are paused. Should I try upgrading those as well?

In T274463#7835100, @jcrespo wrote:

[...]
BTW, restores directly to a different host are possible, although a bit cumbersome at the moment (because encryption reasons). We want to make this easier in the future.

We discussed in last ITC meeting that a dedicated GitLab update and maintenance window is not needed now. The last downtimes for updates and maintenance lasted between 2 to 5 minutes and were announced some hours ahead. With current usage of GitLab we agreed that this is not an issue. Also a fixed window would slow down progress on infrastructure tasks around GitLab because we have to wait for the next window.

I have access now, thanks a lot!

I like the idea of putting the bullseye runner runner-1020 into the gitlab-runners project. That reduces overhead around the puppet and hiera configuration.

Thanks for the quick help! However I still have problems accessing some task. For example I can not access T304938, which is marked as security.

Backup is present in bacula for the new folder structure:

In T274463#7830451, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2022-04-05T00:32:47Z] <mutante> gitlab.wikimedia.org was down because gitlab1001 ran out of disk space. ran 'apt-get clean' to free 13G which made it recover... T274463 - <+icinga-wm> RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK

In T274463#7823634, @Dzahn wrote:

@Jelto @Arnoldokoth See above. I added a new disk to gitlab2001 and gitlab1001. On gitlab2001 I have also done the other necessary steps to actually use it, while I have left gitlab1001 untouched so far.

• created virtual disk on ganeti level
• rebooted on ganeti level (not enough to boot on machine level)
• ran into the bug with the host not coming back, exactly like the other day, fixed interface name again
• created partition table
• created partition
• created file system, ext4
• created /mnt/gitlab-backup
• mounted on /mnt/gitlab-backup
• used blkid to get UUID of new partition
• edited /etc/fstab and inserted "UUID=c5235682-ac21-46a9-85ee-9603f694a6a4 /mnt/gitlab-backup ext4 errors=remount-ro 0 2" (0 = no dumping, 2 = no fsck unlike the root file system)
• rebooted to check if it survices the reboot and gets auto-mounted, it does.

There is now this:

/dev/vdb1        98G   61M   93G   1% /mnt/gitlab-backup

And now backups could be switched from /srv/gitlab-backup to /mnt/gitlab-backup.

And then we would have to do those steps above also for gitlab1001.

@Arnoldokoth and I updated production instance gitlab1001 and gitlab-runners successfully.

@Arnoldokoth and I updated the test instance gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud and the replica gitlab2001.wikimedia.org to gitlab-ce 14.9.1-ce.0.

This will happen tomorrow/Tuesday due to scheduling conflicts.

@Arnoldokoth and I will do the upgrade of GitLab + Runners on Monday after 4pm UTC.

I mirrored wmf-sre-laptop to GitLab and created a very basic proof-of-concept CI to build the Debian package on Trusted Runners. The current implementation has limitations and is not complete. I created T304491 to further discuss the whole topic of Debian package builds on GitLab CI, as this is a bit out of scope for this task.

I created a dedicated task to automate the test instance creation: T302976

With the help of @Majavah the correct configuration of private and public/floating IP was found. https and cloning over SSH works now. Thanks again!

The keepalived VIP configuration was not clear for me from looking at the horizon interface.

Thanks a lot for setting up the additional port! I can confirm that the port is present in Horizon Interface.

SSH access to the test instance is not working because of different networking behavior on WMCS/VPS. The public floating IP ("service ip") is NATed to the VM. So we can not bind on this address directly.
I requested a second networking port in T302803 and hope we can map/NAT the floating IP to this second port to replicate the production configuration (with NGINX and git SSH daemon listening on a different address).