Page MenuHomePhabricator
Create Task
Maniphest T396622

Re-create GitLab test instance
Closed, ResolvedPublic
Actions

Description

I accidentally deleted the GitLab test instance at https://gitlab.devtools.wmcloud.org when fighting with horizon.

We should recreate a new instance at some point to be able to test scripts, changes and upgrades independent from the production infrastructure.

There is a gitlab-prod-backup volume in WMCS which should contain some backups.

We should use the opportunity to at least clean up the WMCS setup a bit, currently its quite messy.

Useful links:
https://wikitech.wikimedia.org/wiki/GitLab/Test_Instance#Setup_new_test_instances
https://phabricator.wikimedia.org/T302976

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
gitlab: pass ensure flag to auto_restarts::serviceoperations/puppetproduction+1 -0
gitlab: fix typo in hiera configoperations/puppetproduction+2 -2
gitlab: disable nftables prometheus exporter script in wmcsoperations/puppetproduction+13 -3
devtools: update hiera config for new bookworm hostsoperations/puppetproduction+23 -70
gitlab: disable second sshd on test instanceoperations/puppetproduction+8 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jelto created this task.Jun 11 2025, 2:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 11 2025, 2:32 PM
Jelto added a subtask: T394382: Upgrade GitLab to major version 18.Jun 11 2025, 2:35 PM
Jelto mentioned this in T396701: GitLab Security Release: 18.0.2, 17.11.4, 17.10.8 .Jun 12 2025, 6:26 AM
Jelto claimed this task.Jun 12 2025, 11:00 AM
Jelto triaged this task as Medium priority.
Jelto moved this task from Incoming to Work in Progress on the collaboration-services board.

I created a new instance gitlab-1001 in devtools. But I can not attache the volume with the backups because this volume is still stuck to the old instance. I opened T396739: Volume is stuck to deleted instance in devtools project to unblock this.

Jelto changed the task status from Open to Stalled.Jun 13 2025, 8:06 AM

This is blocked on T396739: Volume is stuck to deleted instance in devtools project

Jelto mentioned this in T396739: Volume is stuck to deleted instance in devtools project.Jun 17 2025, 8:26 AM
Jelto mentioned this in T397641: Persistent puppet failure on gitlab-runner-1008.devtools.eqiad1.wikimedia.cloud.Jun 23 2025, 3:55 PM
Jelto added a subtask: T397641: Persistent puppet failure on gitlab-runner-1008.devtools.eqiad1.wikimedia.cloud.
Jelto changed the task status from Stalled to Open.Jun 24 2025, 6:59 AM

The volume is available again and attached to the new test instance, I'll proceed setting up the test instance.

gerritbot added a comment.Jun 25 2025, 1:29 PM

Change #1163774 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: disable second sshd on test instance

https://gerrit.wikimedia.org/r/1163774

gerritbot added a project: Patch-For-Review.Jun 25 2025, 1:29 PM
gerritbot added a comment.Jun 25 2025, 1:40 PM

Change #1163774 merged by Jelto:

[operations/puppet@production] gitlab: disable second sshd on test instance

https://gerrit.wikimedia.org/r/1163774

Jelto added a comment.Jun 25 2025, 1:58 PM

I was able to re-create the test instance at https://gitlab.devtools.wmcloud.org/ using the backup which was stored on the backup volume. I had to downgrade the GitLab package to match the version of the backup.

https://gitlab.devtools.wmcloud.org/explore shows at least some repositories, puppet is happy and the network/VIP setup is much easier by dropping the second sshd (see change above).

Login/OIDC seems broken, runners have to be registered again (T397641), some tokens have to be updated and I have to update the docs.

ABran-WMF awarded a token.Jun 25 2025, 1:58 PM
Jelto added a comment.Jun 25 2025, 2:18 PM
In T396622#10947262, @Jelto wrote:

Login/OIDC seems broken

The test instance used the gitlab_replica_oidc identifier instead of the proper gitlab_oidc identifier because test instance thought it was a replica. I've set profile::gitlab::active_host: gitlab-1001.devtools.eqiad1.wikimedia.cloud properly. I'll add that to the docs. Login works now!

Jelto removed a subtask: T394382: Upgrade GitLab to major version 18.Jun 25 2025, 2:23 PM
Jelto added a parent task: T394382: Upgrade GitLab to major version 18.
Maintenance_bot removed a project: Patch-For-Review.Jun 25 2025, 2:32 PM
Jelto closed subtask T397641: Persistent puppet failure on gitlab-runner-1008.devtools.eqiad1.wikimedia.cloud as Resolved.Jun 25 2025, 2:43 PM
Andrew closed subtask T396739: Volume is stuck to deleted instance in devtools project as Resolved.Jun 25 2025, 7:31 PM
gerritbot added a comment.Jun 26 2025, 2:02 PM

Change #1164222 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] devtools: update hiera config for new bookworm hosts

https://gerrit.wikimedia.org/r/1164222

gerritbot added a project: Patch-For-Review.Jun 26 2025, 2:02 PM
gerritbot added a comment.Jun 26 2025, 2:06 PM

Change #1164222 merged by Jelto:

[operations/puppet@production] devtools: update hiera config for new bookworm hosts

https://gerrit.wikimedia.org/r/1164222

Maintenance_bot removed a project: Patch-For-Review.Jun 26 2025, 2:31 PM
gerritbot added a comment.Jun 27 2025, 8:19 AM

Change #1164389 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: disable nftables prometheus exporter script in wmcs

https://gerrit.wikimedia.org/r/1164389

gerritbot added a project: Patch-For-Review.Jun 27 2025, 8:19 AM
gerritbot added a comment.Jun 30 2025, 7:40 AM

Change #1164389 merged by Jelto:

[operations/puppet@production] gitlab: disable nftables prometheus exporter script in wmcs

https://gerrit.wikimedia.org/r/1164389

gerritbot added a comment.Jun 30 2025, 7:57 AM

Change #1164944 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: fix typo in hiera config

https://gerrit.wikimedia.org/r/1164944

gerritbot added a comment.Jun 30 2025, 8:01 AM

Change #1164944 merged by Jelto:

[operations/puppet@production] gitlab: fix typo in hiera config

https://gerrit.wikimedia.org/r/1164944

Maintenance_bot removed a project: Patch-For-Review.Jun 30 2025, 8:31 AM
gerritbot added a comment.Jun 30 2025, 9:55 AM

Change #1164965 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: pass ensure flag to auto_restarts::service

https://gerrit.wikimedia.org/r/1164965

gerritbot added a project: Patch-For-Review.Jun 30 2025, 9:55 AM
gerritbot added a comment.Jun 30 2025, 10:00 AM

Change #1164965 merged by Jelto:

[operations/puppet@production] gitlab: pass ensure flag to auto_restarts::service

https://gerrit.wikimedia.org/r/1164965

Maintenance_bot removed a project: Patch-For-Review.Jun 30 2025, 10:31 AM
Jelto closed this task as Resolved.Jun 30 2025, 12:16 PM

I'll resolve this task, the test instance is up and running again and I simplified the setup a bit and fixed some bugs in puppet (see changes above).

Dzahn reopened subtask T396739: Volume is stuck to deleted instance in devtools project as Open.Jul 1 2025, 5:48 PM
Jelto closed subtask T396739: Volume is stuck to deleted instance in devtools project as Resolved.Jul 2 2025, 12:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits