Change #1052384 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/puppet@production] stewards: clone user DB repo from GitLab

https://gerrit.wikimedia.org/r/1052384

From the repository end of things, I can easily generate repository-specific credentials (both HTTPS push token or a SSH key). After exploring for a bit, it is possible to include HTTPS credentials within a git clone command (something like git clone https://username:password@gitlab.wikimedia.org/repos/stewards/users.git works just fine). It also appears git::clone support overriding the generated remote via the origin parameter. With those two things combined, we should be able to construct the full URL based on the secret from private Puppet, and clone the repository via HTTPS.

@Dzahn I tried uploading a Puppet patch for this, but no guarantees on its correctness. It's also possible we already adopted a different approach – happy to hear about it. Curious what you think about what I created.

Change #1052384 merged by Dzahn:

[operations/puppet@production] stewards: clone user DB repo from GitLab

https://gerrit.wikimedia.org/r/1052384

• Added the secret token into the private repo on the puppetmaster

• Slightly amended and deployed the change above
  • Now in stewards1001:/srv/repos/users-db/.git/config the [remote "origin"] has the gitlab URL and it contains the secret token.

With the current settings of "ensure present", puppet is not expected to do an actual git pull. This would still be up to a human.

But it does manage the origin and looks like it should work.

Feel free to try it. I have left that to you for now.

The latest commit right now is yours from July 6.

Mentioned in SAL (#wikimedia-operations) [2024-07-11T02:45:38Z] <mutante> stewards2001 - sudo mv /srv/repos/users-db /root/ - run puppet and let it recreate the usersdb repo - this time pulling from gitlab - T369780 T369430

T369780 was auto-created by monitoring and made me notice the puppet run changing the git remote to gitlab failed on stewards2001 in codfw.

The sudo /usr/bin/git remote set-url origin .. command that puppet runs failed with error: No such remote 'origin'.

The easiest fix was to just move the users-db directory out of the way (it's in /root) and then run puppet and let it recreate it.

This worked without issue and the nice part is that this actually proofs a fresh clone from gitlab from the private repo works.

[urbanecm@stewards1001 /srv/repos/users-db (master|u=)]$ git pull
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0), pack-reused 0 (from 0)
Unpacking objects: 100% (3/3), 1.01 KiB | 259.00 KiB/s, done.
From https://gitlab.wikimedia.org/repos/stewards/users
 * [new branch]      P66165     -> origin/P66165
Already up to date.
[urbanecm@stewards1001 /srv/repos/users-db (master|u=)]$

At stewards2001, I see the repo populated now, and pulling works equally well.

Looks to work! Thanks @Dzahn. I added a minor comment on the revised patch (related to naming), but since it is nothing significant, we likely don't need to dive into it further. Resolving, since the scope is completed.

Change #1053783 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] stewards: rename userdb_gitlab_token variable

https://gerrit.wikimedia.org/r/1053783

Change #1053783 merged by Dzahn:

[operations/puppet@production] stewards: rename userdb_gitlab_token variable

https://gerrit.wikimedia.org/r/1053783