-1. Scoping resources by WMF team excludes volunteers from participating and often doesn't reflect reality anyway.

It's also striking that the set of projects people work on just isn't that cleanly mapped to any particular organizational structure. Once you've been a technical contributor for a while, you've almost certainly collected responsibilities that no org chart reflects accurately.

(quite from https://phabricator.wikimedia.org/phame/post/view/273/gitlab_rethinking_how_we_handle_access_control/)

Something like deploy-jenkins would not have that problem.

@taavi, that is a valid concern.

There is already a jenkins-deploy Unix user though, so to reduce confusion maybe we can go with something like deploy-jenkins-service.

The existing format is "deploy_$service", deploy_zuul, deploy_ci_docroot, deploy_service,... so I would prefer "deploy_jenkins".

A new keypair / identity has been created in the private puppet repo:

remote:  modules/secret/secrets/keyholder/deploy_jenkins     | 8 ++++++++
remote:  modules/secret/secrets/keyholder/deploy_jenkins.pub | 1 +

Next this needs to be used either from scap hiera data or directly with the keyholder::agent class in a profile on contint servers.

Change 867294 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] scap: add stanza for jenkins deploy, new keyholder identity

https://gerrit.wikimedia.org/r/867294

cc: @Jelto I followed https://wikitech.wikimedia.org/wiki/Keyholder#Generating_a_key_for_a_new_identity_or_update_an_existing_one but also made an edit to it that it's ok to use the same passphrase for all (regular) deployment keys.

And I will add tomorrow what kind of puppet change is needed to actually _use_ the new key which was not documented yet but I believe is:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/867294

This is after reading some comments in the keyholder::agent class how most of the instances of it are generated from Hiera data instead of directly using the class.

A bunch of services also use the class directly though.

Thanks a lot for adding the new identity @Dzahn

I don't know if there's another way to grant access to the new key, but what I'm aware of involves granting access to a Unix group in hieradata/role/common/deployment_server/kubernetes.yaml. I can follow up on that though, there's also a couple of other Puppet changes I need to add.

The one thing I didn't have permissions to do was to add the new identity keypair and rearm+restart keyholder.

Change 867294 merged by Dzahn:

[operations/puppet@production] scap: add stanza for jenkins-ci and jenkins-releases deploy

https://gerrit.wikimedia.org/r/867294

Change 868736 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] deployment_server: add keyholder/group config for jenkins-ci deploy

https://gerrit.wikimedia.org/r/868736

So.. question here.. We have 2 different "jenkins", jenkins on contint* and jenkins on releases*.

Should there be one keyholder identity for both jenkins deploys or one identity for each of them?

Change 868738 had a related patch set uploaded (by Dzahn; author: Dzahn):

[labs/private@master] keyholder: add fake deployment keys for jenkins deploy

https://gerrit.wikimedia.org/r/868738

Change 868738 merged by Dzahn:

[labs/private@master] keyholder: add fake deployment keys for jenkins deploy

https://gerrit.wikimedia.org/r/868738

If you are ok with "deployment-ci-admins" to be used as the admin group that can deploy jenkins.. then we can merge the above and close this as resolved I think.

please review https://gerrit.wikimedia.org/r/c/operations/puppet/+/868736/2/hieradata/role/common/deployment_server/kubernetes.yaml

Here is the, not immediately obvious, list of people who are in that group:

First, deployment-ci-admins is an alias for "contint_admins_members".

&contint_admins_members [*ops_members, *contint_roots_members,
              bd808, cscott, krinkle, reedy, marktraceur,
              gjg, addshore, taavi]

So that's contint_roots plus some extra members.

Now contint_roots means releng_members and James.

members: &contint_roots_members [*releng_members, jforrester]

and finally:

members: &releng_members [hashar, thcipriani, brennen, dancy, jhuneidi, demon, dduvall, jnuche]

@thcipriani This is a bit like an access request, what do you think about the above?

There is the extra caveat that we have 2 different jenkins sets. the one on contint* but also the ones on releases*.

We may or may not use 2 different keyholder identities for them and releases* servers have other admin groups on them.

releases* has contint-roots but not this one for example.

In T324014#8474913, @Dzahn wrote:

@thcipriani This is a bit like an access request, what do you think about the above?

I don't imagine anyone in the contint-admin group that isn't in the contint-root group would use this key.

There is the extra caveat that we have 2 different jenkins sets. the one on contint* but also the ones on releases*.

We may or may not use 2 different keyholder identities for them and releases* servers have other admin groups on them.

releases* has contint-roots but not this one for example.

Probably for the same reason as above: that server is used for releases, but not broader CI, so nobody outside of contint-root has the need to use it.

I think I'd vote contint-root, but a question I have is: is there a way to add the contint-root group to the deployment server without it granting the privileges from the admin/data/data.yaml file? I guess we'd have to create another group with the same members as contint-root to avoid that is that right?

I think I'd vote contint-root, but a question I have is: is there a way to add the contint-root group to the deployment server without it granting the privileges from the admin/data/data.yaml file? I guess we'd have to create another group with the same members as contint-root to avoid that is that right?

The idea from the beginning was to add a new group, I believe we still want to stick to that. Also, we need to grant access to both contint and releases instances so they can be both deployed from the same process via Scap3.

@Dzahn I can take care of the required remaining changes, all that we needed from your side was the creation of the Keyholder identity. @hashar is already off for the holidays so we won't be rolling this out until early next year. If it's important from your side to close this task right away, please feel free to do so - we can always reopen the task if there are any issues.

@Dzahn sorry, I just saw your patch at https://gerrit.wikimedia.org/r/c/operations/puppet/+/868736/. Please ignore my previous comment and thanks for the patch, I added a couple of comments there.

Gotcha! yea, the -1 is accurate. I will upload another patch for a new group.

Change 869276 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] admin: create new group deployment-jenkins

https://gerrit.wikimedia.org/r/869276

Change 869276 merged by Dzahn:

[operations/puppet@production] admin: create new group deployment-jenkins

https://gerrit.wikimedia.org/r/869276

new admin group deployment-jenkins (gid: 838) has been created on deploy* and releases* servers.

[deploy1002:~] $ grep deployment-jenkins /etc/group
deployment-jenkins:x:838:hashar,thcipriani,brennen,dancy,jhuneidi,demon,dduvall,jnuche,jforrester

[releases1002:~] $ grep deployment-jenkins /etc/group
deployment-jenkins:x:838:hashar,thcipriani,brennen,dancy,jhuneidi,demon,dduvall,jnuche,jforrester

Change 868736 merged by Dzahn:

[operations/puppet@production] deployment_server: add keyholder/group config for jenkins-ci deploy

https://gerrit.wikimedia.org/r/868736

Mentioned in SAL (#wikimedia-operations) [2023-01-04T23:00:03Z] <mutante> deploy1002 - re-arming keyholder T324014

Mentioned in SAL (#wikimedia-operations) [2023-01-04T23:01:09Z] <mutante> deploy2002 - re-arming keyholder T324014

Awesome, I think that covers everything we need. Should be able to try it out soon. Thanks a lot @Dzahn!