Regarding GitLab, the timeline for the production instance is "June, about 8 months from now". I think our goal for buildpacks is sooner than that, so even if we decide using prod GitLab is the way to go, we'll still need an interim solution (potentially one set up in a Cloud VPS?).

Some intial considerations:

• We would need to make sure that there are appropriate holes in the firewall so GitLab can talk to Toolforge Kubernetes (I'm assuming the CI runners will be able to, but I don't know if that's enough)
• GitLab has its own k8s compatibility timetable, so we'd have to ensure that our k8s upgrade schedule is in sync with the GitLab upgrade schedule
• If we use GitLab then it's one less thing for Cloud Services/Toolforge to independently maintain/administer. It's also one less thing for users to learn.
• Would we allow people to directly use other git hosting services or would we only support our GitLab? It does support automated mirroring of repositories, but that comes with a time delay.

The k8s compatibility timetable you mention lists the very oldest version of k8s that upstream Kubernetes supports is the very newest version that Gitlab supports. That could be a problem for us from a security perspective. At this time, our aim is to upgrade to 1.17 because we are plodding along keeping up with the oldest version, but we may want to get slightly ahead in time, so I'm saying this while we are in a bad state, but we aim to at least track the oldest version that gets backported patches.

I am skeptical that we will be able to directly include the Gitlab Kubernetes tooling from prod in this directly, but it could happen. That would also require a dramatic shift in the security model we have in our k8s system to give Gitlab a relatively free hand inside Toolforge. That's something to think about and contain well (should be doable).

All this does seem like one hosted in WMCS for this purpose alone might be a likely bet if we use Gitlab, but it also might be a bit funny...Gitlab talking to Gitlab...(not entirely unlike our puppet setup).

Just some thoughts.

Based on what it took to get a buildpack tool deployed in toolsbeta, I added what inputs the CD system should get, what exactly it would do, and what I as a user would expect. Please add to it or let me know if I'm missing/overlooking something.

In T265684#6586584, @Bstorm wrote:

The k8s compatibility timetable you mention lists the very oldest version of k8s that upstream Kubernetes supports is the very newest version that Gitlab supports. That could be a problem for us from a security perspective. At this time, our aim is to upgrade to 1.17 because we are plodding along keeping up with the oldest version, but we may want to get slightly ahead in time, so I'm saying this while we are in a bad state, but we aim to at least track the oldest version that gets backported patches.

Also historically we have not done a great job staying up to date with Gerrit upstream releases, hopefully GitLab will be a different story though. I think we should come up with a set of questions/asks to get some more input from RelEng/the GitLab migration team on what exactly we should expect.

I am skeptical that we will be able to directly include the Gitlab Kubernetes tooling from prod in this directly, but it could happen. That would also require a dramatic shift in the security model we have in our k8s system to give Gitlab a relatively free hand inside Toolforge. That's something to think about and contain well (should be doable).

Based on my skim of the group-level docs it seems like we could limit access to just a "Toolforge" group. Do the concerns w/r to the security model apply just to GitLab or would these be for any CD system?

All this does seem like one hosted in WMCS for this purpose alone might be a likely bet if we use Gitlab, but it also might be a bit funny...Gitlab talking to Gitlab...(not entirely unlike our puppet setup).

I'm not sure whether the "learn only one system" advantage of having two GitLabs would outweigh the "wait which GitLab" confusion :P

Fixed up to match what we are doing right now. I've beat gitlab to death and can confirm it's frustrating to make do what we want with the security we want. That said, it can use CNB out of the box with a privileged container (it uses the pack command). However, you have to give it full access and control over a k8s cluster and...no. We aren't giving a web interface that is controlled by tool users full clusteradmin control of the k8s cluster without a lot more safeguards.

That's why gitlab offers nothing more than any other project for us directly. Dedicated clusters for CI are another matter with a customized plugin for Toolforge. Tekton pipelines is very easy to operate in a cluster, and the right trigger could work. I'm building out how to make the security model behave right so that it only does what we want and does not grant unfettered access to a docker repo.