Currently there are only Runners in WMCS. We first have to adapt this setup to also have a set of Runners in a trusted environment outside of WMCS. The task T295481 is used to create such runners. If we have secure Runners in place we can use this to create additional, special purpose runners (like this one fore data-engineering).

So this task is a little bit blocked until we have created such secure Runners. But I think some things could happen in parallel here, like figuring out the networking requirements with networking folks.

It might be helpful if the runner could use the docker executor, but if that isn't feasible then we would also be happy to use a runner with the shell executor.

From security perspective shell executors should not be used as they don't offer any real separation of jobs and from the Runner. So we should use at least the Docker Executors (see).

Thanks for the reply @Jelto,

Perhaps I or one of my team could help with the set up this particular secure runner, given that it is quite a specialized use case and that it's not going to be used for anything to do with Mediawiki deployments, or anything like that.

I note @brennen's comment here: T292094#7493614

For very specialized runners, I do suspect a bring-your-own approach is best.

So I'm more than happy to work with you on this specialized requirement, if that's feasible.

From security perspective shell executors should not be used as they don't offer any real separation of jobs and from the Runner. So we should use at least the Docker Executors (see).

The docker executor is fine with us too, it's the most convenient and flexible solution by far, but equally we'd be happy to manage the security issues around the shell executor as well, given that we would only be using it for trusted builds.

What's the best thing that I can do to help keep this from being blocked for too long? Should I write a document with a proposal for a data engineering runner setup that your team can review?

Pausing this task, since we are not currently working on it. I think that it would still be useful to have a meeting with the release engineering team to discuss whether or not this is a desired path forward.

Would you be happy to grant me (as an SRE within the team) ownership of that group, so that I could perform this operation as a self-service task?

Confirming here you're an owner of the data-engineering group in gitlab

In T295045#8332237, @thcipriani wrote:

Would you be happy to grant me (as an SRE within the team) ownership of that group, so that I could perform this operation as a self-service task?

Confirming here you're an owner of the data-engineering group in gitlab

Thanks for that @thcipriani - Yes, I have the necessary rights now.
When I originally wrote this ticket, the data-engineering group hadn't yet been moved under the /repos top-level group and I didn't have the privileges at the time. I'll update the description to reflect the changes since then.

Just passing by to +1 this idea.

@BTullis mentioned above the airflow-dags project, but it would also immediately benefit the conda-analytics project, as we are effectively doing the same workaround of copying the Dockerfile into gitlab CI steps. See T321736 for details.