Page MenuHomePhabricator

Document long-term requirements for GitLab job runners
Open, Needs TriagePublic

Description

With a view to our long-term solution for hosting GitLab job runners (beyond the initial pass using WMCS VMs in the project requested at T285913), we should document what we want and need, covering at least:

  • Compute
  • Privacy
  • Elastic demand
  • Data we want for measuring use and performance
  • Whether these can safely run on a third-party platform

(See T287279 for setting up runners on WMCS.)

Event Timeline

I started to collect some thoughts for the long-term GitLab Runner setups here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#Future_Gitlab_Runner_setup_(T286958)

Feel free to add, comment or edit. There are some open questions. So we should chat/talk soon about your goals and requirements for the GitLab Runner setup. Maybe you have some practical experience from https://phabricator.wikimedia.org/T287279 already.

Also ccing @Joe here, because he offered to give some technical input as well.

Tagging Security-Team for awareness, per discussion.

Hello,

The Data-Engineering (aka Analytics) team have been discussing a potential GitLab CI based solution to a current requirement and I thought that this ticket might be a good way to start the conversation about it with ServiceOps.

The current requirement focuses around the development and deployment of Airflow DAGs.

We would like to host a new repository (probably) called airflow-dags on GitLab, but we do not yet know what group/namespace we should use, nor precisely what permissions will be required. Multiple teams will maintain code within the repository, but we have yet to determine exactly what the levels of authorization and automation will be for deployment.

We would like the deployment part of the system to be capable of:

  • accessing our current Airflow instances, which are in the Analytics VLAN.
  • accessing HDFS, which requires the use of Kerberos
  • uploading artefacts (Jars and python wheels) to Archiva

It struck me that a private runner, sited within the analytics VLAN, might be a good way of achieving this result.
I don't think it particularly matters whether it is a shell based runner, or whether it uses the docker executor. Happy to look at either option.

Copying in @Ottomata , @odimitrijevic, @mforns for reference.

I'm happy to make a separate ticket for this, or to discuss elsewhere.
Thanks.

(edit) - I have made a follow-up ticket here: T295045: Allow a shared, protected runner for the data-engineering group in GitLab

The security team would be more than happy to review the documentation when it gets to the draft phase