Page MenuHomePhabricator

Request access to trusted runners for data-engineering/spark
Closed, ResolvedPublic

Description

We would like to use the GitLab Trusted Runners for building spark artifacts please.

At the moment we have a build pipeline for spark in the production-images repository. It currently builds and publishes three docker images:

  • A spark-build image, which contains the build state
  • A spark image, which contains a binary spark distribution that is created from spark-build
  • A spark-operator image, which contains the same as spark-image, wih an added go binary to

However, we need to expand on the capabilities of this pipeline by doing the following:

  • Building for multiple minor versions of spark (currently, 3.1 to 3.4)
  • Adding a debian packaging component to help deploy the various different versions of the spark-3.x-yarn-shuffle jar file to our production hosts

I have started work on migrating the spark build pipeline to GitLab-CI in order to achieve these goals.
https://gitlab.wikimedia.org/repos/data-engineering/spark/

However, I won't be able to publish the images until such time as I can use a trusted runner.

In addition to this, I am hitting a 20 minute execution timeout in the wmcs based runners at the moment, since each spark build takes a long time to run.

I would therefore like to be able to access the trusted runners so that we can get past both of these blockers.

Details

TitleReferenceAuthorSource BranchDest Branch
Add repos/data-engineering/spark to trusted runnersrepos/releng/gitlab-trusted-runner!44dancyreview/dancy/add-de-sparkmain
Customize query in GitLab

Event Timeline

I've been following the instruction here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners#Request_access_to_Trusted_Runners
...and I have a couple of observations. Perhaps the docs or the settings need to be updated.

  1. There is a reference to a phabricator template to request access, but the link appears to be incorrect.

image.png (102×945 px, 29 KB)

  1. The instructions say:

To request access, add your project to the projects.json file and open a merge request. Please add the id (go to Settings > General in your project), the name of the project and a reason why access is needed.

It appears that I do not have the required rights to create a new branch on the project, so I can't create a merge request.

(base) btullis@marlin:~/wmf/gitlab-trusted-runner$ git push -u origin add_data_eng_spark 
remote: 
remote: ========================================================================
remote: 
remote: You are not allowed to push code to this project.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
BTullis moved this task from In Progress to Blocked / Waiting on the Data-Platform-SRE board.

Removing myself as the assignee, since it appears that I will need assistance from someone in either the Release-Engineering-Team or serviceops teams.

It appears that I do not have the required rights to create a new branch on the project, so I can't create a merge request.

I will take care of this for you but in the future to work around something like this you can create a fork of the repo, make the change in your fork, and the create a merge request from your fork into the target repo.

It appears that I do not have the required rights to create a new branch on the project, so I can't create a merge request.

I will take care of this for you but in the future to work around something like this you can create a fork of the repo, make the change in your fork, and the create a merge request from your fork into the target repo.

Ah, great. Many thanks for taking care of it. I'll do that next time.

dancy claimed this task.

You're all set.

I've been following the instruction here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners#Request_access_to_Trusted_Runners
...and I have a couple of observations. Perhaps the docs or the settings need to be updated.

  1. There is a reference to a phabricator template to request access, but the link appears to be incorrect.

image.png (102×945 px, 29 KB)

@Jelto, what template did you intend to link to here? Right now it just links to the Phabricator main page.