Page MenuHomePhabricator
Create Task
Maniphest T346244

Request access to trusted runners for data-engineering/spark
Closed, ResolvedPublic
Actions

Description

We would like to use the GitLab Trusted Runners for building spark artifacts please.

At the moment we have a build pipeline for spark in the production-images repository. It currently builds and publishes three docker images:

  • A spark-build image, which contains the build state
  • A spark image, which contains a binary spark distribution that is created from spark-build
  • A spark-operator image, which contains the same as spark-image, wih an added go binary to

However, we need to expand on the capabilities of this pipeline by doing the following:

  • Building for multiple minor versions of spark (currently, 3.1 to 3.4)
  • Adding a debian packaging component to help deploy the various different versions of the spark-3.x-yarn-shuffle jar file to our production hosts

I have started work on migrating the spark build pipeline to GitLab-CI in order to achieve these goals.
https://gitlab.wikimedia.org/repos/data-engineering/spark/

However, I won't be able to publish the images until such time as I can use a trusted runner.

In addition to this, I am hitting a 20 minute execution timeout in the wmcs based runners at the moment, since each spark build takes a long time to run.

I would therefore like to be able to access the trusted runners so that we can get past both of these blockers.

Related Objects
Search...

Event Timeline

BTullis created this task.Sep 13 2023, 1:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 13 2023, 1:46 PM
BTullis claimed this task.Sep 13 2023, 1:46 PM
BTullis moved this task from Incoming to In Progress on the Data-Platform-SRE board.
BTullis added a comment.Sep 13 2023, 2:50 PM

I've been following the instruction here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners#Request_access_to_Trusted_Runners
...and I have a couple of observations. Perhaps the docs or the settings need to be updated.

  1. There is a reference to a phabricator template to request access, but the link appears to be incorrect.

image.png (102×945 px, 29 KB)

  1. The instructions say:

To request access, add your project to the projects.json file and open a merge request. Please add the id (go to Settings > General in your project), the name of the project and a reason why access is needed.

It appears that I do not have the required rights to create a new branch on the project, so I can't create a merge request.

(base) btullis@marlin:~/wmf/gitlab-trusted-runner$ git push -u origin add_data_eng_spark 
remote: 
remote: ========================================================================
remote: 
remote: You are not allowed to push code to this project.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
BTullis added a parent task: T344910: Deploy additional yarn shuffler services to support several versions of spark in parallel.Sep 13 2023, 2:50 PM
BTullis removed BTullis as the assignee of this task.Sep 13 2023, 2:53 PM
BTullis moved this task from In Progress to Blocked / Waiting on the Data-Platform-SRE board.
BTullis added projects: Release-Engineering-Team, serviceops.

Removing myself as the assignee, since it appears that I will need assistance from someone in either the Release-Engineering-Team or serviceops teams.

dancy added a comment.Sep 13 2023, 2:54 PM
In T346244#9164100, @BTullis wrote:

It appears that I do not have the required rights to create a new branch on the project, so I can't create a merge request.

I will take care of this for you but in the future to work around something like this you can create a fork of the repo, make the change in your fork, and the create a merge request from your fork into the target repo.

BTullis added a comment.Sep 13 2023, 2:56 PM
In T346244#9164113, @dancy wrote:
In T346244#9164100, @BTullis wrote:

It appears that I do not have the required rights to create a new branch on the project, so I can't create a merge request.

I will take care of this for you but in the future to work around something like this you can create a fork of the repo, make the change in your fork, and the create a merge request from your fork into the target repo.

Ah, great. Many thanks for taking care of it. I'll do that next time.

CodeReviewBot added a project: Patch-For-Review.Sep 13 2023, 2:57 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/44

Add repos/data-engineering/spark to trusted runners

CodeReviewBot added a comment.Sep 13 2023, 2:58 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/44

Add repos/data-engineering/spark to trusted runners

dancy closed this task as Resolved.Sep 13 2023, 3:01 PM
dancy claimed this task.

You're all set.

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2023, 3:10 PM
dancy added a comment.Sep 13 2023, 5:34 PM
In T346244#9164100, @BTullis wrote:

I've been following the instruction here: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners#Request_access_to_Trusted_Runners
...and I have a couple of observations. Perhaps the docs or the settings need to be updated.

  1. There is a reference to a phabricator template to request access, but the link appears to be incorrect.

image.png (102×945 px, 29 KB)

@Jelto, what template did you intend to link to here? Right now it just links to the Phabricator main page.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits