- Title of session: State of GitLab CI and migration party
- Session description: : A short introduction to the current status of GitLab and GitLab CI at WMF. This talk should give an overview about the new version control and CI tool GitLab. We will take a look the different kind of trust and access requirements which are needed to build artifacts in GitLab. We will also cover the three-tiered trust model of different GitLab Runners, access control, security hardening and what's still open. I'd be happy if you bring your own existing/new project or use-cases that should be migrated to GitLab.
- Username for contact: @Jelto
- Session duration (25 or 50 min): 25 min
- Session type (presentation, workshop, discussion, etc.): Presentation in Small Hacking Room Corner followed by discussion
- Language of session (English, Arabic, etc.): English
- Prerequisites (some Python, etc.):
- basic CI/CD knowledge
- best case: a project you want to start or migrate to GitLab
- Any other details to share?:
- Sign up at: https://gitlab.wikimedia.org/
- https://wikitech.wikimedia.org/wiki/GitLab
- https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner
- https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner
- https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner
- https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/gitlab_runner/
- Interested? Add your username below:
Description
Description
Event Timeline
Comment Actions
@Jelto: Thanks for participating in the Hackathon! We hope you had a great time.
- If this session / event took place: Please change the task status to resolved via the Add Action... → Change Status dropdown.
- If there are session notes (e.g. on Etherpad or a wiki page), or if the session was recorded, please make sure these resources are linked from this task.
- If there are specific follow-up tasks from this session / event: Please create dedicated tasks and add another active project tag to those tasks, so others can find those tasks (as likely nobody in the future will look at the Hackathon workboard when trying to find something they are interested in).
- In this session / event did not take place: Please set the task status to declined.
Thank you,
Phabricator housekeeping service
Comment Actions
Session notes:
== State of GitLab CI and migration party ==
Date & time: Friday, May 19th at 14:30 pm EEST / 11:30 am UTC
== Relevant links ==
* Phabricator task: https://phabricator.wikimedia.org/T336266
== Participants ==
* around 20 people
== Notes ==
GitLab Short Overview
- community consultation happened and revealed dissatisfaction with Gerrit
- steep learning curve
- evaluated different tools
- gitlab has lower friction to create new repos, easier to set up
(spirit of gerrit interrupted by blowing over umbrellas)
-small amount of traction now been running for 1.5 years publicly
- 1100 projects and 700 users
- 4000 merge requests created
- gitlab.wikimedia.org -> login with wikitech
- not fully open, need to get on wait list and then approved
- takes about a day to get approved
Gitlab CI
- gitlab = server
- gitlab-runner are the individual CI jobs
- gitlab-runners can live in different environments
- can run it locally in docker
- can be in a deticated VM in docker or shell
- or in Kubernetes
CI is more difficult
- we want to be open and accept code from all over but we also want to have reviewed code
- needs to be a balance between running the code pipeline and accepting all code
- this is not a new problem
- in the past we:
- had trsuted contibuters
- exceessive code review
- multi stage CI jobs
- self hosting
- Ended up with a multi tier system of runners
- personal gitlab runner
- no guarantee of maintenance
- on your machine or wherever you want to use
- shared runners
- available for most projects
- WMCS and Digital Ocean
- trusted runners
- only available for certain workloads that need more sercuity
Shared Runners
- available for most projects and execute any CI job
- Executed on wikimedia cloud or digital ocean
Trusted Runners
- example pushing images to docker registry
- more protected and secured
- have to explicitly allow the project
- need maintainer permissions to create jobs
- in gerrit this would be being able to +2
security hardening
- run in our data center
- not everyone has access - need production access
- only certain tags are allowed there is a tag
Self managed runners
- they have to be maintained by you/your team and take care of securiy updates on your own
Job scheduling
- one project can create multiple kinds of jobs
- for example could have a volunteer working on a feature branch it could use a different kind
- would get run on shared runners first then when that passes runs on the trusted runner
- job scheduling depends on branch, projects and tags
- feature branch runs on shared runners, as you don't know if its safe to execute in production
- once it was reviewed and merged, it will be run on the protected
- you can use CI tags to influence job scheduling
- tags:
- trusted
-if you specify trusted and its not in the list of trusted projects, it will be stuck and not run
Ci tooling
- Ci can be enhanced by including .gitlab-ci.yml
- example for docker in kokkuri project
- currently only supports building images, working on adding documentation and programming language support
- there is a pipleline converter from gerrit/jenkins pipeline
- some bots have been updated
next steps
- onboarding of new projects
- more tooling and templating is needed for more CI use-cases
- open shared runners to all projectsQuestions:
How do trusted runners work together. if one has secrets can the others use it? - currently there are 6 hosts in production - They consume secrets from GitLab "CI variables", so all Trusted Runners can use it - by default they run inside a docker container, would need to figure out how to break out of the container - it may be worth looking into this in the future - start from vms and look from there What is the support for secrets? what do you do with secret variables? - gitlab supports project secrets where only the maintainers and owners can specify certain variables - and then those variables are available in the jobs - long term with sensitive secrets maybe think about other secret tools - having all the secrest in one place is a large single point of failure - you can add them from the UI and the command line
Slides: