| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • LarsWirzenius | T217325 Consider and evaluate possible new CI tooling | |||
| Resolved | • dduvall | T218334 Evaluate Jenkins X |
Event Timeline
Comment Actions
I followed the Getting Started section of the Jenkins X documentation to get it installed locally using minikube.
Summary
Jenkins X feels like an over-engineered kitchen sink that imposes an obtuse and opinionated workflow that will not fit our needs without a lot of customization effort. It expects an installation per team, and I don't wish the installation process on anyone (from Release Engineering or any other team).
Hard pass.
Installation
I installed jx from Homebrew on macOS, version 1.3.991.
$ brew install jx ... $ jx version Failed to retrieve team settings: failed to setup the dev environment for namespace 'default': the server could not find the requested resource (post environments.jenkins.io) - falling back to default settings... Failed to find helm installs: failed to run 'helm list' command in directory '', output: 'Error: could not find tiller' Failed to get helm version: failed to run 'helm version --short' command in directory '', output: 'Client: v2.11.0+g2e55dbe Error: could not find tiller' NAME VERSION jx 1.3.991 Kubernetes cluster v1.13.4 kubectl v1.13.4 git git version 2.6.1 Operating System Mac OS X 10.13.6 build 17G5019
Cluster installation
It took a total of 4 tries to get it installed after multiple failures due to:
- Incompatible versions of local binaries to which jx shells out, in my case helm and kubectl which I upgraded via brew to solve the compatibility issues
- Insufficient CPU and Memory for all of the pods that jx deploys to minikube. First I thought this was purely my fault due to initializing the minikube cluster on my own (with the default 2 CPU and 2G memory settings) but then I recreated the cluster through jx and even its defaults were not enough.
I finally got all pods to fully initialize by running jx create cluster minikube --cpu=4 --memory=4096 --disk-size=80GB.
Options and defaults
There are so many options! Even with the above command, I was presented with options for "Full CD Pipeline or CI and Manual Release" and a slew of GitHub settings. I really got the impression that it both tries to work with too many subsystems and somehow also manages to impose a rigid workflow. The workflow can be changed by developing one's own build pack, but that only seems to give us the ability to setup project configuration defaults, perhaps not change core assumptions of the project.
This command creates a new Kubernetes cluster, installing required local dependencies and provisions the Jenkins X platform
Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on your laptop for users looking to try out Kubernetes or develop with it day-to-day.
Examples:
jx create cluster minikube
Options:
--buildpack='': The name of the build pack to use for the Team
--cleanup-temp-files=true: Cleans up any temporary values.yaml used by helm install [default true]
--cloud-environment-repo='https://github.com/jenkins-x/cloud-environments': Cloud Environments Git repo
-c, --cpu='': Number of CPUs allocated to the Minikube VM. Defaults to 3.
--default-admin-password='': the default admin password to access Jenkins, Kubernetes Dashboard, ChartMuseum and Nexus
--default-admin-username='admin': the default admin username to access Jenkins, Kubernetes Dashboard, ChartMuseum and Nexus
--default-environment-prefix='': Default environment repo prefix, your Git repos will be of the form 'environment-$prefix-$envName'
-s, --disk-size='': Total amount of storage allocated to the Minikube VM. Defaults to 150GB
--docker-registry='': The Docker Registry host or host:port which is used when tagging and pushing images. If not specified it defaults to the internal registry unless there is a better provider default (e.g. ECR on AWS/EKS)
--domain='': Domain to expose ingress endpoints. Example: jenkinsx.io
--draft-client-only=false: Only install draft client
--environment-git-owner='': The Git provider organisation to create the environment Git repositories in
--exposecontroller-pathmode='': The ExposeController path mode for how services should be exposed as URLs. Defaults to using subnets. Use a value of `path` to use relative paths within the domain host such as when using AWS ELB host names
--exposer='Ingress': Used to describe which strategy exposecontroller should use to access applications
--external-ip='': The external IP used to access ingress endpoints from outside the Kubernetes cluster. For bare metal on premise clusters this is often the IP of the Kubernetes master. For cloud installations this is often the external IP of the ingress LoadBalancer.
--git-api-token='': The Git API token to use for creating new Git repositories
--git-private=false: Create new Git repositories as private
--git-provider-kind='': Kind of Git server. If not specified, kind of server will be autodetected from Git provider URL. Possible values: bitbucketcloud, bitbucketserver, gitea, gitlab, github, fakegit
--git-provider-url='https://github.com': The Git server URL to create new Git repositories inside
--git-username='': The Git username to use for creating new Git repositories
--gitops=false: Creates a git repository for the Dev environment to manage the installation, configuration, upgrade and addition of Apps in Jenkins X all via GitOps
--global-tiller=true: Whether or not to use a cluster global tiller
--helm-client-only=false: Only install helm client
--helm-tls=false: Whether to use TLS with helm
--helm3=false: Use helm3 to install Jenkins X which does not use Tiller
-v, --hyperv-virtual-switch='': Additional options for using HyperV with Minikube
--ingress-cluster-role='cluster-admin': The cluster role for the Ingress controller
--ingress-deployment='jxing-nginx-ingress-controller': The name of the Ingress controller Deployment
--ingress-namespace='kube-system': The namespace for the Ingress controller
--ingress-service='jxing-nginx-ingress-controller': The name of the Ingress controller Service
--install-only=false: Force the install command to fail if there is already an installation. Otherwise lets update the installation
--kaniko=false: Use Kaniko for building docker images
--keep-exposecontroller-job=false: Prevents Helm deleting the exposecontroller Job and Pod after running. Useful for debugging exposecontroller logs but you will need to manually delete the job if you update an environment
--kubernetes-version='': Kubernetes version
--local-cloud-environment=false: Ignores default cloud-environment-repo and uses current directory
--local-helm-repo-name='releases': The name of the helm repository for the installed ChartMuseum
-m, --memory='': Amount of RAM allocated to the Minikube VM in MB. Defaults to 4096 MB.
--namespace='jx': The namespace the Jenkins X platform should be installed into
--ng=false: Use the Next Generation Jenkins X features like Prow, Tekton, No Tiller, Vault, Dev GitOps
--no-default-environments=false: Disables the creation of the default Staging and Production environments
--no-gitops-env-apply=false: When using GitOps to create the source code for the development environment and installation, don't run 'jx step env apply' to perform the install
--no-gitops-env-repo=false: When using GitOps to create the source code for the development environment this flag disables the creation of a git repository for the source code
--no-gitops-env-setup=false: When using GitOps to install the development environment this flag skips the post-install setup
--no-gitops-vault=false: When using GitOps to create the source code for the development environment this flag disables the creation of a vault
--no-tiller=false: Whether to disable the use of tiller with helm. If disabled we use 'helm template' to generate the YAML from helm charts then we use 'kubectl apply' to install it to avoid using tiller completely.
--on-premise=false: If installing on an on premise cluster then lets default the 'external-ip' to be the Kubernetes master IP address
--prow=false: Enable Prow to implement Serverless Jenkins and support ChatOps on Pull Requests
--recreate-existing-draft-repos=false: Delete existing helm repos used by Jenkins X under ~/draft/packs
--register-local-helmrepo=false: Registers the Jenkins X ChartMuseum registry with your helm client [default false]
--remote-tiller=true: If enabled and we are using tiller for helm then run tiller remotely in the kubernetes cluster. Otherwise we run the tiller process locally.
--skip-cluster-role=false: Don't enable cluster admin role for user
--skip-ingress=false: Skips the installation of ingress controller. Note that a ingress controller must already be installed into the cluster in order for the installation to succeed
--skip-installation=false: Provision cluster only, don't install Jenkins X into it
--skip-setup-tiller=false: Don't setup the Helm Tiller service - lets use whatever tiller is already setup for us.
--tekton=false: Enables the Tekton pipeline engine (which used to be called knative build pipeline) along with Prow to provide Serverless Jenkins. Otherwise we default to use Knative Build if you enable Prow
--tiller-cluster-role='cluster-admin': The cluster role for Helm's tiller
--tiller-namespace='kube-system': The namespace for the Tiller when using a global tiller
--timeout='6000': The number of seconds to wait for the helm install to complete
--urltemplate='': For ingress; exposers can set the urltemplate to expose
--user-cluster-role='cluster-admin': The cluster role for the current user to be able to administer helm
--username='': The Kubernetes username used to initialise helm. Usually your email address for your Kubernetes account
--vault=false: Sets up a Hashicorp Vault for storing secrets during installation (supported only for GKE)
--version='': The specific platform version to install
--versions-repo='https://github.com/jenkins-x/jenkins-x-versions': Jenkins X versions Git repo
-d, --vm-driver='': VM driver is one of: [hyperkit hyperv kvm kvm2 virtualbox vmwarefusion xhyve]
Usage:
jx create cluster minikube [flags] [options]
Use "jx options" for a list of global command-line options (applies to all commands).Focused on GitHub, no Gerrit support yet
Jenkins X seems built around the GitHub PR model and presents prompts for collecting GitHub account information by default. It needs an auth key so it can automatically create certain repos and add things to your repository. It wasn't clear to me what purpose these repos were supposed to serve (both are randomly named and one is for "staging" and one for "production") and it was annoying that it polluted my GitHub namespace by default.
Gerrit support does not exist yet but is on the roadmap.
Complexity
Overall, this system is very complicated, feels cobbled together (usually decoupled is a good thing but in this case it makes the system very hard to reason about) and obtuse. I couldn't get access to the UIs to evaluate further due to the default nip.io based URLs not working (and trying to customize the services to use NodePort caused the local deployment to fail).
Here are all the things it runs in the cluster. This is per installation which, again, is supposed to be done per team.
$ kubectl get pods NAME READY STATUS RESTARTS AGE build-controller-7c785c95d4-rclvq 1/1 Running 0 9m23s buildnum-75d8548cbf-rc6m7 1/1 Running 0 9m18s crier-99f6b8cd8-jdtgz 1/1 Running 0 9m18s deck-cbb8dfd87-8sk7l 1/1 Running 1 9m18s deck-cbb8dfd87-dz27t 1/1 Running 1 9m18s hook-6bf85c9ccf-7sskm 1/1 Running 0 9m18s hook-6bf85c9ccf-x98j6 1/1 Running 0 9m18s horologium-6dc8bd8bf7-5kszp 1/1 Running 0 9m18s jenkins-x-chartmuseum-67f75c4884-k89jh 1/1 Running 0 7m56s jenkins-x-controllerbuild-588ff4f79-q894r 1/1 Running 0 7m56s jenkins-x-controllercommitstatus-6b4cc4c859-thl7z 1/1 Running 0 7m56s jenkins-x-controllerrole-5dbb6799d-9t9qm 1/1 Running 0 7m56s jenkins-x-controllerteam-6f6985bb6c-x5brr 1/1 Running 0 7m56s jenkins-x-controllerworkflow-7cd5c894b5-ck284 1/1 Running 0 7m56s jenkins-x-docker-registry-589d4d79c4-687vd 1/1 Running 0 7m56s jenkins-x-heapster-fbdccf687-tqmnl 2/2 Running 0 99s jenkins-x-mongodb-5b9957c694-k27lk 1/1 Running 1 7m56s jenkins-x-monocular-api-bb87f69c4-mglq9 1/1 Running 2 7m56s jenkins-x-monocular-prerender-64fb689cfd-wnhlm 1/1 Running 0 7m55s jenkins-x-monocular-ui-857b86b4b4-5x685 1/1 Running 0 7m55s jenkins-x-nexus-57d7dcdb79-bpqpl 1/1 Running 0 7m55s pipeline-845855dd6-4p8sj 1/1 Running 0 9m18s plank-69bcfbf499-8gz9m 1/1 Running 0 9m18s prow-build-5bffcdc654-dt9fk 1/1 Running 0 9m18s sinker-f747df588-jvfkj 1/1 Running 0 9m18s tide-7ddccdcc99-snrzm 1/1 Running 0 9m18s
I'm not sad that I couldn't fully evaluate jx as I was trepidatious that it would clobber something in the blubber repo and commit the results without asking. Hide your repos and run for the hills.