Page MenuHomePhabricator

Build testing k8s setup in ci-staging
Closed, DeclinedPublic

Description

In order to evaluate various technologies for the CI pipeline project, it would be helpful to have some small k8s setup inside a labs project.

There are puppet roles, lets try it in ci-staging.

Details

Related Gerrit Patches:

Event Timeline

dduvall triaged this task as Medium priority.Mar 16 2017, 5:07 PM
dduvall moved this task from Backlog to Build PoC on the Release Pipeline board.

Change 345192 had a related patch set uploaded (by Dduvall):
[operations/puppet@production] [DO NOT MERGE] ci: Experimental k8s cluster for ci

https://gerrit.wikimedia.org/r/345192

So I think I got most of what was required for setup into the submitted puppet patch. The only other manual steps include:

Add linewise entries to /etc/kubernets/tokenauth for:

  • the client-infrastructure account (see k8s_infrastructure_users hieradata in submitted patch)
  • an admin account for use with kubectl on the master
  • format for each account is [token],[username],[uid] format, so e.g. [long random token],admin,admin for the admin account

Configure kubectl for shell users that need admin access, using corresponding details from the tokenfile. E.g. there's a /root/.kube/config at the moment that was created by executing the following:

kubectl config set-credentials admin --token=[admin token in tokenauth file]
kubectl config set-cluster ci-staging --server=https://ci-staging-k8s-master.ci-staging.eqiad.wmflabs
kubectl config set-context ci-staging --cluster=ci-staging --user=admin --namespace=default
kubectl config use-context ci-staging

The resulting /root/.kube/config should look something like this:

root@ci-staging-k8s-master:~# kubectl config view
apiVersion: v1
clusters:
- cluster:
    server: https://ci-staging-k8s-master.ci-staging.eqiad.wmflabs:6443
  name: ci-staging
contexts:
- context:
    cluster: ci-staging
    namespace: default
    user: admin
  name: ci-staging
current-context: ci-staging
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: [admin token in tokenauth file]

Also needed add an annotation to the default namespace to satisfy our custom RegistryEnforcer admin controller:

kubectl edit namespace default

And add:

metadata:
  annotations:
    RunAsUser: "33"
dduvall moved this task from Build PoC to Backlog on the Release Pipeline board.Mar 28 2017, 11:02 PM
greg added a subscriber: greg.Jun 21 2017, 4:45 PM

Tyler thinks Dan said this is done, right? :)

Change 345192 abandoned by Dduvall:
[DO NOT MERGE] ci: Experimental k8s cluster for ci

Reason:
Used for experimentation. We may come back to this later but it will likely require a substantial refactor.

https://gerrit.wikimedia.org/r/345192

dduvall closed this task as Declined.Jul 17 2017, 4:40 PM

A k8s cluster was successfully set up but it was only used for a brief experiment. The k8s puppet modules have since changed and if we do go forward with our own k8s cluster for testing, the patch will require refactoring.