(https://github.com/wikimedia/operations-puppet/blob/production/modules/k8s/templates/initscripts/kube-apiserver.systemd.erb for the commandline used)

Had same problem with the basic auth provider as well.

there is an ABAC policy file restricting the one account I have to only one namespace, but that also seems to have no effect.

After reading through the source for a bit I'm convinced that this is just because I was using the insecure port. I'll verify shortly.

Insecure port is likely reason for this.

You are setting " --insecure-bind-address=<%= @master_ip %> ".
unless master_ip is 127.0.0.1, that's probably not doing what you want.

@Etune yup, that was the issue. Now at F0909 20:26:59.469630 29907 helpers.go:71] Error in configuration: more than one authentication method found for ; found [token basicAuth], only one is allowed despite the fact that I'm speciying only --token-auth-file.

I had previously used basic auth, and had the same user account there. I wonder if that is 'cached' somewhere (etcd perhaps?) despite the restarts? Doesn't look obvious from the code.

You probably need to edit your .kube/config

Ok, that theory is bunk now. Still trying to figure out why it thinks basica uth is still on.

my local user's .kube/config? doesn't exist...

you probably have a users[i].user.password and a users[i].user.token in that file.

This is client side error, not apiserver side.

Am using:

kubectl --insecure-skip-tls-verify -s https://tools-k8s-master-01.tools.eqiad.wmflabs:6443  --namespace='lolrrit-wm'  --token=='<password>'   get pods

(and had tried with --username and --user as well)

Are you seeing that fatal log message in the apiserver log file?

Nope in the client, when I try out kubectl.

Ok, so if I set the token in .kube/config and then select it with --user... I get:

yuvipanda@tools-k8s-master-01:~$ kubectl --insecure-skip-tls-verify -s https://tools-k8s-master-01.tools.eqiad.wmflabs:6443  --namespace='lolrrit-wm' --user='lolrrit-wm'  get pods
error: couldn't read version from server: the server does not allow access to the requested resource

where the ABAC file is:

yuvipanda@tools-k8s-master-01:~$ sudo cat /etc/kube/abac
{"user": "lolrrit-wm", "namespace": "lolrrit-wm"}

So this should techncially work...

I also couldn't really find any documentation on .kube/config

so if I add another line:

{"user": "lolrrit-wm"}

to the ABAC file it works. I now have to check if the namespace filter is available in the verison of kubernetes we're running (v1.1.0-alpha.1)

Ok, so the code definitely exists (and so do the docs) for namespace field ABAC in the version we are running.

Adding just a `{"namespace": "lolrrit-wm"}' doesn't help things

Hah, looks like I'm running into https://github.com/kubernetes/kubernetes/issues/13097

I've cherry-picked a fix to ^ and it's all good now. Ongoing discussions on the issue and pull request associated with it.

Now to figure out the appropriate bits of ABAC required to let kubelet etc do their jobs :)

It's all good now! We have a 'client-infrastructure' account for kubelet / kube-proxy, and 'admin' and 'namespaced' accounts. This is all controlled from hieradata/common.yaml in the secret repository in /var/lib/git/labs/private on tools-puppetmaster-01. Needs backup, however...