Page MenuHomePhabricator

`helmfile -e staging -i apply` fails for Toolhub due to missing ConfigMap
Closed, ResolvedPublicBUG REPORT

Description

Helmfile attempts to apply all of the kubernetes artifacts generated by the toolhub chart and then rolls back the deployment with this error message:

STDERR:
  Error: release main failed: Deployment.apps "toolhub-main" is invalid: spec.template.spec.containers[2].volumeMounts[0].name: Not found: "toolhub-main-mcrouter-config"

The missing resource is a ConfigMap containing the JSON config for mcrouter in the pod. The output of helmfile -e staging diff appears to show the ConfigMap:

toolhub, toolhub-main-mcrouter-config, ConfigMap (v1) has been added:
-
+ # Source: toolhub/templates/configmap.yaml
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+   name: toolhub-main-mcrouter-config
+   labels:
+     app: toolhub
+     chart: toolhub-0.0.4
+     release: main
+     heritage: Tiller
+ data:
+   config.json: |-
+     {
+       "pools": {
+         "eqiad-servers": {
+           "servers": [
+           "10.64.0.124:11211:ascii:plain",
+           "10.64.0.125:11211:ascii:plain",
+           "10.64.0.64:11211:ascii:plain",
+           "10.64.0.65:11211:ascii:plain",
+           "10.64.16.140:11211:ascii:plain",
+           "10.64.16.21:11211:ascii:plain",
+           "10.64.16.102:11211:ascii:plain",
+           "10.64.16.190:11211:ascii:plain",
+           "10.64.32.133:11211:ascii:plain",
+           "10.64.32.148:11211:ascii:plain",
+           "10.64.32.151:11211:ascii:plain",
+           "10.64.32.153:11211:ascii:plain",
+           "10.64.32.157:11211:ascii:plain",
+           "10.64.32.158:11211:ascii:plain",
+           "10.64.48.90:11211:ascii:plain",
+           "10.64.48.91:11211:ascii:plain",
+           "10.64.48.92:11211:ascii:plain",
+           "10.64.48.93:11211:ascii:plain"
+         ]
+         },
+         "eqiad-servers-failover": {
+           "servers": [
+           "10.64.0.53:11211:ascii:plain",
+           "10.64.32.101:11211:ascii:plain",
+           "10.64.48.32:11211:ascii:plain"
+         ]
+         },
+         "codfw-servers": {
+           "servers": [
+           "10.192.0.83:11211:ascii:plain",
+           "10.192.0.84:11211:ascii:plain",
+           "10.192.0.85:11211:ascii:plain",
+           "10.192.0.86:11211:ascii:plain",
+           "10.192.16.60:11211:ascii:plain",
+           "10.192.16.61:11211:ascii:plain",
+           "10.192.16.62:11211:ascii:plain",
+           "10.192.16.194:11211:ascii:plain",
+           "10.192.32.159:11211:ascii:plain",
+           "10.192.32.40:11211:ascii:plain",
+           "10.192.32.161:11211:ascii:plain",
+           "10.192.32.162:11211:ascii:plain",
+           "10.192.32.163:11211:ascii:plain",
+           "10.192.48.76:11211:ascii:plain",
+           "10.192.48.77:11211:ascii:plain",
+           "10.192.48.78:11211:ascii:plain",
+           "10.192.48.79:11211:ascii:plain",
+           "10.192.48.80:11211:ascii:plain"
+         ]
+         },
+         "codfw-servers-failover": {
+           "servers": [
+           "10.192.0.156:11211:ascii:plain",
+           "10.192.16.147:11211:ascii:plain",
+           "10.192.48.138:11211:ascii:plain"
+         ]
+         },
+         "eqiad-proxies": {
+           "servers": [
+           "10.64.0.124:11211:ascii:plain",
+           "10.64.0.125:11211:ascii:plain",
+           "10.64.0.64:11211:ascii:plain",
+           "10.64.0.65:11211:ascii:plain",
+           "10.64.16.140:11211:ascii:plain",
+           "10.64.16.21:11211:ascii:plain",
+           "10.64.16.102:11211:ascii:plain",
+           "10.64.16.190:11211:ascii:plain",
+           "10.64.32.133:11211:ascii:plain",
+           "10.64.32.148:11211:ascii:plain",
+           "10.64.32.151:11211:ascii:plain",
+           "10.64.32.153:11211:ascii:plain",
+           "10.64.32.157:11211:ascii:plain",
+           "10.64.32.158:11211:ascii:plain",
+           "10.64.48.90:11211:ascii:plain",
+           "10.64.48.91:11211:ascii:plain",
+           "10.64.48.92:11211:ascii:plain",
+           "10.64.48.93:11211:ascii:plain"
+         ]
+         },
+         "eqiad-proxies-failover": {
+           "servers": [
+           "10.64.0.124:11211:ascii:plain",
+           "10.64.0.125:11211:ascii:plain",
+           "10.64.0.64:11211:ascii:plain",
+           "10.64.0.65:11211:ascii:plain",
+           "10.64.16.140:11211:ascii:plain",
+           "10.64.16.21:11211:ascii:plain",
+           "10.64.16.102:11211:ascii:plain",
+           "10.64.16.190:11211:ascii:plain",
+           "10.64.32.133:11211:ascii:plain",
+           "10.64.32.148:11211:ascii:plain",
+           "10.64.32.151:11211:ascii:plain",
+           "10.64.32.153:11211:ascii:plain",
+           "10.64.32.157:11211:ascii:plain",
+           "10.64.32.158:11211:ascii:plain",
+           "10.64.48.90:11211:ascii:plain",
+           "10.64.48.91:11211:ascii:plain",
+           "10.64.48.92:11211:ascii:plain",
+           "10.64.48.93:11211:ascii:plain"
+         ]
+         },
+         "codfw-proxies": {
+           "servers": [
+           "10.192.0.83:11211:ascii:plain",
+           "10.192.0.84:11211:ascii:plain",
+           "10.192.0.85:11211:ascii:plain",
+           "10.192.0.86:11211:ascii:plain",
+           "10.192.16.60:11211:ascii:plain",
+           "10.192.16.61:11211:ascii:plain",
+           "10.192.16.62:11211:ascii:plain",
+           "10.192.16.194:11211:ascii:plain",
+           "10.192.32.159:11211:ascii:plain",
+           "10.192.32.40:11211:ascii:plain",
+           "10.192.32.161:11211:ascii:plain",
+           "10.192.32.162:11211:ascii:plain",
+           "10.192.32.163:11211:ascii:plain",
+           "10.192.48.76:11211:ascii:plain",
+           "10.192.48.77:11211:ascii:plain",
+           "10.192.48.78:11211:ascii:plain",
+           "10.192.48.79:11211:ascii:plain",
+           "10.192.48.80:11211:ascii:plain"
+         ]
+         },
+         "codfw-proxies-failover": {
+           "servers": [
+           "10.192.0.83:11211:ascii:plain",
+           "10.192.0.84:11211:ascii:plain",
+           "10.192.0.85:11211:ascii:plain",
+           "10.192.0.86:11211:ascii:plain",
+           "10.192.16.60:11211:ascii:plain",
+           "10.192.16.61:11211:ascii:plain",
+           "10.192.16.62:11211:ascii:plain",
+           "10.192.16.194:11211:ascii:plain",
+           "10.192.32.159:11211:ascii:plain",
+           "10.192.32.40:11211:ascii:plain",
+           "10.192.32.161:11211:ascii:plain",
+           "10.192.32.162:11211:ascii:plain",
+           "10.192.32.163:11211:ascii:plain",
+           "10.192.48.76:11211:ascii:plain",
+           "10.192.48.77:11211:ascii:plain",
+           "10.192.48.78:11211:ascii:plain",
+           "10.192.48.79:11211:ascii:plain",
+           "10.192.48.80:11211:ascii:plain"
+         ]
+         }
+       },
+       "routes": [
+         {
+           "aliases": [
+             "/eqiad/toolhub/"
+           ],
+           "route": {
+             "failover": "PoolRoute|eqiad-servers-failover",
+             "failover_errors": [
+               "tko"
+             ],
+             "failover_exptime": 600,
+             "normal": "PoolRoute|eqiad-servers",
+             "type": "FailoverWithExptimeRoute"
+           }
+         }
+       ]
+     }

Event Timeline

bd808 triaged this task as High priority.
bd808 created this task.
bd808 moved this task from Backlog to In Progress on the Toolhub board.

Attempting to verify that the ConfigMap is well formed produces an interesting error message:

$ kube_env toolhub staging
$ kubectl apply --validate=true -f /home/bd808/T290283-configmap.yaml
Error from server (Forbidden): error when creating "/home/bd808/T290283-configmap.yaml": configmaps is forbidden: User "toolhub" cannot create resource "configmaps" in API group "" in the namespace "toolhub"

Eh. Not too interesting on further inspection. The "toolhub" user really only has permissions to view things per kubectl auth can-i --list. The service user that tiller runs as has all the create and delete rights.

In Shellbox, I create a configmap with the following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ template "wmf.releasename" . }}-shellbox-config
  labels:
    app: {{ template "wmf.chartname" . }}
    chart: {{ template "wmf.chartid" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  config.json: |-
{{ .Files.Get "config/shellbox-config.json" | indent 4 }}

Then my volume mounts are:

volumeMounts:
  - name: shellbox-config
    mountPath: "/srv/app/config"
    readOnly: true

In Toolhub I see that the volumeMount name is:

volumeMounts:
  - name: {{ template "wmf.releasename" . }}-mcrouter-config
    mountPath: /etc/mcrouter

which expands to "toolhub-main-mcrouter-config". I wonder if dropping the releasename in the volumeMount, so it's just "mcrouter-config" would work.

I actually have no idea why this works in Shellbox, given that the actual configmap name is shellbox-main-shellbox-config.

I actually have no idea why this works in Shellbox, given that the actual configmap name is shellbox-main-shellbox-config.

I looked it up and now I remember. https://kubernetes.io/docs/concepts/storage/volumes/#configmap explains that for each volume you give it a name that maps to the config map's name.

In Shellbox _volumes.tpl I have:

# Additional app-specific volumes.
- name: shellbox-config
  configMap:
      name: {{ template "wmf.releasename" . }}-shellbox-config

Change 716633 had a related patch set uploaded (by Legoktm; author: Legoktm):

[operations/deployment-charts@master] toolhub: Fix mounting of mcrouter-config volume

https://gerrit.wikimedia.org/r/716633

Change 716633 merged by jenkins-bot:

[operations/deployment-charts@master] toolhub: Fix mounting of mcrouter-config volume

https://gerrit.wikimedia.org/r/716633

Thank you very much for figuring out what was missing in my chart @Legoktm!