A friend pointed out https://github.com/yannh/kubeconform as replacement for kubeyaml and I did a quick test on my local machine:
```
k8s 1.16 only, kubeyaml:
$ time rake check_admin
real 1m58.262s
user 9m44.107s
sys 0m43.407s
k8s 1.16 only, kubeconform, warm cache:
$ time rake check_admin
real 0m36.026s
user 0m42.240s
sys 0m8.257s
k8s 1.16 only, kubeconform, cold cache:
$ time rake check_admin
real 0m37.283s
user 0m43.660s
sys 0m8.819s
```
I had to build the schema for CRDs we use (which was pretty straight forward):
```
mkdir -p /var/tmp/kubeconform/{schema,cache}; cd /var/tmp/kubeconform/schema
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/calico-crds/templates/crds.yaml
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/cfssl-issuer-crds/templates/crds.yaml
helm template -s templates/crds.yaml --set installCRDs=true wmf-stable/cert-manager | python3 openapi2jsonschema.py /dev/stdin
helm template -s templates/crds.yaml wmf-stable/knative-serving-crds | ./openapi2jsonschema.py /dev/stdin
python3 openapi2jsonschema.py https://github.com/istio/istio/raw/1.9.5/manifests/charts/base/crds/crd-all.gen.yaml
```
And replaced the kubeyaml call in asset.rb (removing the custom splitting and threading code completely) with something like:
```
kubeconform -cache /var/tmp/kubeconform/cache -kubernetes-version #{versions} \
-strict -summary \
-schema-location default \
-schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json" \
-schema-location '/var/tmp/kubeconform-schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
-skip CustomResourceDefinition
# To have CustomResourceDefinition checked as well, the non standalone schema has to be passed (after default), like:
kubeconform \
-schema-location default \
-schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json" \
-schema-location '/var/tmp/kubeconform/schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
-summary \
-strict \
-kubernetes-version 1.16.15
```
Skipping CRDs because of https://github.com/yannh/kubeconform/issues/100
There also is a project building upon this that allows spec validation against custom policies which might be interesting: https://github.com/datreeio/datree