A friend pointed out https://github.com/yannh/kubeconform as replacement for kubeyaml and I did a quick test on my local machine:
```
k8s 1.16 only, kubeyaml:
$ time rake check_admin
real 1m58.262s
user 9m44.107s
sys 0m43.407s
k8s 1.16 only, kubeconform, warm cache:
$ time rake check_admin
real 0m36.026s
user 0m42.240s
sys 0m8.257s
k8s 1.16 only, kubeconform, cold cache:
$ time rake check_admin
real 0m37.283s
user 0m43.660s
sys 0m8.819s
```
I had to build the schema for CRDs we use (which was pretty straight forward:
```
mkdir /tmp/kubeconform-crd-schema; cd /tmp/kubeconform-crd-schema
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/calico-crds/templates/crds.yaml
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/cfssl-issuer-crds/templates/crds.yaml
helm template -s templates/crds.yaml --set installCRDs=true wmf-stable/cert-manager | python3 openapi2jsonschema.py -
```
And replaced the kubeyaml call in asset.rb with something like:
```
kubeconform -cache /tmp/kubeconform-cache -kubernetes-version #{versions} \
-strict -summary \
-schema-location default -schema-location '/tmp/kubeconform-crd-schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
-skip CustomResourceDefinition
```
Skipping CRDs because of https://github.com/yannh/kubeconform/issues/100
There also is a project building upon this that allows spec validation against custom policies which might be interesting: https://github.com/datreeio/datree