Page MenuHomePhabricator

Replace kubeyaml in deployment-charts CI
Closed, ResolvedPublic

Description

A friend pointed out https://github.com/yannh/kubeconform as replacement for kubeyaml and I did a quick test on my local machine:

k8s 1.16 only, kubeyaml:
$ time rake check_admin
real    1m58.262s
user    9m44.107s
sys     0m43.407s

k8s 1.16 only, kubeconform, warm cache:
$ time rake check_admin
real    0m36.026s
user    0m42.240s
sys     0m8.257s

k8s 1.16 only, kubeconform, cold cache:
$ time rake check_admin
real    0m37.283s
user    0m43.660s
sys     0m8.819s

I had to build the schema for CRDs we use (which was pretty straight forward):

mkdir -p /var/tmp/kubeconform/{schema,cache}; cd /var/tmp/kubeconform/schema
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/calico-crds/templates/crds.yaml
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/cfssl-issuer-crds/templates/crds.yaml
helm template -s templates/crds.yaml --set installCRDs=true wmf-stable/cert-manager | python3 openapi2jsonschema.py /dev/stdin
helm template -s templates/crds.yaml wmf-stable/knative-serving-crds  | ./openapi2jsonschema.py /dev/stdin
helm template -s templates/kserve.yaml wmf-stable/kserve  | ./openapi2jsonschema.py /dev/stdin
python3 openapi2jsonschema.py https://github.com/istio/istio/raw/1.9.5/manifests/charts/base/crds/crd-all.gen.yaml

And replaced the kubeyaml call in asset.rb (removing the custom splitting and threading code completely) with something like:

kubeconform -cache /var/tmp/kubeconform/cache -kubernetes-version #{versions} \
  -strict -summary \
  -schema-location default \
  -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json" \
  -schema-location '/var/tmp/kubeconform-schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
  -skip CustomResourceDefinition

# To have CustomResourceDefinition checked as well, the non standalone schema has to be passed (after default), like:
kubeconform \
  -schema-location default \
  -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json" \
  -schema-location '/var/tmp/kubeconform/schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
  -summary \
  -strict \
  -kubernetes-version 1.16.15

Skipping CRDs because of https://github.com/yannh/kubeconform/issues/100
There also is a project building upon this that allows spec validation against custom policies which might be interesting: https://github.com/datreeio/datree

Event Timeline

JMeybohm created this task.

Change 791794 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Replace kubeyaml with kubeconform (if available)

https://gerrit.wikimedia.org/r/791794

Looking at datree, it's very interesting it can be installed as an helm plugin, so that it can be just used on helm charts from helm itself; however I don't think it can be integrated with helmfile, so we should use it as a standalone cli tool on the generated yaml anyways.

Looking at datree, it's very interesting it can be installed as an helm plugin, so that it can be just used on helm charts from helm itself; however I don't think it can be integrated with helmfile, so we should use it as a standalone cli tool on the generated yaml anyways.

I agree but I don't plan on integrating that right now as it seems like more work than just replacing kubeyaml.

Change 792267 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove null creationTimestamp from CRDs

https://gerrit.wikimedia.org/r/792267

JMeybohm raised the priority of this task from Low to High.

Change 792577 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Add a rake task to generate JSON schema for chart CRDs on the fly

https://gerrit.wikimedia.org/r/792577

kubeconform debian package is ready as well (needs gerrit repo etc.) but I'm not sure about the best way to deal with the kubernetes json schema (the repo is quite big, 14GB on disk).

We could:

  1. Create a separeate repo and commit just the versions we need
  2. Create a full mirror from github

For shipping to CI runners etc.

  1. Build a separate debian package containing the schema we need
  2. Always fetch from network (gerrit) using kubeconform's cache (so only one fetch per CI run)

I guess fetching from gerrit is not ideal, especially as we don't change kubernetes versions very often. So a kubernetes-json-schema debian package would be better idea. A full github mirror does not seem needed as well. So maybe just something like the repack script we use for go packages that automates the sparse checkout and commit for new k8s versions?

Sparse checkouts from github do work (still 163MB) for manual import:

git clone --filter=blob:none --no-checkout https://github.com/yannh/kubernetes-json-schema.git
cd kubernetes-json-schema/
git sparse-checkout set --cone
git checkout master
git sparse-checkout set v1.16.15 v1.16.15-standalone-strict v1.19.16 v1.19.16-standalone-strict v1.23.6 v1.23.6-standalone-strict

Change 792999 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/kubeconform@master] Add debian directory

https://gerrit.wikimedia.org/r/792999

Change 793028 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] Debian glue for operations/debs/kubeconform

https://gerrit.wikimedia.org/r/793028

Change 793028 merged by jenkins-bot:

[integration/config@master] Debian glue for operations/debs/kubeconform

https://gerrit.wikimedia.org/r/793028

kubeconform debian package is ready as well (needs gerrit repo etc.) but I'm not sure about the best way to deal with the kubernetes json schema (the repo is quite big, 14GB on disk).

We could:

  1. Create a separeate repo and commit just the versions we need
  2. Create a full mirror from github

For shipping to CI runners etc.

  1. Build a separate debian package containing the schema we need
  2. Always fetch from network (gerrit) using kubeconform's cache (so only one fetch per CI run)

I guess fetching from gerrit is not ideal, especially as we don't change kubernetes versions very often. So a kubernetes-json-schema debian package would be better idea. A full github mirror does not seem needed as well. So maybe just something like the repack script we use for go packages that automates the sparse checkout and commit for new k8s versions?

Sparse checkouts from github do work (still 163MB) for manual import:

git clone --filter=blob:none --no-checkout https://github.com/yannh/kubernetes-json-schema.git
cd kubernetes-json-schema/
git sparse-checkout set --cone
git checkout master
git sparse-checkout set v1.16.15 v1.16.15-standalone-strict v1.19.16 v1.19.16-standalone-strict v1.23.6 v1.23.6-standalone-strict

Frankly I'd say we can keep this relatively simple. Building a debian package doesn't relly give us any advantage as I doubt we'd use it anywhere but in the helm-linter image.

I would propose instead you create a local repo on either gitlab or gerrit, with a script allowing to checkout just the stuff we want from upstream, then we clone it inside the image.

Frankly I'd say we can keep this relatively simple. Building a debian package doesn't relly give us any advantage as I doubt we'd use it anywhere but in the helm-linter image.

I would propose instead you create a local repo on either gitlab or gerrit, with a script allowing to checkout just the stuff we want from upstream, then we clone it inside the image.

I thought about having the tools and schema available on local machines. But you're probably right that its enough to have the schema in the helm-linter image. I'll create a mirror and ship the script as part of the kubconform debian package, so users can decide to fetch the schema locally. Thanks!

Change 793494 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] helm-linter: Switch from kubeyaml to kubeconform for validation

https://gerrit.wikimedia.org/r/793494

Change 793494 merged by jenkins-bot:

[integration/config@master] helm-linter: Add kubeconform for kuberntes spec validation

https://gerrit.wikimedia.org/r/793494

Change 793506 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] jjb: update helm-linter job to releng/helm-linter:0.4.0

https://gerrit.wikimedia.org/r/793506

Change 793509 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Add crds.yaml fixtures to charts and istio schema

https://gerrit.wikimedia.org/r/793509

Change 792577 abandoned by JMeybohm:

[operations/deployment-charts@master] Add a rake task to generate JSON schema for chart CRDs on the fly

Reason:

suqashed

https://gerrit.wikimedia.org/r/792577

Change 792999 merged by jenkins-bot:

[operations/debs/kubeconform@master] Add debian directory

https://gerrit.wikimedia.org/r/792999

Mentioned in SAL (#wikimedia-operations) [2022-05-20T07:52:03Z] <jayme> imported kubeconform 0.4.13-1 to buster-,bullseye-wikimedia - T306165

Change 793506 merged by jenkins-bot:

[integration/config@master] jjb: update helm-linter job to releng/helm-linter:0.4.0

https://gerrit.wikimedia.org/r/793506

Change 792267 merged by jenkins-bot:

[operations/deployment-charts@master] Remove null creationTimestamp from CRDs

https://gerrit.wikimedia.org/r/792267

Change 793509 merged by jenkins-bot:

[operations/deployment-charts@master] Add crds.yaml fixtures to charts and istio schema

https://gerrit.wikimedia.org/r/793509

Change 791794 merged by jenkins-bot:

[operations/deployment-charts@master] Replace kubeyaml with kubeconform (if available)

https://gerrit.wikimedia.org/r/791794

We're now validating with kubeconform