Page MenuHomePhabricator
Create Task
Maniphest T306165

Replace kubeyaml in deployment-charts CI
Closed, ResolvedPublic
Actions

Description

A friend pointed out https://github.com/yannh/kubeconform as replacement for kubeyaml and I did a quick test on my local machine:

k8s 1.16 only, kubeyaml:
$ time rake check_admin
real    1m58.262s
user    9m44.107s
sys     0m43.407s

k8s 1.16 only, kubeconform, warm cache:
$ time rake check_admin
real    0m36.026s
user    0m42.240s
sys     0m8.257s

k8s 1.16 only, kubeconform, cold cache:
$ time rake check_admin
real    0m37.283s
user    0m43.660s
sys     0m8.819s

I had to build the schema for CRDs we use (which was pretty straight forward):

mkdir -p /var/tmp/kubeconform/{schema,cache}; cd /var/tmp/kubeconform/schema
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/calico-crds/templates/crds.yaml
python3 openapi2jsonschema.py ~/code/wmf/operations/deployment-charts/charts/cfssl-issuer-crds/templates/crds.yaml
helm template -s templates/crds.yaml --set installCRDs=true wmf-stable/cert-manager | python3 openapi2jsonschema.py /dev/stdin
helm template -s templates/crds.yaml wmf-stable/knative-serving-crds  | ./openapi2jsonschema.py /dev/stdin
helm template -s templates/kserve.yaml wmf-stable/kserve  | ./openapi2jsonschema.py /dev/stdin
python3 openapi2jsonschema.py https://github.com/istio/istio/raw/1.9.5/manifests/charts/base/crds/crd-all.gen.yaml

And replaced the kubeyaml call in asset.rb (removing the custom splitting and threading code completely) with something like:

kubeconform -cache /var/tmp/kubeconform/cache -kubernetes-version #{versions} \
  -strict -summary \
  -schema-location default \
  -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json" \
  -schema-location '/var/tmp/kubeconform-schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
  -skip CustomResourceDefinition

# To have CustomResourceDefinition checked as well, the non standalone schema has to be passed (after default), like:
kubeconform \
  -schema-location default \
  -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}/{{ .ResourceKind }}{{ .KindSuffix }}.json" \
  -schema-location '/var/tmp/kubeconform/schema/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' \
  -summary \
  -strict \
  -kubernetes-version 1.16.15

Skipping CRDs because of https://github.com/yannh/kubeconform/issues/100
There also is a project building upon this that allows spec validation against custom policies which might be interesting: https://github.com/datreeio/datree

Details

SubjectRepoBranchLines +/-
cfssl-issuer-crds: Bump chart versionoperations/deployment-chartsmaster+1 -1
Replace kubeyaml with kubeconform (if available)operations/deployment-chartsmaster+328 -9
Add crds.yaml fixtures to charts and istio schemaoperations/deployment-chartsmaster+8 K -0
Remove null creationTimestamp from CRDsoperations/deployment-chartsmaster+0 -4
jjb: update helm-linter job to releng/helm-linter:0.4.0integration/configmaster+1 -1
helm-linter: Add kubeconform for kuberntes spec validationintegration/configmaster+10 -2
Add debian directoryoperations/debs/kubeconformmaster+173 -0
Add a rake task to generate JSON schema for chart CRDs on the flyoperations/deployment-chartsmaster+8 K -3
Debian glue for operations/debs/kubeconformintegration/configmaster+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm triaged this task as Low priority.Apr 14 2022, 8:29 AM
JMeybohm created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2022, 8:29 AM
JMeybohm updated the task description. (Show Details)Apr 14 2022, 8:30 AM
JMeybohm mentioned this in T307943: Update Kubernetes clusters to v1.23.May 9 2022, 5:10 PM
JMeybohm updated the task description. (Show Details)May 13 2022, 2:43 PM
gerritbot added a comment.May 15 2022, 5:14 PM

Change 791794 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Replace kubeyaml with kubeconform (if available)

https://gerrit.wikimedia.org/r/791794

gerritbot added a project: Patch-For-Review.May 15 2022, 5:14 PM
JMeybohm updated the task description. (Show Details)May 16 2022, 10:19 AM
Joe subscribed.May 16 2022, 10:57 AM

Looking at datree, it's very interesting it can be installed as an helm plugin, so that it can be just used on helm charts from helm itself; however I don't think it can be integrated with helmfile, so we should use it as a standalone cli tool on the generated yaml anyways.

JMeybohm added a comment.May 16 2022, 2:46 PM
In T306165#7930435, @Joe wrote:

Looking at datree, it's very interesting it can be installed as an helm plugin, so that it can be just used on helm charts from helm itself; however I don't think it can be integrated with helmfile, so we should use it as a standalone cli tool on the generated yaml anyways.

I agree but I don't plan on integrating that right now as it seems like more work than just replacing kubeyaml.

JMeybohm updated the task description. (Show Details)May 16 2022, 2:51 PM
JMeybohm updated the task description. (Show Details)May 16 2022, 2:54 PM
gerritbot added a comment.May 16 2022, 6:22 PM

Change 792267 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove null creationTimestamp from CRDs

https://gerrit.wikimedia.org/r/792267

JMeybohm claimed this task.May 17 2022, 6:41 AM
JMeybohm raised the priority of this task from Low to High.
gerritbot added a comment.May 17 2022, 10:17 AM

Change 792577 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Add a rake task to generate JSON schema for chart CRDs on the fly

https://gerrit.wikimedia.org/r/792577

JMeybohm added a comment.May 17 2022, 5:25 PM

kubeconform debian package is ready as well (needs gerrit repo etc.) but I'm not sure about the best way to deal with the kubernetes json schema (the repo is quite big, 14GB on disk).

We could:

  1. Create a separeate repo and commit just the versions we need
  2. Create a full mirror from github

For shipping to CI runners etc.

  1. Build a separate debian package containing the schema we need
  2. Always fetch from network (gerrit) using kubeconform's cache (so only one fetch per CI run)

I guess fetching from gerrit is not ideal, especially as we don't change kubernetes versions very often. So a kubernetes-json-schema debian package would be better idea. A full github mirror does not seem needed as well. So maybe just something like the repack script we use for go packages that automates the sparse checkout and commit for new k8s versions?

Sparse checkouts from github do work (still 163MB) for manual import:

git clone --filter=blob:none --no-checkout https://github.com/yannh/kubernetes-json-schema.git
cd kubernetes-json-schema/
git sparse-checkout set --cone
git checkout master
git sparse-checkout set v1.16.15 v1.16.15-standalone-strict v1.19.16 v1.19.16-standalone-strict v1.23.6 v1.23.6-standalone-strict
gerritbot added a comment.May 18 2022, 9:35 AM

Change 792999 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/kubeconform@master] Add debian directory

https://gerrit.wikimedia.org/r/792999

gerritbot added a comment.May 18 2022, 11:39 AM

Change 793028 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] Debian glue for operations/debs/kubeconform

https://gerrit.wikimedia.org/r/793028

JMeybohm added a parent task: T307943: Update Kubernetes clusters to v1.23.May 18 2022, 1:40 PM
gerritbot added a comment.May 18 2022, 7:30 PM

Change 793028 merged by jenkins-bot:

[integration/config@master] Debian glue for operations/debs/kubeconform

https://gerrit.wikimedia.org/r/793028

Joe added a comment.May 19 2022, 8:58 AM
In T306165#7935694, @JMeybohm wrote:

kubeconform debian package is ready as well (needs gerrit repo etc.) but I'm not sure about the best way to deal with the kubernetes json schema (the repo is quite big, 14GB on disk).

We could:

  1. Create a separeate repo and commit just the versions we need
  2. Create a full mirror from github

For shipping to CI runners etc.

  1. Build a separate debian package containing the schema we need
  2. Always fetch from network (gerrit) using kubeconform's cache (so only one fetch per CI run)

I guess fetching from gerrit is not ideal, especially as we don't change kubernetes versions very often. So a kubernetes-json-schema debian package would be better idea. A full github mirror does not seem needed as well. So maybe just something like the repack script we use for go packages that automates the sparse checkout and commit for new k8s versions?

Sparse checkouts from github do work (still 163MB) for manual import:

git clone --filter=blob:none --no-checkout https://github.com/yannh/kubernetes-json-schema.git
cd kubernetes-json-schema/
git sparse-checkout set --cone
git checkout master
git sparse-checkout set v1.16.15 v1.16.15-standalone-strict v1.19.16 v1.19.16-standalone-strict v1.23.6 v1.23.6-standalone-strict

Frankly I'd say we can keep this relatively simple. Building a debian package doesn't relly give us any advantage as I doubt we'd use it anywhere but in the helm-linter image.

I would propose instead you create a local repo on either gitlab or gerrit, with a script allowing to checkout just the stuff we want from upstream, then we clone it inside the image.

JMeybohm added a comment.May 19 2022, 10:44 AM
In T306165#7941085, @Joe wrote:

Frankly I'd say we can keep this relatively simple. Building a debian package doesn't relly give us any advantage as I doubt we'd use it anywhere but in the helm-linter image.

I would propose instead you create a local repo on either gitlab or gerrit, with a script allowing to checkout just the stuff we want from upstream, then we clone it inside the image.

I thought about having the tools and schema available on local machines. But you're probably right that its enough to have the schema in the helm-linter image. I'll create a mirror and ship the script as part of the kubconform debian package, so users can decide to fetch the schema locally. Thanks!

gerritbot added a comment.May 19 2022, 3:25 PM

Change 793494 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] helm-linter: Switch from kubeyaml to kubeconform for validation

https://gerrit.wikimedia.org/r/793494

gerritbot added a comment.May 19 2022, 4:10 PM

Change 793494 merged by jenkins-bot:

[integration/config@master] helm-linter: Add kubeconform for kuberntes spec validation

https://gerrit.wikimedia.org/r/793494

gerritbot added a comment.May 19 2022, 4:13 PM

Change 793506 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] jjb: update helm-linter job to releng/helm-linter:0.4.0

https://gerrit.wikimedia.org/r/793506

gerritbot added a comment.May 19 2022, 4:18 PM

Change 793509 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Add crds.yaml fixtures to charts and istio schema

https://gerrit.wikimedia.org/r/793509

gerritbot added a comment.May 19 2022, 4:19 PM

Change 792577 abandoned by JMeybohm:

[operations/deployment-charts@master] Add a rake task to generate JSON schema for chart CRDs on the fly

Reason:

suqashed

https://gerrit.wikimedia.org/r/792577

gerritbot added a comment.May 20 2022, 7:35 AM

Change 792999 merged by jenkins-bot:

[operations/debs/kubeconform@master] Add debian directory

https://gerrit.wikimedia.org/r/792999

Stashbot added a comment.May 20 2022, 7:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-20T07:52:03Z] <jayme> imported kubeconform 0.4.13-1 to buster-,bullseye-wikimedia - T306165

gerritbot added a comment.May 20 2022, 8:39 AM

Change 793506 merged by jenkins-bot:

[integration/config@master] jjb: update helm-linter job to releng/helm-linter:0.4.0

https://gerrit.wikimedia.org/r/793506

gerritbot added a comment.Jun 1 2022, 9:50 AM

Change 792267 merged by jenkins-bot:

[operations/deployment-charts@master] Remove null creationTimestamp from CRDs

https://gerrit.wikimedia.org/r/792267

gerritbot added a comment.Jun 1 2022, 9:51 AM

Change 793509 merged by jenkins-bot:

[operations/deployment-charts@master] Add crds.yaml fixtures to charts and istio schema

https://gerrit.wikimedia.org/r/793509

gerritbot added a comment.Jun 1 2022, 9:51 AM

Change 791794 merged by jenkins-bot:

[operations/deployment-charts@master] Replace kubeyaml with kubeconform (if available)

https://gerrit.wikimedia.org/r/791794

JMeybohm closed this task as Resolved.Jun 2 2022, 1:55 PM

We're now validating with kubeconform

JMeybohm mentioned this in T316348: Remove kubeyaml from deployment-charts CI.Aug 26 2022, 1:25 PM
Clement_Goubert changed the status of subtask T316348: Remove kubeyaml from deployment-charts CI from Open to In Progress.Oct 14 2022, 1:54 PM
Jdforrester-WMF closed subtask T316348: Remove kubeyaml from deployment-charts CI as Resolved.Oct 14 2022, 8:54 PM
gerritbot added a comment.Nov 9 2022, 10:11 AM

Change 854966 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] cfssl-issuer-crds: Bump chart version

https://gerrit.wikimedia.org/r/854966

gerritbot added a comment.Nov 9 2022, 10:38 AM

Change 854966 merged by jenkins-bot:

[operations/deployment-charts@master] cfssl-issuer-crds: Bump chart version

https://gerrit.wikimedia.org/r/854966

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL