opened upstream ticket: https://github.com/kyverno/kyverno/issues/10458

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/142

helpers: add toolforge_kyverno_load_many_resources.sh

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/337

kyverno: reintroduce resource limits

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/142

helpers: add toolforge_kyverno_load_many_resources.sh

I tested this:

• increased the number of cluster nodes in lima-kilo https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/commit/64b49733fc0e56b4d652141074f5ba336c07081e
• reintroduced kyverno resource limits https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/337
• created 4000 kyverno policies in one namespace, with this script https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/commit/837c7a113235663a60288d682cf861033d7d2669
• try to create a job via toolforge job, and see both the kubernetes control plane components (api-server, controller manager, etc) and kyverno itself crash, making the cluster unusable

I had the hope that introducing resource limits on kyverno would reduce the blast radious.

I have a very bad feeling about this.

I don't think we can move forward with this software :-(

I sent additional information to upstream, in particular I shared with them how to reproduce the problem, in case they are interested:

https://github.com/kyverno/kyverno/issues/10458#issuecomment-2178997459

Another data point: yesterday @fnegri pointed me to their slack channel #kyverno. In the channel, I saw an event invitation for the same day about a new feature in kyverno that aparently deals with performance and cluster overload.

See https://community.cncf.io/events/details/cncf-cncf-online-programs-presents-cloud-native-live-kyvernos-report-server-a-new-approach-to-policy-report-management/

Managing policy and governance in busy Kubernetes clusters was difficult due to the high volume of policy reports, cluster policy reports, and ephemeral reports generated by Kyverno. This caused overloading of the API server and etcd, leading to poor cluster performance. Kyverno's new Reports Server addresses this issue by offloading these reports to a separate database, resulting in a 70% reduction in etcd consumption. Attend this session to discover how the Kyverno team tackled this complex problem using API Aggregation and the advantages of storing reports in a dedicated database.

aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/149

helpers/toolforge_kyverno_load_many_resources.sh: be more realistic

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/149

helpers/toolforge_kyverno_load_many_resources.sh: be more realistic

I did further tests, including:

• having a more realistic load test script: https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/blob/825be38c17b849745d878e593b2d90eb7e0ac466/helpers/toolforge_kyverno_load_many_resources.sh
• increased base memory for the lima-kilo lima-vm from 8GiB to 16GiB
• on the kyverno side, applied upstream recommendations regarding optional services, resource limits and replicas, see https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/f4995ab9c9b2d85e64110134f646e0cc62838fba/components/kyverno/values/common/settings.yaml

With all this, I was able to create 4000 policy resources, each on its own namespace, without a single crash on any of the relevant components, with a clean run of the functional tests:

local.tf-test@lima-kilo:~$ toolforge_run_functional_tests.sh 
builds-api/build-smoke-test.bats
 ✓ start build [449]
 ✓ list build [462]
 ✓ tail logs and wait (slow) [176648]
 ✓ show finished build (slow) [470]
 ✓ delete build [1277]
 ✓ quota [491]
 ✓ delete all [454]
 ✓ clean [876]
envvars-api/envvars-smoke-test.bats
 ✓ create envvar [445]
 ✓ list envvars [461]
 ✓ show envvars [456]
 ✓ envvars are set inside jobs [21202]
 ✓ delete envvar [1311]
 ✓ quota [479]
jobs-api/continuous-job-healthcheck.bats
 ✓ run a continuous job with script healthcheck passing [15567]
 ✓ run a continuous job with script healthcheck failing [15229]
jobs-api/continuous-job-port.bats
 ✓ run a continuous job without port shows no port [1388]
 ✓ run a continuous job with a port shows port [1423]
 ✓ run a continuous job with a port exposes port [45637]
jobs-api/continuous-job-smoke-test.bats
 ✓ run a simple continuous job [26391]
jobs-api/dump-and-load.bats
 ✓ do a simple dump and load [3237]
 ✓ doing a load does not flush all other jobs (T364204) [3729]
jobs-api/one-off-job-smoke-test.bats
 ✓ run a simple one-off job [26308]
jobs-api/scheduled-job-smoke-test.bats
 ✓ run a simple scheduled job [33406]
webservice/webservice-smoke-test.bats
 ✓ status of stopped webservice [450]
 ✓ start webservice [16090]
 ✓ get logs [43812]
 ✓ restart [16040]
 - can be reached by external url (skipped) [36]
 ✓ stop [1074]

30 tests, 0 failures, 1 skipped in 458 seconds

I have reported upstream my latest results here: https://github.com/kyverno/kyverno/issues/10458#issuecomment-2180491360

aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/337

kyverno: reintroduce resource limits