/me really interested on the option for dropping the 3rd party component on the 1.26 upgrade

we evaluate dropping kyverno in favor of VAPs

Can we add an option where this is "drop kyverno" instead?

In T362233#9705907, @dcaro wrote:

we evaluate dropping kyverno in favor of VAPs

Can we add an option where this is "drop kyverno" instead?

This is very speculative at this point. Why would you like that statement to be included?

In T362233#9705910, @aborrero wrote:

In T362233#9705907, @dcaro wrote:

we evaluate dropping kyverno in favor of VAPs

Can we add an option where this is "drop kyverno" instead?

This is very speculative at this point. Why would you like that statement to be included?

Because any option that ends without a 3rd party in the mid-run (it's actually just one k8s upgrade away!) is way better than any other option that does. Most of the risk and long-term maintenance involved in having a 3rd party just vanishes.

If that's not part of the solution, then you have to start weighting long-term support, stability and such.

In T362233#9705940, @dcaro wrote:

Because any option that ends without a 3rd party in the mid-run (it's actually just one k8s upgrade away!) is way better than any other option that does. Most of the risk and long-term maintenance involved in having a 3rd party just vanishes.

If that's not part of the solution, then you have to start weighting long-term support, stability and such.

I don't think a commitment to drop kyverno at point X in the future makes sense. We haven't done anything like this for any of the other 3rd party components we have in Toolforge.
I'm fine with evaluating our options regarding kyverno vs VAPs when we get to the point where that's an actual possibility. Today this feels a bit like a guessing game.

I don't have any data at this point that indicates that long-term support or stability of kyverno could introduce risks for Toolforge. If you do, please share :-P

In T362233#9706061, @aborrero wrote:

In T362233#9705940, @dcaro wrote:

Because any option that ends without a 3rd party in the mid-run (it's actually just one k8s upgrade away!) is way better than any other option that does. Most of the risk and long-term maintenance involved in having a 3rd party just vanishes.

If that's not part of the solution, then you have to start weighting long-term support, stability and such.

I don't think a commitment to drop kyverno at point X in the future makes sense. We haven't done anything like this for any of the other 3rd party components we have in Toolforge.
I'm fine with evaluating our options regarding kyverno vs VAPs when we get to the point where that's an actual possibility. Today this feels a bit like a guessing game.

I don't agree with this, the timeline to upgrade to kubernetes 1.26 would be less than a year, that's the clear <1year away point in which we would have to drop kyverno. I don't think it's too far in the future.

Please add an option in which we decide to drop kyverno with the 1.26 upgrade.

I don't have any data at this point that indicates that long-term support or stability of kyverno could introduce risks for Toolforge. If you do, please share :-P

It's a cncf incubating project (that means unstable) pushed by a single company with an enterprise downstream version on top of it (that means that they will change it as they see fit, and with a non-small chance of them either changing the license or the company dropping the project).

In T362233#9706072, @dcaro wrote:

Please add an option in which we decide to drop kyverno with the 1.26 upgrade.

Please feel free to add it yourself :-)

I don't have any data at this point that indicates that long-term support or stability of kyverno could introduce risks for Toolforge. If you do, please share :-P

It's a cncf incubating project (that means unstable) pushed by a single company with an enterprise downstream version on top of it (that means that they will change it as they see fit, and with a non-small chance of them either changing the license or the company dropping the project).

I don't think incubating means unstable, per the CNCF definition:

The number of contributors to the kyverno project is high, beyond what the single company does.

You mention an enterprise downstream version, but per the Nirmata docs, this doesn't seem to be the case. They just offer value added services, see here:

We use plenty of 3rd party components that are way riskier than kyverno for the reasons you pointed out (nginx, haproxy, calico, gitlab, etc) and we don't seem to have any concrete plan to get rid of them, beyond what I'm defending here: we will adapt when the need arises.

I think the tradeoff here is good enough.

In T362233#9706547, @aborrero wrote:

In T362233#9706072, @dcaro wrote:

Please add an option in which we decide to drop kyverno with the 1.26 upgrade.

Please feel free to add it yourself :-)

I don't have any data at this point that indicates that long-term support or stability of kyverno could introduce risks for Toolforge. If you do, please share :-P

It's a cncf incubating project (that means unstable) pushed by a single company with an enterprise downstream version on top of it (that means that they will change it as they see fit, and with a non-small chance of them either changing the license or the company dropping the project).

I don't think incubating means unstable, per the CNCF definition:

Unstable as in unstable project, not unstable software (it's way more likely that it's direction might change).

The number of contributors to the kyverno project is high, beyond what the single company does.

You mention an enterprise downstream version, but per the Nirmata docs, this doesn't seem to be the case. They just offer value added services, see here:

They mention that they provide earlier CVE/bugfixes than the OSS one, and that they prioritize features over the OSS ones, that to me means that they use a fork of the OSS one with their enterprise support on top of that one.

In that same panphlet they have a side-to-side between kyveno OSS and nimrata enterpise.

They refer to it as 'Nirmata Enterprise for Kyverno is the enterprise-grade distribution'.

"Kyverno OSS is a full-featured and production-ready solution, but organizations with business-critical applications require the
additional value of Nirmata Enterprise for Kyverno."

That's not only support and training on top of it. I think there's enough in that same link you pass to think they have their own fork of the OSS version with their enterprise bits on top.

We use plenty of 3rd party components that are way riskier than kyverno for the reasons you pointed out (nginx, haproxy, calico, gitlab, etc) and we don't seem to have any concrete plan to get rid of them, beyond what I'm defending here: we will adapt when the need arises.

That does not mean that we should add even more.

I think the tradeoff here is good enough.

All this is part of the discussion that needs to happen, please add the option.

In T362233#9706547, @aborrero wrote:

In T362233#9706072, @dcaro wrote:

Please add an option in which we decide to drop kyverno with the 1.26 upgrade.

Please feel free to add it yourself :-)

Missed this, I'll do

I'm fine with both option 1 and option 3. I don't see a huge practical difference between "we evaluate dropping kyverno" (option 1) or "replacing kyverno [...] on the 1.26 k8s upgrade" (option 3). In both cases we are going to try dropping Kyverno after the 1.26 upgrade, unless we find some big blockers in doing so.

In both cases, we should keep track of the evolution of Kyverno, and avoid using any Kyverno-only features (if they exist) that could make it harder to migrate to VAP. From a quick search, I found a mention of a policy that cannot be translated.

In T362233#9726517, @fnegri wrote:

In both cases we are going to try dropping Kyverno after the 1.26 upgrade, unless we find some big blockers in doing so.

I thin this is not true with option 1, as I understand it, "we can consider migrating our policies" does not mean that we will even consider, and does not force us to commit to do so either.

With option 3 the commitment is clear, so we can plan accordingly and dedicate the resources needed (as opposed to not planning for it, and thus never happens).

Just acknowledging that I've seen this discussion, but that I don't know enough about this topic to have a preference.

scheduled discussion meeting for 2024-04-30.

The decision about commiting to drop the extra component on the upgrade to k8s 1.26 might become way more relevant with T363683: Decision request - kubernetes upgrade workgroup, if we decide to try to upgrade monthly.
As the timeline would be:

• adopt policy agent
• right after upgrade to 1.25
• one month after upgrade to 1.26 + drop policy engine)

The output of the decision meeting was that we go with option 3, with an additional caveat:

we want to drop Kyverno in favor of VallidationAdmissionPolicies after we upgrade K8s to 1.26 and before we upgrade it to 1.29. If we get to the point where we upgrade to 1.29 and we're still using Kyverno, we will hold a new decision request to agree on a new plan.