Please make sure not to pre-pull images on tainted nodes (which are masters and kask/sessionstore currently).

My usecase was actually to create a new instance with the same name, not keeping any data. So from that you say I assume deleting and creating a new one with the same name should do the trick for me. Thanks

Is there maybe a workaround for this that could be used instead?

helmfile_log_sal has support for that already:

# Allow to explicitely suppress logging to SAL
SUPPRESS_SAL=${SUPPRESS_SAL:-false}

That sounds about right. Although you absolutely can specify the latter rules as a ClusterRole object and than bind it to just a namespace using a RoleBinding (instead of a ClusterRoleBinding). That way you don't have to declare the Role in every Namespace you want to use (just the RoleBinding).

In T322635#8397044, @BTullis wrote:

[...]
Lastly, in terms of these RBAC rules, a ClusterRoleBinding is created. This specifies the namespace spark-operator so it limits the power of the spark-operator to work within its own namespace.

+ # Source: spark-operator/templates/rbac.yaml
+ apiVersion: rbac.authorization.k8s.io/v1
+ kind: ClusterRoleBinding
+ metadata:
+   name: spark-operator-spark-data-engineering
+   annotations:
+     "helm.sh/hook": pre-install, pre-upgrade
+     "helm.sh/hook-delete-policy": hook-failed, before-hook-creation
+     "helm.sh/hook-weight": "-10"
+   labels:
+     app: spark-operator
+     chart: spark-operator-0.0.1
+     release: spark-data-engineering
+     heritage: Helm
+ subjects:
+   - kind: ServiceAccount
+     name: spark-operator
+     namespace: spark-operator
+ roleRef:
+   kind: ClusterRole
+   name: spark-operator-spark-data-engineering
+   apiGroup: rbac.authorization.k8s.io

I will discuss the other resources in the parent ticket.

In T307943#8388861, @MoritzMuehlenhoff wrote:

Let's bump to v1.23.14 before migrating? That version fixes two new security issues:

CVE-2022-3294: Node address isn't always verified when proxying:
https://groups.google.com/g/kubernetes-announce/c/eR0ghAXy2H8

CVE-2022-3162: Unauthorized read of Custom Resources:
https://groups.google.com/g/kubernetes-announce/c/oR2PUBiODNA

This might be helpful to calculate initial values: https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-architecture#memory_cpu

Fixed for cfss-issuer in chart version 0.3.0

We not have exact major.minor version as an enum type puppet and with dedicated apt components

In T322453#8381008, @dduvall wrote:

@JMeybohm can you provide the nginx access log entries from that time period as well? I'm trying to rule out auth failure as a factor and docker-registry log entries do not include the subrequests between nginx and jwt-authorizer.

I took a quick look and AIUI our registry does support application/vnd.docker.distribution.manifest.list.v2+json already. That is really not a new thing and we are not that far behind upstream releases.

In T322579#8378397, @Joe wrote:

[...]
Also: both the registry and nginx keep access logs, so I guess it's enough to export one of the two.

In T321201#8366250, @Clement_Goubert wrote:

[...]
@JMeybohm @Joe @akosiaris If this seems like the right way, I will start writing the "Kubernetes/Remove_a_service" wikitech doc.

I think this might require an update of our docker-registry (which we have not planned for currently). Is this blocking something apart from testing with multi arch images (T272500)?

Thanks @jbond !

IIRC the application deployment cluster we ditched because of missing HA capabilities. Option 1. we might have considered as well but found that it was not easy to "get right" in an automated way (without adding quite some complexity). Especially in cases where a job has to resume from a tombstone/safepoint. But ofc. as long as the "job artifact/the jar file" is bundled in a container, which is deployed via helm charts, it makes it easy to recreate this exact state in the cluster (without knowing anything about flink).

I'm going to resolve this as we're updating all of the said clients with T307943: Update Kubernetes clusters to v1.23

In T321657#8345852, @taavi wrote:

make better use of the quota in the projects

Not sure what you mean by this - wouldn't you still get hit by the CPU quota anyways? As far as I can tell CPU is currently a much more limited resource than what RAM is, so this might not be worth the extra complexity.

Now it has

ouch, this never made it to prod

This needs SRE support to depool eventstreams from one DC. helmfile destroy/helmfile appy can be be done by deployers as well.

Fixed for calico with v3.23.3

In T318707#8273903, @bking wrote:

Speaking from a position of almost total ignorance:

Do we only care about the pods spawned by the helm charts in the deployment-charts repo ? Just trying to figure out which clusters/pods are affected.

Basically yes. But ultimately this will affect all pods in all clusters - and we should try to keep backwards compatibility as far as possible.

In T318705#8269372, @bking wrote:

How can we tell if this is working?

Use an envoy metric that doesn't match the regex, such as

rate(envoy_cluster_default_total_match_count{app="api-gateway", envoy_cluster_name=~".*"}[5m])

After the change is merged, this should not return any hits.

Changed the alerts from using p99 to using p95, resolving this again.