We should evaluate and enable audit logging for the kube-apiserver. This would help to keep track of accidental manipulation of cluster objects or harmful operations.
We have to evaluate which resources and actions we want to log to keep the volume low. I would assume we don't want to log get and watch actions. Furthermore we have to think about how to access the audit log (logstash/elasticsearch or local only?).
More information:
https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
This topic came up in https://phabricator.wikimedia.org/T251305 because with helm3 we may lose some reliability for audit capabilities.
As part of T273507: PodSecurityPolicies will be deprecated with Kubernetes 1.21 audit-logging support has been added to the puppet codebase along with a very simple logging policy that only logs actions modifying pod objects (because that's what was needed in that context). For a more generic approach we could look at the config GCE generates: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L1113
Currently the kube-apiservers write audit logs (if enabled) to /var/log/kubernetes/audit.log, rotating after reaching 100MB