Kubernetes does store events in etcd for an hour (by default) and we can view them with kubectl.
Unfortunately this does not allow for easy searching or aggregation and is not helpful for investigating things that had happened longer than an hour ago and we can also not create alerts from those events.
I think we should store the events externally and elasticsearch is a good candidate for this.
There are two project I know for routing k8s events to elasticsearch/kafka (in no particular order):
And then there is Grafana Loki, but that is completely new and unknown stuff I guess.
Is there a chance we can push the events to elasticsearch, directly or via kafka?