Currently after one day of running, I start getting the errors:
jobs-api-674d88fdd7-wfd5j webservice to create fsnotify watcher: too many open files ... jobs-api-674d88fdd7-wfd5j nginx to create fsnotify watcher: too many open files
When trying to tail logs using stern (and kubectl logs), we should increase considerably the limit again just to make sure we don't hit it so soon.