Spark sessions can provision kerberos tickets in a more predictable manner.
Right now kerberos tickets are expiring after 24 hours, this leaves spark sessions hanging after 24 hours and not able to move forward. We should probably extend this timeframe and document the period in which kerberos ticket expires.
At the same time there are plenty jupyter notebooks that are left idle, not used, and still consuming resources.
We need to :
- support longer running jobs in jupyter (3/4 days)
- kill jupyter notebooks that are not being used