Currently we save things like emails, mount, and filelog directly in the kubernetes object as labels.

this task is about analysing the possibility of moving these and any other labels that we can to annotations.

Why:

• kubernetes labels don't do well with dynamic data. For example trying to make changes to kubernetes labels with kubectl apply * will blow up. This became an issue because we are in the process moving the jobs load logic to the runtime and implementing it with kubectl apply * instead ( this move to kubectl apply * is necessary because it allows us to use the same logic for many other things we are working on)

Concerns:
There seems to be a number of non jobs-api services that make use of the labels created by jobs-api. We may need to refactor these services so the move to annotation won't break them.

Possible Implementation Steps

• refactor jobs-api to add jobs.toolforge.org/* fields to both labels and annotations.
• write a one time script to add jobs.toolforge.org/* annotations to toolforge kubernetes job objects with no jobs.toolforge.org/* annotations (mostly old legacy jobs)
• when all existing and newly created jobs have annotations, switch jobs-emailer, and other services that use labels to use annotations instead
• finally refactor jobs-api to stop adding jobs.toolforge.org/* labels