Page MenuHomePhabricator

[jobs-api] Refactor before webservice support
Open, HighPublic

Description

Some things that we can simplify before adding the webservice complexity:

  • Remove flask-restful (we don't use rest), and use bare flask, this will make sure we can decouple API models from business models, will be specially useful once we start generating the code from the openapi definition
  • Split the 3 model sets:
    • Api requests (ex. data sent when starting a job, or requesting a job)
    • Internal logic (ex. ScheduledJob, ContinuousJob, OneOffJob should be classes by themselves as they are quite different and have different data attached to them)
    • K8s mapping, we can abstract this in something we might want to call 'engine', so we might have a better time swapping/modifying it in the future, for example:
from abc import ABC, abstractmethod

class Engine(ABC):
   @abstractmethod
   def run_continuous_job(self, continuous_job: ContinuousJob) -> RunResult:
      pass

   @abstractmethod
   def run_scheduled_job(self, scheduled_job: ScheduledJob) -> RunResult:
      pass
...


class K8sEngine(Engine):
    def run_continuous_job(self, continuous_job: ContinuousJob) -> RunResult:
        ... do stuff on k8s to get the job running

That would help keep the k8s side, the business logic side, and the API side models separated.

  • Use a database to store our internal models (this could be etcd with custom resources, read-only for users as we want to only modify them through the API), that would allow us to untangle k8s objects from business model objects (ex. having to parse the k8s template.spec.command, or having to add some versioning information to the k8s objects), this would be a step after the Engine above, and could be something like:
from abc import ABC, abstractmethod

class Store(ABC):
   @abstractmethod
   def get_continuous_job(self, job_id: str) -> ContinuousJob:
      pass

   @abstractmethod
   def get_continuous_jobs(self, tool_name: str) -> list[ContinuousJob]:
      pass
...


class K8sStore(Engine):
   def get_continuous_job(self, job_id: str) -> ContinuousJob:
      ... retrieve all the jobs from k8s, could be started as the same logic we currently have, listing all the deployments in the user's namespace, etc., and then be replaced by reading the custom resources

Details

TitleReferenceAuthorSource BranchDest Branch
[jobs-api] create seperate api.py and move flask things thererepos/cloud/toolforge/jobs-api!91raymond-ndibesplit_logic_from_apimain
Customize query in GitLab