Change Details

Implement the functionality of synchronizing HDFS locations of refinery HQL files automatically on merging into the master branch of GitLab repository.

Implement the functionality of synchronizing HDFS locations of refinery HQL files automatically on merging into the master branch of GitLab repository. Follow the project proposal as described in [[ https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE | this document ]]. **Done is**: [] Implemented stand-alone Flask Web application that: - Supports configuration of different repositories through a config file - Provides REST API endpoints to initiate the synchronization process for each of the configured repositories - Supports pulling the latest Git repository updates - Recognizes files/folders to be synchronized to HDFS locations, through config file(s) - Performs transactional synchronization from Git repository to HDFS locations - Keeps a log of all requests and synchronization operations performed [] Application that is fully operable within the context of our Kubernetes cluster - App/service is accessible to REST API calls from WMF GitLab - Has functional access to WMF GitLab repositories - Has functional access to WMF HDFS - Will not be public facing - Will not be accessible to any hosts other than GitLab - Will require authentication.

Implement the functionality of synchronizing HDFS locations of refinery HQL files automatically on merging into the master branch of GitLab repository. Follow the project proposal as described in [[ https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE | this document ]]. **Done is**: [] Implemented stand-alone Flask Web application that: - Supports configuration of different repositories through a config file - Provides REST API endpoints to initiate the synchronization process for each of the configured repositories - Supports pulling the latest Git repository updates - Recognizes files/folders to be synchronized to HDFS locations, through config file(s) - Performs transactional synchronization from Git repository to HDFS locations - Keeps a log of all requests and synchronization operations performed [] Application that is fully operable within the context of our Kubernetes cluster - App/service is accessible to REST API calls from WMF GitLab - Has functional access to WMF GitLab repositories - Has functional access to WMF HDFS - Will not be public facing - Will not be accessible to any hosts other than GitLab - Will require authentication.