Change Details

Implement the functionality of synchronizing HDFS locations of refinery HQL files automatically on merging into the master branch of GitLab repository. Follow the project proposal as described in [[ https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE | this document ]]. Design work done here: {T360968} **Done is**: [] Existing workflow_utils project modified to support its original use cases as well as this new one [] Add support for interacting with GitLab repositories through a designated script [] If needed, modify artifact.py script to support any additional functionality [] Implemented stand-alone Flask Web application that: [] Supports configuration of different repositories through a config file [] Provides REST API endpoints to initiate the synchronization process for each of the configured repositories [] Supports pulling the latest Git repository updates through workflow_utils package [] Recognizes files/folders to be synchronized to HDFS locations, through config file(s) [] Performs transactional synchronization from Git repository to HDFS locations [] Implemented synchronization task log web page [] Provides a web endpoint that lists all synchronization tasks received from Gitlab, with their status, in chronological order [] Application that is fully operable within the context of our Kubernetes cluster [] App/service is accessible to REST API calls from WMF GitLab [] Has functional access to WMF GitLab repositories [] Has functional access to WMF HDFS [] Will not be public facing [] Will not be accessible to any hosts other than GitLab [] Will require authentication. [] repository, k8s, DNS and other names are named 'blunderbuss' and not 'hdfs-synchronizer' --- this is blunderbuss :)

Implement the functionality of synchronizing HDFS locations of refinery HQL files automatically on merging into the master branch of GitLab repository. Follow the project proposal as described in [[ https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE | this document ]]. Design work done here: {T360968} **Done is**: [x] Existing workflow_utils project modified to support its original use cases as well as this new one [x] Add support for interacting with GitLab repositories through a designated script [x] If needed, modify artifact.py script to support any additional functionality [x] Implemented stand-alone Flask Web application that: [x] Supports configuration of different repositories through a config file [x] Provides REST API endpoints to initiate the synchronization process for each of the configured repositories [x] Supports pulling the latest Git repository updates through workflow_utils package [x] Recognizes files/folders to be synchronized to HDFS locations, through config file(s) [x] Performs transactional synchronization from Git repository to HDFS locations [] Implemented synchronization task log web page [] Provides a web endpoint that lists all synchronization tasks received from Gitlab, with their status, in chronological order [x] Application that is fully operable within the context of our Kubernetes cluster [x] App/service is accessible to REST API calls from WMF GitLab [x] Has functional access to WMF GitLab repositories [x] Has functional access to WMF HDFS [x] Will not be public facing [x] Will not be accessible to any hosts other than GitLab [] Will require authentication. [x] repository, k8s, DNS and other names are named 'blunderbuss' and not 'hdfs-synchronizer' [] Develop a GitLab CI/CD Component to be used by projects/repositories to integrate their CI/CD workflows with Blunderbuss --- this is blunderbuss :)

Implement the functionality of synchronizing HDFS locations of refinery HQL files automatically on merging into the master branch of GitLab repository. Follow the project proposal as described in [[ https://docs.google.com/document/d/1gytt1rzO5wO1IWmmrGKdfuwSlVvFyQ9pFK87wePUYFE | this document ]]. Design work done here: {T360968} **Done is**: [x] Existing workflow_utils project modified to support its original use cases as well as this new one [x] Add support for interacting with GitLab repositories through a designated script [x] If needed, modify artifact.py script to support any additional functionality [x] Implemented stand-alone Flask Web application that: [x] Supports configuration of different repositories through a config file [x] Provides REST API endpoints to initiate the synchronization process for each of the configured repositories [x] Supports pulling the latest Git repository updates through workflow_utils package [x] Recognizes files/folders to be synchronized to HDFS locations, through config file(s) [x] Performs transactional synchronization from Git repository to HDFS locations [] Implemented synchronization task log web page [] Provides a web endpoint that lists all synchronization tasks received from Gitlab, with their status, in chronological order [x] Application that is fully operable within the context of our Kubernetes cluster [x] App/service is accessible to REST API calls from WMF GitLab [x] Has functional access to WMF GitLab repositories [x] Has functional access to WMF HDFS [x] Will not be public facing [x] Will not be accessible to any hosts other than GitLab [] Will require authentication. [x] repository, k8s, DNS and other names are named 'blunderbuss' and not 'hdfs-synchronizer' [] Develop a GitLab CI/CD Component to be used by projects/repositories to integrate their CI/CD workflows with Blunderbuss --- this is blunderbuss :)