Narratives

• As an SRE, I want the ability to observe the ImageMatchingAlgorithm data pipeline metrics, so that I know how well it is performing compared to defined SLOs
• As a PET data engineer, I want the ability to observe the ImageMatchingAlgorithm data pipeline metrics, so that I can have a baseline to ensure we have optimized how our pipeline performs.
• As an Analytics Engineer, I want the ability to observe the ImageMatchingAlgorithm data pipeline metrics, so that I can be aware of how the data infrastructure resources are being utilized.

Acceptance Criteria

• As an PET Data Engineer, I want the ability to generate a csv file with the following metrics, so that I can have a baseline of how the pipeline performs.
  • in/out records
  • CPU Usage
  • Memory Usage
  • Executor Counts
  • Runtime and resource utilization for algo

Notes

• This is the initial part of how we start to collect metrics. We will iterate so that we have designated stores for our collected metrics.

Subtasks