Page MenuHomePhabricator

Move leftover Refine related timers to Airflow
Open, Needs TriagePublic

Description

Netflow timers refine_netflow.timer and monitor_refine_netflow.timer. Puppet code here. Leftovers from moving refine jobs to airflow: T307505: Refine jobs should be scheduled by Airflow.

  • Refine_netflow.timer
    • Runs a Spark Refine job that processes netflow data
    • Transforms raw netflow data from /wmf/data/raw/netflow/wmf/data/event/netflow in the Hive event database
    • Should be portable to airflow easily.
    • Looks like Refine is deprecated (used by this timer)
  • Monitor_refine_netflow.timer
    • Replace monitors with Airflow alerts

List of all timers and more details in Google Doc.