Page MenuHomePhabricator

Refine to Hive with Airflow – Update Refine Documentation on Wikitech
Closed, ResolvedPublic

Description

Update the Wikitech documentation for Refine, including:

  • A general review and refresh of existing content.
  • Adding guidance on how to choose Spark job ingestion scale parameters, using Grafana topic size metrics as a reference point.
  • how to deal with Refine errors
  • updates to Ops week page on how to admin Refine on Airflow
  • instructions on how to backfill refined event data (Example of 'hacky backfill')

This should help future maintainers tune resource usage more effectively based on real-time data.

Event Timeline

Ottomata updated the task description. (Show Details)
Ottomata updated the task description. (Show Details)