This task is about enabling the ingestion of Event Platform streams into WMF Data Lake Iceberg tables.
Much of the implementation work has already been done as part of T356762: [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation.
This task is not about migrating all existent event.* Hive tables to Iceberg.
Note that if we do this, the ingested tables can use actual timestamp types rather than ISO8601 strings.
Done is
- Iceberg tables use timestamp types rather than strings for timestamp fields. (already enabled in this patch). See also T278467#11386781.
- Refine can ingest event streams into Iceberg tables via a configuration (e.v. EventStreamConfig analytics_iceberg_ingestion.enabled: true.)
- T377600: [refine] Add support for custom Hive Iceberg partitioning