Without partitioning if we INSERT OVERWRITE (rather than INSERT INTO) then we would overwrite all of the existing data. With partitioning we only overwrite if the partition exists.
Currently this is an issue for:
Those tables would need to be re-created with per-month partitioning.
This would also make them much faster to query.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T312880 Improve movement metric calculation as analytics-product user | |||
| Duplicate | None | T307598 Modify ETL Update queries to INSERT OVERWRITE | |||
| Open | nshahquinn-wmf | T359207 Improve the delivery of the movement movements (SDS 2.6.2) | |||
| Invalid | nshahquinn-wmf | T359697 Document tables managed by the movement_metrics ETL job | |||
| Open | None | T359700 Ensure that movement metrics Superset dashboards are functional and performant | |||
| Open | nshahquinn-wmf | T333225 Migrate the movement_metrics ETL jobs to Airflow | |||
| Declined | None | T308077 Add partitioning to editors tables |
As part of T359207, I will be converting these intermediate tables to use Iceberg, which removes the need for partitioning.