Epic designed to hold all tasks related to implementation of Trino/MinIO/Hive-Standalone-Metastore, as part of a Fundraising analytics platform overhaul.
`Significant Fundraising Analytics Platform Changes`
At the start of the project the Fundraising analytics platform had the following on-premises hardware/software components :
* MariaDB analytics database (frdb)
* banner impression logs collected by kafkatee (frban)
* data cube creation with with python scripts and Django Banner Stats (fran)
* process scheduling with process-control (fran)
* Superset frontend (fran)
This project makes the following on-premises platform changes:
* network upgrade from 1Gbps to 10Gbps to accomodate anticipated Trino/MinIO network activity
* replace MariaDB data store with a Trino (fransc/fransw) and MinIO (franio) data lake
* replace process-control scheduling with Dagster (fran)
* migrate cube generation code and scheduling to Dagster/DBT (fran)
* migrate frontend from Superset to Metabase (fran)
* replace banner impression pipeline with direct ingestion to Trino/MinIO from kafka (deprecates frban)
Project implementation has been broken into these general phases:
FY24-25
* migrate active analytics services to codfw
* deploy codfw Trino cluster of 1 coordinator (fransc), 3 workers (fransw)
* deploy codfw MinIO cluster of 3 nodes (franio)
* deploy minimal eqiad Trino cluster of 1 coordinator (fransc), 1 worker (fransw)
* deploy eqiad MinIO cluster of 3 nodes (franio)
* deploy services in parallel to existing Fundraising analytics services
FY25-26
* additional MinIO node at each datacenter for production availability
* adjust eqiad Trino worker node count based on codfw cluster performance
* possibly deprecate frban servers after migrating banner impression pipeline to Trino/MinIO