Epic designed to hold all tasks related to implementation of Trino/MinIO/Hive-Standalone-Metastore, as part of a Fundraising analytics platform overhaul.
`Significant Fundraising Analytics Platform Changes`
At the start of the project the Fundraising analytics platform had the following on-premises hardware/software components :
* MariaDB analytics database (frdb)
* banner impression logs collected by kafkatee (frban)
* data cube creation with with python scripts and Django Banner Stats (fran)
* process scheduling with process-control (fran)
* Superset frontend (fran)
This project makes the following on-premises platform changes:
* network upgrade from 1Gbps to 10Gbps to accomodate anticipated Trino/MinIO network activity
* replace MariaDB data store with a Trino (fransc/fransw) and MinIO (franio) data lake
* replace process-control scheduling with Dagster (fran)
* migrate cube generation code and scheduling to Dagster/DBT (fran)
* migrate frontend from Superset to Metabase (fran)
* replace banner impression pipeline with direct ingestion to Trino/MinIO from kafka (deprecates frban)
Project implementation has been broken into these general phases:
FY24-25
* migrate active analytics services to codfw
* deploy codfw Trino cluster of 1 coordinator (fransc), 3 workers (fransw)
* deploy codfw MinIO cluster of 3 nodes (franio)
* deploy minimal eqiad Trino cluster of 1 coordinator (fransc), 1 worker (fransw)
* deploy eqiad MinIO cluster of 3 nodes (franio)
* deploy services in parallel to existing Fundraising analytics services
* limited reliability expectations on new platform during development
* no expectation of availability of failover to eqiad rundant cluster
FY25-26
* additional MinIO node at each datacenter for production availability
* adjust eqiad Trino worker node count based on codfw cluster performance
* bring eqiad redundant cluster fully online
* phase out old banner impression pipeline (deprecate frban)
* phase out MariaDB+Superset platform
Network Notes:
* no change to analytics platform location, all hosts remain within frack-fundraising vlan
* the most significant traffic is expected between Trino and MinIO nodes
* cross-vlan and cross-firewall traffic are not expected to change dramatically
* there will some cross-datacenter traffic for data replication to the redundant MinIO cluster
* there should be a reduction in cross-datacenter traffic from analytics MariaDB replication
* analytics platform and services are outside of PCI scope