Page MenuHomePhabricator
Create Task
Maniphest T377362

EPIC: Trino/MinIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation
Open, In Progress, MediumPublic
Actions

Description

Epic designed to hold all tasks related to implementation of Trino/MinIO/Hive-Standalone-Metastore, as part of a Fundraising analytics platform overhaul.

Significant Fundraising Analytics Platform Changes

At the start of the project the Fundraising analytics platform had the following on-premises hardware/software components :

  • MariaDB analytics database (frdb)
  • banner impression logs collected by kafkatee (frban)
  • data cube creation with with python scripts and Django Banner Stats (fran)
  • process scheduling with process-control (fran)
  • Superset frontend (fran)

This project makes the following on-premises platform changes:

  • network upgrade from 1Gbps to 10Gbps to accomodate anticipated Trino/MinIO network activity
  • replace MariaDB data store with a Trino (fransc/fransw) and MinIO (franio) data lake
  • replace process-control scheduling with Dagster (fran)
  • migrate cube generation code and scheduling to Dagster/DBT (fran)
  • migrate frontend from Superset to Metabase (fran)
  • replace banner impression pipeline with direct ingestion to Trino/MinIO from kafka (deprecates frban)

Project implementation has been broken into these general phases:

FY24-25

  • migrate active analytics services to codfw
  • deploy codfw Trino cluster of 1 coordinator (fransc), 3 workers (fransw)
  • deploy codfw MinIO cluster of 3 nodes (franio)
  • deploy minimal eqiad Trino cluster of 1 coordinator (fransc), 1 worker (fransw)
  • deploy eqiad MinIO cluster of 3 nodes (franio)
  • deploy services in parallel to existing Fundraising analytics services
  • limited reliability expectations on new platform during development
  • no expectation of availability of failover to eqiad rundant cluster

FY25-26

  • additional MinIO node at each datacenter for production availability
  • adjust eqiad Trino worker node count based on codfw cluster performance
  • bring eqiad redundant cluster fully online
  • phase out old banner impression pipeline (deprecate frban)
  • phase out MariaDB+Superset platform

Network Notes:

  • no change to analytics platform location, all hosts remain within frack-fundraising vlan
  • the most significant traffic is expected between Trino and MinIO nodes
  • cross-vlan and cross-firewall traffic are not expected to change dramatically
  • there will some cross-datacenter traffic for data replication to the redundant MinIO cluster
  • there should be a reduction in cross-datacenter traffic from analytics MariaDB replication
  • analytics platform and services are outside of PCI scope

Related Objects
Search...

StatusSubtypeAssignedTask
 In ProgressgregT377362 EPIC: Trino/MinIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation
 DuplicateJgreenT377364 [Trino/Hive] Setup and configuration
 ResolvedJgreenT377368 EPIC: Hive (hive-standalone-metastore) Implementation
 ResolvedJgreenT377635 [Hive] Investigate packaging, install, security monitoring.
 ResolvedJgreenT377629 [Hive] Investigate Hadoop packaging, install, security monitoring.
 ResolvedJgreenT378087 [mariadb-connector-j] Investigate packaging, install, security monitoring.
 ResolvedArnis.dizdariT377622 [Trino] Provision users & training
 ResolvedArnis.dizdariT377623 [Trino] Connect to dbt
 ResolvedArnis.dizdariT377624 [Trino] Connect to MariaDB (CRM data)
 OpenAStein-WMFT377625 Connect Trino to banner data (Kafka topic)
 ResolvedAStein-WMFT379121 Ingest Zendesk data into MinIO
 OpenNoneT379120 [Trino] Connect to Acoustic
 ResolvedAStein-WMFT385748 [Trino] Connect to Google Sheets
 ResolvedIAckerman-WMFT382439 [Trino] Google Sheets integration security review
Restricted Task
 ResolvedDwisehauptT377363 EPIC: Metabase Implementation
 ResolvedArnis.dizdariT377366 [Metabase] Setup and configuration
 ResolvedJgreenT375030 [Metabase] Investigate packaging, install, security monitoring.
 ResolvedIrisbdcT373631 Connect current data set to Metabase instance
 ResolvedIrisbdcT377626 [Metabase] Provision users & Training
 ResolvedArnis.dizdariT377627 [Metabase] Connect to Trino
 Resolvedyuki_velirT377628 [Metabase] Migration support
 ResolvedJgreenT382386 [Metabase] Package and install starburst metabase driver
 ResolvedDwisehauptT382438 Determine approach to managing Metabase access
 ResolvedJgreenT383256 [Metabase] Switch from metabase "open source" to "enterprise"
 ResolvedJgreenT385813 [Metabase] Enable TLS for mariadb connection.
 ResolvedJgreenT386259 Service monitoring for FR analytics platform.
 ResolvedJgreenT405228 Enable prometheus-jmx-exporter for FR hive-standalone-metastore
 ResolvedJgreenT386269 [Trino] Trino Python Client
 OpenAStein-WMFT385743 SurveyMonkey load data into platform
 OpenJgreenT386878 Solve the user config file problem for FR analytics.
 ResolvedDwisehauptT392090 EPIC: MinIO implementation
 ResolvedDwisehauptT377641 install/configure franio2001 franio2002 franio2003
 ResolvedDwisehauptT367819 Q1:rack/setup/install franio200[1-3]
 ResolvedJgreenT367123 [MinIO] Investigate packaging, install, security monitoring.
 ResolvedJgreenT385031 TLS connection for hive-standalone-metaserver with minio
 ResolvedVRiley-WMFT367820 Q1:rack/setup/install franio100[1-3]
 ResolvedDwisehauptT392112 [MinIO] Improve cluster to minimum system configuration for production
Unknown Object (Task)
 ResolvedDwisehauptT405981 Q2:rack/setup/install franio2004
Unknown Object (Task)
 ResolvedJgreenT405980 Q2:rack/setup/install franio1004
 ResolvedJgreenT392249 [MinIO] Investigate load-balancing approaches to eliminate SPOF
 OpenNoneT392093 EPIC: Trino implementation
 ResolvedAStein-WMFT391886 [Trino] Upgrade to Trino 475 to solve the Hive lock issue
 ResolvedJgreenT386391 [Trino] Develop procedure and scripting for Trino cluster maintenance.
 ResolvedJgreenT367818 [Trino] Investigate install, operation and maintenance, and security monitoring.
 OpenArnis.dizdariT377361 [Trino] Recovery Plan & Documentation
 ResolvedJgreenT377365 [Trino] servers
 ResolvedDwisehauptT367816 Q1:rack/setup/install fransc2001
 ResolvedPapaulT375151 codfw:frack:servers migration task
 ResolvedDwisehauptT367800 Q1:rack/setup/install fransw200[1-3].frack.codfw.wmnet
 ResolvedDwisehauptT367814 Q1:rack/setup/install fransc1001
 ResolvedDwisehauptT367801 Q1:rack/setup/install fransw1001.frack.eqiad.wmnet
 ResolvedDwisehauptT392131 [Trino] additional worker nodes for eqiad
Restricted Task
 OpenJMandoT377344 EPIC: Dagster Implementation
 ResolvedDavidbookeT377347 [Dagster] Training
 ResolvedAStein-WMFT377354 [Dagster] Orchestrate scripts
 ResolvedDavidbookeT379316 All Donations Dagster Configuration
 OpenNoneT377355 Set up monitoring in Dagster
 ResolvedArnis.dizdariT377356 [Dagster] setup & configuration
 OpenNoneT382378 Create dev environment for Dagster
 ResolvedDwisehauptT392747 [Dagster] Determine how to keep acceptable levels of dbt assets
 ResolvedDwisehauptT397942 Update Fundraising analytics server backups
 OpenNoneT397947 Migrate Fundraising analytics process-control jobs to Dagster

Event Timeline

Irisbdc created this task.Oct 16 2024, 4:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2024, 4:40 PM
Irisbdc moved this task from Backlog to In Progress on the Fundraising analytics stack board.Oct 16 2024, 4:42 PM
Irisbdc added a project: Epic.Oct 18 2024, 6:05 PM
Jgreen added a subtask: T367818: [Trino] Investigate install, operation and maintenance, and security monitoring..Oct 18 2024, 7:24 PM
Jgreen added a subtask: T367123: [MinIO] Investigate packaging, install, security monitoring..Oct 18 2024, 7:27 PM
Irisbdc added a subtask: T377361: [Trino] Recovery Plan & Documentation.Oct 18 2024, 8:07 PM
Irisbdc added a subtask: T377622: [Trino] Provision users & training.Oct 18 2024, 8:21 PM
Irisbdc added a subtask: T377623: [Trino] Connect to dbt.
Irisbdc added a subtask: T377624: [Trino] Connect to MariaDB (CRM data).Oct 18 2024, 8:25 PM
Irisbdc added a subtask: T377625: Connect Trino to banner data (Kafka topic).
Jgreen renamed this task from EPIC: Trino/minIO/Hive Implementation to EPIC: Trino/minIO/Hive/Hadoop Implementation.Oct 18 2024, 8:44 PM
Jgreen updated the task description. (Show Details)
Jgreen changed the status of subtask T377368: EPIC: Hive (hive-standalone-metastore) Implementation from Duplicate to Resolved.Oct 21 2024, 2:23 PM
Jgreen changed the task status from Open to In Progress.Oct 28 2024, 5:37 PM
Jgreen triaged this task as Medium priority.
Jgreen changed the status of subtask T377364: [Trino/Hive] Setup and configuration from Open to In Progress.
Irisbdc added a subtask: T377365: [Trino] servers.Oct 28 2024, 6:32 PM
Jgreen changed the status of subtask T377364: [Trino/Hive] Setup and configuration from In Progress to Open.Nov 5 2024, 6:46 PM
Jgreen added subtasks: T379121: Ingest Zendesk data into MinIO, T379120: [Trino] Connect to Acoustic.Nov 5 2024, 10:54 PM
Ottomata added projects: Data-Platform, Data-Engineering, Data-Platform-SRE.Nov 8 2024, 2:21 PM
Ottomata added subscribers: Ahoelzl, BTullis, Gehel and 2 others.
Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Nov 8 2024, 2:57 PM
Gehel moved this task from Scratch to Watching on the Data-Platform-SRE board.
Ahoelzl moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Nov 13 2024, 9:03 PM
Irisbdc closed subtask T377629: [Hive] Investigate Hadoop packaging, install, security monitoring. as Resolved.Dec 10 2024, 7:16 PM
Irisbdc closed subtask T377624: [Trino] Connect to MariaDB (CRM data) as Resolved.
Irisbdc closed subtask T377364: [Trino/Hive] Setup and configuration as Resolved.
Irisbdc closed subtask T377623: [Trino] Connect to dbt as Resolved.Dec 17 2024, 7:00 PM
Irisbdc closed subtask T377622: [Trino] Provision users & training as Resolved.Dec 18 2024, 6:48 PM
Irisbdc added a subtask: T382439: [Trino] Google Sheets integration security review.Dec 18 2024, 7:15 PM
• VirginiaPoundstone added a project: Data-Engineering-Radar.Jan 6 2025, 9:13 PM
Ottomata mentioned this in T361214: Data Platform - Public dashboard support.Jan 22 2025, 12:43 PM
Jgreen closed subtask T385031: TLS connection for hive-standalone-metaserver with minio as Resolved.Jan 30 2025, 9:09 PM
Jgreen removed a subtask: T382439: [Trino] Google Sheets integration security review.Feb 5 2025, 8:52 PM
Jgreen renamed this task from EPIC: Trino/minIO/Hive/Hadoop Implementation to EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation.Feb 6 2025, 3:28 PM
Jgreen updated the task description. (Show Details)Feb 6 2025, 3:31 PM
Jgreen added a subtask: T377363: EPIC: Metabase Implementation.Feb 6 2025, 5:38 PM
Jgreen added a subtask: T386259: Service monitoring for FR analytics platform..Feb 13 2025, 6:06 PM
Jgreen added a subtask: T386269: [Trino] Trino Python Client.Feb 19 2025, 2:15 PM
Jgreen closed subtask T386269: [Trino] Trino Python Client as Resolved.Feb 19 2025, 4:05 PM
Jgreen added a subtask: T385743: SurveyMonkey load data into platform.Feb 19 2025, 5:26 PM
Jgreen closed subtask T367123: [MinIO] Investigate packaging, install, security monitoring. as Resolved.Apr 8 2025, 3:03 PM
Jgreen removed Jgreen as the assignee of this task.Apr 8 2025, 3:09 PM
Jgreen subscribed.

Epic task, doesn't make sense for me to own it!

IAckerman-WMF assigned this task to greg.Apr 8 2025, 5:05 PM
Restricted Application added a project: User-greg. · View Herald TranscriptApr 8 2025, 5:05 PM
Jgreen removed a subtask: T367123: [MinIO] Investigate packaging, install, security monitoring..Apr 16 2025, 1:47 PM
Jgreen removed a subtask: T385031: TLS connection for hive-standalone-metaserver with minio.Apr 16 2025, 1:58 PM
Jgreen removed a subtask: T377365: [Trino] servers.
Jgreen removed a subtask: T377361: [Trino] Recovery Plan & Documentation.
Jgreen removed a subtask: T367818: [Trino] Investigate install, operation and maintenance, and security monitoring..
Jgreen removed a subtask: T377629: [Hive] Investigate Hadoop packaging, install, security monitoring..Apr 16 2025, 2:06 PM
Jgreen removed a subtask: T386391: [Trino] Develop procedure and scripting for Trino cluster maintenance..Apr 16 2025, 2:13 PM
Jgreen added a subtask: Restricted Task.Apr 16 2025, 2:21 PM
Jgreen renamed this task from EPIC: Trino/minIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation to EPIC: Trino/MinIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation.Apr 17 2025, 12:35 PM
Jgreen updated the task description. (Show Details)
Jgreen updated the task description. (Show Details)Apr 17 2025, 12:44 PM
Jgreen updated the task description. (Show Details)Apr 17 2025, 1:11 PM
BTullis removed a project: Data-Platform.Apr 28 2025, 9:57 PM
Jgreen added a subtask: T377344: EPIC: Dagster Implementation.May 8 2025, 1:17 PM
AStein-WMF closed subtask T385748: [Trino] Connect to Google Sheets as Resolved.Jul 23 2025, 9:29 PM
BTullis removed a project: Data-Platform-SRE.Sep 9 2025, 6:16 PM
Jgreen closed subtask T386259: Service monitoring for FR analytics platform. as Resolved.Oct 6 2025, 5:28 PM
AStein-WMF changed the status of subtask T379121: Ingest Zendesk data into MinIO from Open to Stalled.Oct 30 2025, 8:15 PM
Dwisehaupt closed subtask T377363: EPIC: Metabase Implementation as Resolved.Mar 10 2026, 5:55 PM
Dwisehaupt closed subtask T397942: Update Fundraising analytics server backups as Resolved.Mar 10 2026, 6:11 PM
Dwisehaupt closed subtask T392090: EPIC: MinIO implementation as Resolved.Mar 10 2026, 6:13 PM
AStein-WMF changed the status of subtask T379121: Ingest Zendesk data into MinIO from Stalled to In Progress.Mar 24 2026, 4:50 PM
AStein-WMF closed subtask T379121: Ingest Zendesk data into MinIO as Resolved.Mar 27 2026, 7:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits