[MinIO] Investigate load-balancing approaches to eliminate SPOF
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jgreen
	Apr 17 2025, 5:19 PM

Project Tags

Referenced Files

None

Subscribers

View All 11 Subscribers

Description

We have a small MinIO cluster and have Trino configured to use the first node (by hostname) in the cluster as its S3 endpoint. This creates a SPOF, if that node goes down Trino loses its MinIO connection though other MinIO nodes are available. We should come up with an approach to distribute connections across available MinIO nodes since they're functionally equivalent.

Trino doesn't appear to be able to work with a list of S3 endpoints. If the goal were just to distribute load we could use a basic DNS RR. But to handle down MinIO servers, what about a service like HAProxy on each Trino server as a layer between Trino and the MinIO pool?

Related Objects
Search...

Status	Assigned	Task
In Progress	greg	T377362 EPIC: Trino/MinIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation
Resolved	Dwisehaupt	T392090 EPIC: MinIO implementation
Resolved	Jgreen	T392249 [MinIO] Investigate load-balancing approaches to eliminate SPOF

Event Timeline

Jgreen created this task.Apr 17 2025, 5:19 PM

Jgreen updated the task description. (Show Details)Apr 28 2025, 1:49 PM

Bikeshedding with Dallas we came up with this scheme to investigate further:

haproxy on each trino node, bound to loopback IP
haproxy handling TLS termination to Trino, but non-TLS on its service port
haproxy configured to connect https to each MinIO node's S3 port
trino S3 configured to connect to http://localhost:<haproxy-port>

BTullis removed a project: Data-Platform.Apr 28 2025, 9:56 PM

Ahoelzl removed a project: Data-Engineering.Apr 29 2025, 5:51 PM

BTullis removed a project: Data-Platform-SRE.Apr 30 2025, 1:56 PM

Here's an HAProxy config that seems to work, more testing is needed:

global
   maxconn 4096
   user haproxy
   group haproxy
   stats socket /run/haproxy/haproxy.sock mode 666 level user

frontend the_frontend
   bind 127.0.0.1:9000
   default_backend minio_backend
   timeout client 50000

backend minio_backend
   balance roundrobin
   option httpchk
   timeout connect 5000
   timeout server 50000
   server franio2001 franio2001.frack.codfw.wmnet:9000 weight 100 ssl verify required ca-file /etc/trino/ssl/ca.pem
   server franio2002 franio2002.frack.codfw.wmnet:9000 weight 100 ssl verify required ca-file /etc/trino/ssl/ca.pem
   server franio2003 franio2003.frack.codfw.wmnet:9000 weight 100 ssl verify required ca-file /etc/trino/ssl/ca.pem

Trino catalog/*.properties:

hive.s3.endpoint=http://localhost:9000

Hive-Standalone-Metastore:

<property>
    <name>fs.s3a.endpoint</name>
    <value>http://localhost:9000</value>
</property>

<property>
  <name>fs.s3a.connection.ssl.enabled</name>
  <value>false</value>
</property>

I'm not sure which balance method makes sense. Each Trino/HAproxy server only managing its own connections, and we're mainly looking to eliminate the SPOF, so it probably doesn't matter much. Maybe 'random'?

Improved

global
   maxconn 4096
   user haproxy
   group haproxy
   stats socket /run/haproxy/haproxy.sock mode 666 level user

frontend the_frontend
   bind 127.0.0.1:9000
   default_backend minio_backend
   timeout client 50000

backend minio_backend
   balance roundrobin
   option httpchk
   timeout connect 5000
   timeout server 50000
   http-check send meth GET uri /minio/login
   http-check expect string login
   server franio2001 franio2001.frack.codfw.wmnet:9000 weight 100 ssl verify required ca-file /etc/trino/ssl/ca.pem check
   server franio2002 franio2002.frack.codfw.wmnet:9000 weight 100 ssl verify required ca-file /etc/trino/ssl/ca.pem check
   server franio2003 franio2003.frack.codfw.wmnet:9000 weight 100 ssl verify required ca-file /etc/trino/ssl/ca.pem check

Jgreen mentioned this in T337585: FR-Tech FY2425Q4 maintenance window.May 20 2025, 1:03 PM

This is deployed

Jgreen added a project: fundraising-tech-ops.May 28 2025, 7:10 PM

Jgreen updated the task description. (Show Details)

Jgreen moved this task from Triage to Done on the fundraising-tech-ops board.

[MinIO] Investigate load-balancing approaches to eliminate SPOFClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

[MinIO] Investigate load-balancing approaches to eliminate SPOF
Closed, ResolvedPublic
Actions

Related Objects
Search...