Page MenuHomePhabricator

[Trino] Develop procedure and scripting for Trino cluster maintenance.
Closed, ResolvedPublic

Description

We need to be able to work on the Trino cluster without causing issues for things that use it.

There should be separate procedures for coordinator vs worker, since we have enough workers to be able to work on them individually while the rest of the cluster keeps chugging along. To that end:

  • Script to poll what each worker is doing, to determine when a worker is busy vs idle. Is there anything off the shelf for this? We're doing this manually now via query in trino-cli.
  • Script (same script?) to trigger a graceful shutdown on a Trino worker.

Event Timeline

We can use python's trino client to query system.runtime.(tasks|queries|notes) on an interval, I have that much working. Currently looking at how to make the output useful, and to filter away the system.runtime.* queries.

The script trino-top is deployed to fran2001, it uses the dagster virtualenv for its python trino module. Graceful-shutdown signaling works in virtualbox, but it requires firewall access and trino API auth configuration for it to work in production.

Jgreen moved this task from In Progress to Done on the fundraising-tech-ops board.

With API permissions deployed, trino-top works to gracefully shut down workers and provide visibility for worker status. We're able to use this to perform software maintenance on workers on the fly without causing disruption.