Page MenuHomePhabricator

Presto is unstable with more than 5 worker nodes
Closed, ResolvedPublicBUG REPORT

Description

Data Engineering Bug Report or Data Problem Form.

What kind of problem are you reporting?

  • Access related problem
  • Service related problem
  • Data related problem
For a service related problem:
  • What is the nature of the issue?

We recently increased the size of the presto cluster from 5 to 15 worker nodes.

However, the result was a reduction in stability of the cluster, with many failed queries and timeouts. We investigated this as an incident in ticket: T325331: Superset: Presto backend: Unable to access some charts

We tried various combinations of new vs old hosts, but it did not seem to be related to which hosts were in use.

  • 5 hosts works ok
  • 10 hosts demonstrates significant instability
  • 15 hosts is even worse

We also tried increasing the amout of JVM heap available to the coordinator process, but it didn't help.

  • What are the steps to reproduce the issue?

Enable and start presto-server processes on those servers where it is currently disabled: an-presto10[06-15]

  • What happens?

Timeouts and instability with presto in superset and from the command-line.

  • What should happen instead?

The cluster should perform faster and more reliably with 15 nodes instead of 5.

For the DE Team to fill out
Which systems does this effect?
  • Hive
  • Druid
  • Superset
  • Turnilo
  • WikiDumps
  • Wikistats
  • Airflow
  • HDFS
  • Goblin
  • Scqoop
  • Dashiki
  • DataHub
  • Spark
  • Jupyter
  • Modern Event Platform
  • Event Logging
  • Other: Presto
Impact Assessment:

Does this problem qualify as an incident?

  • Yes
  • No

Does this violate an SLO?

  • Yes
  • No
Value CalculatorRank
Will this improve the efficiency of a teams workflow?1-3
Does this have an effect of our Core Metrics?1-3
Does this align with our strategic goals?1-3
Is this a blocker for another team?1-3

Event Timeline

Icinga downtime and Alertmanager silence (ID=4f65baff-bd05-4e2f-8578-eae4b972dc3b) set by btullis@cumin1001 for 30 days, 0:00:00 on 10 host(s) and their services with reason: Still not ready to add these new presto servers to the cluster - btullis

an-presto[1006-1015].eqiad.wmnet

Mentioned in SAL (#wikimedia-analytics) [2023-01-30T16:41:44Z] <btullis> started an-presto1006-1015 again, but disabled the presto service on them once again T323783 and T325809

BTullis renamed this task from NEW BUG REPORT Presto cluster instabililty with more than 5 worker nodes to Presto is unstable with more than 5 worker nodes.Feb 7 2023, 3:24 PM
BTullis triaged this task as High priority.
BTullis added subscribers: Stevemunene, nfraison.
BTullis added a subscriber: JAllemandou.

I'm bringing this ticket into the current Shared-Data-Infrastructure sprint. @Stevemunene and @nfraison and I will focus on it.

As @JAllemandou commented in T323783#8570087 we have some other ideas for things that we can tweak to see if it improves stability:

In addition to this, we have a idea to run the presto coordinator on one of the workers, so that we can give it significantly more memory and not worry about its being on the same machine as the hive metastore and mariadb database (an-coord1001)

Finally, we could look again getting the presto-query-logger working, and use any other techniques to help us to profile the presto traffic and understand why the cluster is more unstable the more nodes we have.

Change 889994 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] presto: add 5 nodes to the prod cluster

https://gerrit.wikimedia.org/r/889994

Change 889995 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] presto: add last 5 nodes to prod cluster

https://gerrit.wikimedia.org/r/889995

Change 889994 merged by Nicolas Fraison:

[operations/puppet@production] presto: add 5 nodes to the prod cluster

https://gerrit.wikimedia.org/r/889994

Change 889995 merged by Nicolas Fraison:

[operations/puppet@production] presto: add last 5 nodes to prod cluster

https://gerrit.wikimedia.org/r/889995