Page MenuHomePhabricator
Create Task
Maniphest T329525

Create Presto test clusters with 10 new nodes and try reproduce issue
Closed, ResolvedPublic
Actions

Assigned To
• nfraison
Authored By
• nfraison
Feb 13 2023, 3:14 PM
Project Tags
Referenced Files
F36855020: 8.workers.tcp.somaxconn.16k.best.behavior.png
Feb 16 2023, 1:28 PM
F36855018: Screenshot from 2023-02-16 14-15-29.png
Feb 16 2023, 1:28 PM
F36852625: wmf.vs.joal.join.automatic.png
Feb 15 2023, 4:57 PM
F36852624: wmf.vs.joal.png
Feb 15 2023, 4:57 PM
F36845408: image.png
Feb 15 2023, 8:31 AM
F36845404: image.png
Feb 15 2023, 8:31 AM
F36825520: image.png
Feb 14 2023, 6:26 PM
F36825493: test-1676387875.log
Feb 14 2023, 5:28 PM
View All 9 Files
Subscribers
Aklapper
BTullis
JAllemandou
jbond
MoritzMuehlenhoff
• nfraison

Description

In order to investigate issue faced when adding 10 nodes to the prod cluster we will create a new cluster connected to the prod and will try reproduce the issue running multiple queries on it

TODO:

  • Update grafana dashboard GC
  • Reset config to rely on default one with only the udp_preference_limit setting and ensure still work
  • Discuss with kdc managing team (for network looking at the low number of loss not sure they can help)
  • See if there are some other kerberos config that can be tuned (knowing we rely on java)
  • re-enable service check on puppet and presto once added back to prod cluster
  • add metrics on jmx exporter (splits state, thread usage of different pool) -> PR
  • revert to default config only add join-distribution-type=AUTOMATIC -> PR
  • remove verbose GC tags: -XX:+PrintGCApplicationConcurrentTime
  • enable tcp first on kdc query
  • write doc with java thread analysis -> https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Java

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
presto: add last 5 nodes to prod clusterNicolas Fraisonoperations/puppetproduction+0 -5
presto: add 5 nodes to the prod clusterNicolas Fraisonoperations/puppetproduction+0 -5
perf(presto): add join-distribution-type to configNicolas Fraisonoperations/puppetproduction+4 -0
perf(presto): remove some configuration tuningNicolas Fraisonoperations/puppetproduction+0 -10
feat(presto): export splits and thread metricsNicolas Fraisonoperations/puppetproduction+20 -0
chore(presto): remove useless gc tag PrintGCApplicationConcurrentTimeNicolas Fraisonoperations/puppetproduction+0 -1
feat(kerberos): add feature to rely on tcp first to query kdcNicolas Fraisonoperations/puppetproduction+16 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• nfraison created this task.Feb 13 2023, 3:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2023, 3:14 PM
• nfraison claimed this task.Feb 13 2023, 3:14 PM
• nfraison added a parent task: T325809: Presto is unstable with more than 5 worker nodes.Feb 13 2023, 3:18 PM

Disabling puppet temporary on an-presto1006 to an-presto1015

sudo disable-puppet "Create presto cluster for perf testing - T329525"
• nfraison added a subscriber: JAllemandou.Feb 14 2023, 4:13 PM

Cluster is available in prod.
To connect to it run from stat1005.eqiad.wmnet (or any other stat) below cli command:

>presto --server https://an-presto1006.eqiad.wmnet:8281 --catalog analytics_hive
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
presto> SHOW TABLES FROM event;
                             Table                              
----------------------------------------------------------------
 advancedsearchrequest                                          
 analytics                                                      
 android_app_appearance_settings_interaction                    
 android_app_session                                            
 android_article_link_preview_interaction                       
 android_article_page_scroll_interaction

@JAllemandou has created a superset dashboard which connect to the new presto cluster: https://superset.wikimedia.org/superset/dashboard/429/

• nfraison added a comment.Feb 14 2023, 4:23 PM

With 9 nodes and 1 coordinator while using the superset dashboard we observe the timeouts and more slower query than current 5 nodes prod.
If we stop 4 nodes we are observing same level of response time than the prod cluster with no more timeout

Prometheus jmx config has been updated to report all available metrics from presto (need to see threads, usage, splits and potentially other ones)

What we observe on queries when slow:

  • Lots of splits -> more than in prod cluster => it seems that depending to the cluster size and conf the number of split per query is not the same (bigger on cluster with more resources)
  • Long duration with some splits in blocked state while nothing else run on the cluster

image.png (1,990×1,206 px, 107 KB)

  • Stage that retrieve data from datanode are fast and not blocked while stage 3/2/1 are blocked for really long time (3/4h-> cpu time)
  • Lots of issues like below ones on the coordinator
WARN        ContinuousTaskStatusFetcher-20230214_161038_00021_tycpp.0.0.0-1268        com.facebook.presto.server.RequestErrorTracker        Error getting task status 20230214_161038_00021_tycpp.0.0.0: https://an-presto1009.eqiad.wmnet:8281/v1/task/20230214_161038_00021_tycpp.0.0.0
 com.facebook.presto.server.ServiceUnavailableException: Server returned SERVICE_UNAVAILABLE: https://an-presto1009.eqiad.wmnet:8281/v1/task/20230214_161038_00021_tycpp.0.0.0/status
         at com.facebook.presto.server.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:57)
        at com.facebook.presto.server.SimpleHttpResponseHandler.onSuccess(SimpleHttpResponseHandler.java:30)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1058)
        at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
  • Same kind of issues from workers
 ERROR        page-buffer-client-callback-90        com.facebook.presto.operator.PageBufferClient        Request to delete https://an-presto1007.eqiad.wmnet:8281/v1/task
/20230214_155554_00009_c5cws.3.0.0/results/6 failed java.lang.RuntimeException: java.util.concurrent.TimeoutException: Total timeout 10000 ms elapsed
DEBUG        page-buffer-client-callback-72        com.facebook.presto.operator.PageBufferClient        Request to https://an-presto1012.eqiad.wmnet:8281/v1/task/async/
20230214_155554_00012_c5cws.4.0.5/results/3/6 failed java.lang.RuntimeException: java.util.concurrent.TimeoutException: Total timeout 10000 ms elapsed
• nfraison added a comment.Feb 14 2023, 5:01 PM

Playing with the different presto configuration tuning doesn't provide any benefits.

We tried first to change scheduler splilts configuration without success

  • node-scheduler.max-splits-per-node: 5000
  • node-scheduler.max-pending-splits-per-task: 100

Then we tried to increase some of the thread pools also without success + no metrics was showing any kind of starvation there:

  • exchange.client-threads: 500
  • exchange.page-buffer-client.max-callback-threads=100
  • exchange.concurrent-request-multiplier=6

Tried also to revert some previous change on max number of task.http-response-threads from 5k to 250 and get back to default for task.concurrency but still no change.

On the system side metrics are not showing any kind of issues:

  • CPU is low on all workers and coordinator
  • GC is low -> by the way our dashboard was badly using increase instead of rate function for computing gc duration per second (gc logs confirm that nothing is happening here)
  • Same on network side

Same things on target systems (hdoop, hive) doesn't show any kind of issues while querries are running

• nfraison added a comment.Feb 14 2023, 5:28 PM

Taking some threaddump on a worker nodes (1 every second during 30s) shows interesting locking between Task- thread and exchange-client thread.
We can observe that some Task- are blocked during more than 30s by this thread while this thread is stuck in that stactrace

	at java.net.PlainDatagramSocketImpl.receive0(Native Method)
	at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
	at java.net.DatagramSocket.receive(DatagramSocket.java:812)
	at sun.security.krb5.internal.UDPClient.receive(NetClient.java:206)
	at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:404)
	at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:364)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.security.krb5.KdcComm.send(KdcComm.java:348)
	at sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:253)
	at sun.security.krb5.KdcComm.send(KdcComm.java:229)
	at sun.security.krb5.KdcComm.send(KdcComm.java:200)
	at sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:221)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:236)
	at sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:477)
	at sun.security.krb5.internal.CredentialsUtil.serviceCredsReferrals(CredentialsUtil.java:369)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:333)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:314)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:169)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:490)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at sun.security.jgss.spnego.SpNegoContext.GSS_initSecContext(SpNegoContext.java:882)
	at sun.security.jgss.spnego.SpNegoContext.initSecContext(SpNegoContext.java:317)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.facebook.airlift.http.client.spnego.SpnegoAuthentication$1.authenticate(SpnegoAuthentication.java:156)
	at com.facebook.airlift.http.client.spnego.SpnegoAuthentication$1.lambda$apply$0(SpnegoAuthentication.java:133)
	at com.facebook.airlift.http.client.spnego.SpnegoAuthentication$1$$Lambda$1109/669855157.run(Unknown Source)
	at net.jodah.failsafe.Functions.lambda$supplierOf$12(Functions.java:296)
	at net.jodah.failsafe.Functions$$Lambda$1113/1615547818.get(Unknown Source)
	at net.jodah.failsafe.Functions.lambda$resultSupplierOf$11(Functions.java:283)
	at net.jodah.failsafe.Functions$$Lambda$1115/1569948075.get(Unknown Source)
	at net.jodah.failsafe.internal.executor.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:67)
	at net.jodah.failsafe.internal.executor.RetryPolicyExecutor$$Lambda$1117/1495411764.get(Unknown Source)
	at net.jodah.failsafe.Execution.executeSync(Execution.java:117)
	at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:319)
	at net.jodah.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:219)
	at com.facebook.airlift.http.client.spnego.SpnegoAuthentication$1.apply(SpnegoAuthentication.java:133)
	at org.eclipse.jetty.client.HttpConnection.applyAuthentication(HttpConnection.java:178)
	at org.eclipse.jetty.client.HttpConnection.normalizeRequest(HttpConnection.java:156)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP$Delegate.send(HttpConnectionOverHTTP.java:245)
	at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.send(HttpConnectionOverHTTP.java:122)
	at org.eclipse.jetty.client.http.HttpDestinationOverHTTP.send(HttpDestinationOverHTTP.java:38)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:346)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:304)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:294)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:269)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:246)
	at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:582)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:726)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:718)
	at com.facebook.airlift.http.client.jetty.JettyHttpClient.executeAsync(JettyHttpClient.java:560)
	at com.facebook.presto.operator.HttpRpcShuffleClient.getResults(HttpRpcShuffleClient.java:89)
	at com.facebook.presto.operator.PageBufferClient.sendGetResults(PageBufferClient.java:274)
	at com.facebook.presto.operator.PageBufferClient.initiateRequest(PageBufferClient.java:263)
	at com.facebook.presto.operator.PageBufferClient.lambda$scheduleRequest$0(PageBufferClient.java:240)
	at com.facebook.presto.operator.PageBufferClient$$Lambda$3117/1681636311.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Here is an exemple of threaddump:

test-1676387875.log1 MBDownload

BTullis subscribed.Feb 14 2023, 5:30 PM
• nfraison added a comment.EditedFeb 14 2023, 6:26 PM

The stack shows that we are blocked on kerberos auth.

Taking a network traces to confirm that we are loosing some packets (our kerberos config rely on udp) and that is the root cause of those long thread block.
We can indeed saw few kerberos TGS-REQ without any response:

image.png (1,287×183 px, 67 KB)

Adding this parameter to the krb5.conf to enforce the usage of TCP for first try:

udp_preference_limit =1

With that parameter number of WARN on the coordinator have drastically been reduce and now the query never timeouts from superset

TODO:

  • Update grafana dashboard GC
  • Reset config to rely on default one with only the udp_preference_limit setting and ensure still work
  • Discuss with kdc managing team (for network looking at the low number of loss not sure they can help)
  • See if there are some other kerberos config that can be tuned (knowing we rely on java)
  • re-enable service check on puppet and presto once added back to prod cluster
  • add metrics on jmx exporter (splits state, thread usage of different pool)
• nfraison added a comment.Feb 15 2023, 8:31 AM

Reset the cluster with default config to ensure that the udp_preference_limit is the only parameter that solve the issue

#node-scheduler.max-splits-per-node=500
#task.concurrency=32
#task.http-response-threads=5000
#task.max-worker-threads=192

Before applying the parameter:

image.png (945×1,219 px, 262 KB)

After applying the parameter

image.png (939×1,222 px, 260 KB)

Also we do not observe anymore the pattern on workers with lots of timeouts

JAllemandou added a comment.Feb 15 2023, 9:12 AM

Awesome debugging :)

• nfraison added a comment.Feb 15 2023, 9:17 AM

An other potential solution if TCP for kerberos is not fine is to rely on timeout and retry option on krb5.conf

max_retries = 3
kdc_timeout = 3000

But it is less efficient and still leads to some java.util.concurrent.TimeoutException within workers and coordinators

• nfraison added a comment.Feb 15 2023, 9:41 AM

From network switch side we do not observe any error:

Does not seems to be linked to network issue

• nfraison added a comment.Feb 15 2023, 10:26 AM

On the kdc server we can observe multiple errors:
https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=krb1001&var-datasource=thanos&var-cluster=misc&from=now-24h&to=now&viewPanel=20

For this issue it could be fine to increase net.core.rmem_max and net.core.netdev_max_backlog

For. ex:
sudo sysctl -w net.core.rmem_max=33554432
sudo sysctl -w net.core.netdev_max_backlog=5000

FI those settings have already been tuned to

/proc/sys/net/core/rmem_max
16777216
/proc/sys/net/core/netdev_max_backlog 
2500
• nfraison added subscribers: jbond, MoritzMuehlenhoff.Feb 15 2023, 10:40 AM

An other approach will be to increase number of krb5kdc workers

The -w numworkers option tells the KDC to fork numworkers processes to listen to the KDC ports and process requests in parallel.  The top level KDC process (whose pid is recorded in the pid file if the -P option is also given) acts as a su‐
       pervisor.  The supervisor will relay SIGHUP signals to the worker subprocesses, and will terminate the worker subprocess if the it is itself terminated or if any other worker process exits.

@MoritzMuehlenhoff @jbond we are facing an issue when increasing our presto clusters due to some udp tgs-req loss on the kdc server.
Moving to tcp for first try (udp_preference_limit set to 1 on client side) solve well the issue -> do you foresee an issue in relying on tcp here?
We could also try to do some tuning on the network stack of the kdc/increase number of krb5kdc workers.
Do you have some recommendation or other ideas?

MoritzMuehlenhoff added a comment.Feb 15 2023, 11:35 AM
In T329525#8617850, @nfraison wrote:

@MoritzMuehlenhoff @jbond we are facing an issue when increasing our presto clusters due to some udp tgs-req loss on the kdc server.
Moving to tcp for first try (udp_preference_limit set to 1 on client side) solve well the issue -> do you foresee an issue in relying on tcp here?

Using TCP should be fine, historically Kerberos was designed for UDP and TCP was only used for some oversized requests. IIRC recent versions of Windows (which implements Kerberos for Active Directory authentication) even default to TCP.

We could also try to do some tuning on the network stack of the kdc/increase number of krb5kdc workers.
Do you have some recommendation or other ideas?

We can raise the number of KDC workers, let's sync up on this tomorrow? (Need to work on some other things today)

• nfraison added a comment.Feb 15 2023, 12:20 PM

@MoritzMuehlenhoff thks for the feedback, I'm available tomorrow for the sync ping me when you will have some time.

• nfraison added a comment.Feb 15 2023, 12:37 PM

Increase a worker is only a matter of an env var to add in /etc/default/krb5-kdc

# Automatically generated.  Only the value of DAEMON_ARGS will be preserved.
# If you change anything in this file other than DAEMON_ARGS, first run
# dpkg-reconfigure krb5-kdc and disable managing the KDC configuration with
# debconf.  Otherwise, changes will be overwritten.
DAEMON_ARGS=" -w 5"
• nfraison updated the task description. (Show Details)Feb 15 2023, 1:57 PM
• nfraison updated the task description. (Show Details)Feb 15 2023, 2:03 PM
• nfraison added a comment.Feb 15 2023, 4:57 PM

Here are some result relying on wmf vs joal table and relying on join automatic:

  • default join:

wmf.vs.joal.png (957×933 px, 201 KB)

  • automatic join:

wmf.vs.joal.join.automatic.png (939×1,120 px, 255 KB)

1.29k splits => wmf table
1.03k splits => joal table

Queries:

wmf:
SELECT date_trunc('month', CAST("date" AS TIMESTAMP)) AS "__timestamp", sum("namespace_zero_distinct_editors") AS "number of editors" FROM (SELECT DATE(DATE_PARSE("month", '%Y-%m')) AS "date", wiki_db AS wiki_db, countries.name AS country, activity_level AS activity_level, users_are_anonymous AS users_are_anonymous, CASE WHEN country_code IN ('AL', 'AM', 'AZ', 'BG', 'BA', 'BY', 'CY', 'CZ', 'EE', 'GE', 'GR', 'HR', 'HU', 'KZ', 'KG', 'LT', 'LV', 'MD', 'MK', 'ME', 'PL', 'RO', 'RU', 'RS', 'SK', 'SI', 'TJ', 'TM', 'TR', 'UA', 'UZ', 'XK') THEN 'Central and Eastern Europe and Central Asia' WHEN country_code IN ('AS', 'AU', 'BN', 'CN', 'FJ', 'FM', 'GU', 'HK', 'ID', 'JP', 'KH', 'KI', 'KR', 'LA', 'MO', 'MH', 'MM', 'MN', 'MP', 'MY', 'NC', 'NR', 'NZ', 'PH', 'PW', 'PG', 'KP', 'PF', 'SG', 'SB', 'TH', 'TL', 'TO', 'TV', 'VN', 'VU', 'WS') THEN 'East, Southeast Asia, and Pacific' WHEN country_code IN ('AW', 'AR', 'AG', 'BS', 'BZ', 'BO', 'BR', 'BB', 'CL', 'CO', 'CR', 'CU', 'CW', 'KY', 'DM', 'DO', 'EC', 'GD', 'GT', 'GY', 'HN', 'HT', 'JM', 'KN', 'LC', 'MF', 'MX', 'NI', 'PA', 'PE', 'PR', 'PY', 'SV', 'SR', 'SX', 'TC', 'TT', 'UY', 'VC', 'VE', 'VG', 'VI') THEN 'Latin America and Caribbean' WHEN country_code IN ('AE', 'BH', 'DJ', 'DZ', 'EG', 'IR', 'IQ', 'IL', 'JO', 'KW', 'LB', 'LY', 'MA', 'MT', 'OM', 'PS', 'QA', 'SA', 'SY', 'TN', 'YE') THEN 'Middle East and North Africa' WHEN country_code IN ('BM', 'CA', 'US') THEN 'North America' WHEN country_code IN ('AD', 'AT', 'BE', 'CH', 'DE', 'DK', 'ES', 'FI', 'FR', 'FO', 'GB', 'GI', 'GL', 'IM', 'IE', 'IS', 'IT', 'LI', 'LU', 'MC', 'NL', 'NO', 'PT', 'SM', 'SE') THEN 'Northern and Western Europe' WHEN country_code IN ('AF', 'BD', 'BT', 'IN', 'LK', 'MV', 'NP', 'PK') THEN 'South Asia' WHEN country_code IN ('AO', 'BI', 'BJ', 'BF', 'BW', 'CF', 'CI', 'CM', 'CD', 'CG', 'KM', 'CV', 'ER', 'ET', 'GA', 'GH', 'GN', 'GM', 'GW', 'GQ', 'KE', 'LR', 'LS', 'MG', 'ML', 'MZ', 'MR', 'MU', 'MW', 'NA', 'NE', 'NG', 'RW', 'SD', 'SN', 'SL', 'SO', 'SS', 'ST', 'SZ', 'SC', 'TD', 'TG', 'TZ', 'UG', 'ZA', 'ZM', 'ZW') THEN 'Sub-Saharan Africa' ELSE 'NULL' END AS region, SUM(distinct_editors) AS distinct_editors, SUM(namespace_zero_distinct_editors) AS namespace_zero_distinct_editors FROM wmf.geoeditors_monthly INNER JOIN canonical_data.countries countries ON country_code = countries.iso_code GROUP BY DATE(DATE_PARSE("month", '%Y-%m')), wiki_db, countries.name, activity_level, users_are_anonymous, CASE WHEN country_code IN ('AL', 'AM', 'AZ', 'BG', 'BA', 'BY', 'CY', 'CZ', 'EE', 'GE', 'GR', 'HR', 'HU', 'KZ', 'KG', 'LT', 'LV', 'MD', 'MK', 'ME', 'PL', 'RO', 'RU', 'RS', 'SK', 'SI', 'TJ', 'TM', 'TR', 'UA', 'UZ', 'XK') THEN 'Central and Eastern Europe and Central Asia' WHEN country_code IN ('AS', 'AU', 'BN', 'CN', 'FJ', 'FM', 'GU', 'HK', 'ID', 'JP', 'KH', 'KI', 'KR', 'LA', 'MO', 'MH', 'MM', 'MN', 'MP', 'MY', 'NC', 'NR', 'NZ', 'PH', 'PW', 'PG', 'KP', 'PF', 'SG', 'SB', 'TH', 'TL', 'TO', 'TV', 'VN', 'VU', 'WS') THEN 'East, Southeast Asia, and Pacific' WHEN country_code IN ('AW', 'AR', 'AG', 'BS', 'BZ', 'BO', 'BR', 'BB', 'CL', 'CO', 'CR', 'CU', 'CW', 'KY', 'DM', 'DO', 'EC', 'GD', 'GT', 'GY', 'HN', 'HT', 'JM', 'KN', 'LC', 'MF', 'MX', 'NI', 'PA', 'PE', 'PR', 'PY', 'SV', 'SR', 'SX', 'TC', 'TT', 'UY', 'VC', 'VE', 'VG', 'VI') THEN 'Latin America and Caribbean' WHEN country_code IN ('AE', 'BH', 'DJ', 'DZ', 'EG', 'IR', 'IQ', 'IL', 'JO', 'KW', 'LB', 'LY', 'MA', 'MT', 'OM', 'PS', 'QA', 'SA', 'SY', 'TN', 'YE') THEN 'Middle East and North Africa' WHEN country_code IN ('BM', 'CA', 'US') THEN 'North America' WHEN country_code IN ('AD', 'AT', 'BE', 'CH', 'DE', 'DK', 'ES', 'FI', 'FR', 'FO', 'GB', 'GI', 'GL', 'IM', 'IE', 'IS', 'IT', 'LI', 'LU', 'MC', 'NL', 'NO', 'PT', 'SM', 'SE') THEN 'Northern and Western Europe' WHEN country_code IN ('AF', 'BD', 'BT', 'IN', 'LK', 'MV', 'NP', 'PK') THEN 'South Asia' WHEN country_code IN ('AO', 'BI', 'BJ', 'BF', 'BW', 'CF', 'CI', 'CM', 'CD', 'CG', 'KM', 'CV', 'ER', 'ET', 'GA', 'GH', 'GN', 'GM', 'GW', 'GQ', 'KE', 'LR', 'LS', 'MG', 'ML', 'MZ', 'MR', 'MU', 'MW', 'NA', 'NE', 'NG', 'RW', 'SD', 'SN', 'SL', 'SO', 'SS', 'ST', 'SZ', 'SC', 'TD', 'TG', 'TZ', 'UG', 'ZA', 'ZM', 'ZW') THEN 'Sub-Saharan Africa' ELSE 'NULL' END) AS "virtual_table" WHERE "users_are_anonymous" = false AND "region" = 'Sub-Saharan Africa' GROUP BY date_trunc('month', CAST("date" AS TIMESTAMP)) LIMIT 50000;
joal:
SELECT date_trunc('month', CAST("date" AS TIMESTAMP)) AS "__timestamp", sum("namespace_zero_distinct_editors") AS "number of editors" FROM (SELECT DATE(DATE_PARSE("month", '%Y-%m')) AS "date", wiki_db AS wiki_db, countries.name AS country, activity_level AS activity_level, users_are_anonymous AS users_are_anonymous, CASE WHEN country_code IN ('AL', 'AM', 'AZ', 'BG', 'BA', 'BY', 'CY', 'CZ', 'EE', 'GE', 'GR', 'HR', 'HU', 'KZ', 'KG', 'LT', 'LV', 'MD', 'MK', 'ME', 'PL', 'RO', 'RU', 'RS', 'SK', 'SI', 'TJ', 'TM', 'TR', 'UA', 'UZ', 'XK') THEN 'Central and Eastern Europe and Central Asia' WHEN country_code IN ('AS', 'AU', 'BN', 'CN', 'FJ', 'FM', 'GU', 'HK', 'ID', 'JP', 'KH', 'KI', 'KR', 'LA', 'MO', 'MH', 'MM', 'MN', 'MP', 'MY', 'NC', 'NR', 'NZ', 'PH', 'PW', 'PG', 'KP', 'PF', 'SG', 'SB', 'TH', 'TL', 'TO', 'TV', 'VN', 'VU', 'WS') THEN 'East, Southeast Asia, and Pacific' WHEN country_code IN ('AW', 'AR', 'AG', 'BS', 'BZ', 'BO', 'BR', 'BB', 'CL', 'CO', 'CR', 'CU', 'CW', 'KY', 'DM', 'DO', 'EC', 'GD', 'GT', 'GY', 'HN', 'HT', 'JM', 'KN', 'LC', 'MF', 'MX', 'NI', 'PA', 'PE', 'PR', 'PY', 'SV', 'SR', 'SX', 'TC', 'TT', 'UY', 'VC', 'VE', 'VG', 'VI') THEN 'Latin America and Caribbean' WHEN country_code IN ('AE', 'BH', 'DJ', 'DZ', 'EG', 'IR', 'IQ', 'IL', 'JO', 'KW', 'LB', 'LY', 'MA', 'MT', 'OM', 'PS', 'QA', 'SA', 'SY', 'TN', 'YE') THEN 'Middle East and North Africa' WHEN country_code IN ('BM', 'CA', 'US') THEN 'North America' WHEN country_code IN ('AD', 'AT', 'BE', 'CH', 'DE', 'DK', 'ES', 'FI', 'FR', 'FO', 'GB', 'GI', 'GL', 'IM', 'IE', 'IS', 'IT', 'LI', 'LU', 'MC', 'NL', 'NO', 'PT', 'SM', 'SE') THEN 'Northern and Western Europe' WHEN country_code IN ('AF', 'BD', 'BT', 'IN', 'LK', 'MV', 'NP', 'PK') THEN 'South Asia' WHEN country_code IN ('AO', 'BI', 'BJ', 'BF', 'BW', 'CF', 'CI', 'CM', 'CD', 'CG', 'KM', 'CV', 'ER', 'ET', 'GA', 'GH', 'GN', 'GM', 'GW', 'GQ', 'KE', 'LR', 'LS', 'MG', 'ML', 'MZ', 'MR', 'MU', 'MW', 'NA', 'NE', 'NG', 'RW', 'SD', 'SN', 'SL', 'SO', 'SS', 'ST', 'SZ', 'SC', 'TD', 'TG', 'TZ', 'UG', 'ZA', 'ZM', 'ZW') THEN 'Sub-Saharan Africa' ELSE 'NULL' END AS region, SUM(distinct_editors) AS distinct_editors, SUM(namespace_zero_distinct_editors) AS namespace_zero_distinct_editors FROM joal.geoeditors_monthly INNER JOIN canonical_data.countries countries ON country_code = countries.iso_code GROUP BY DATE(DATE_PARSE("month", '%Y-%m')), wiki_db, countries.name, activity_level, users_are_anonymous, CASE WHEN country_code IN ('AL', 'AM', 'AZ', 'BG', 'BA', 'BY', 'CY', 'CZ', 'EE', 'GE', 'GR', 'HR', 'HU', 'KZ', 'KG', 'LT', 'LV', 'MD', 'MK', 'ME', 'PL', 'RO', 'RU', 'RS', 'SK', 'SI', 'TJ', 'TM', 'TR', 'UA', 'UZ', 'XK') THEN 'Central and Eastern Europe and Central Asia' WHEN country_code IN ('AS', 'AU', 'BN', 'CN', 'FJ', 'FM', 'GU', 'HK', 'ID', 'JP', 'KH', 'KI', 'KR', 'LA', 'MO', 'MH', 'MM', 'MN', 'MP', 'MY', 'NC', 'NR', 'NZ', 'PH', 'PW', 'PG', 'KP', 'PF', 'SG', 'SB', 'TH', 'TL', 'TO', 'TV', 'VN', 'VU', 'WS') THEN 'East, Southeast Asia, and Pacific' WHEN country_code IN ('AW', 'AR', 'AG', 'BS', 'BZ', 'BO', 'BR', 'BB', 'CL', 'CO', 'CR', 'CU', 'CW', 'KY', 'DM', 'DO', 'EC', 'GD', 'GT', 'GY', 'HN', 'HT', 'JM', 'KN', 'LC', 'MF', 'MX', 'NI', 'PA', 'PE', 'PR', 'PY', 'SV', 'SR', 'SX', 'TC', 'TT', 'UY', 'VC', 'VE', 'VG', 'VI') THEN 'Latin America and Caribbean' WHEN country_code IN ('AE', 'BH', 'DJ', 'DZ', 'EG', 'IR', 'IQ', 'IL', 'JO', 'KW', 'LB', 'LY', 'MA', 'MT', 'OM', 'PS', 'QA', 'SA', 'SY', 'TN', 'YE') THEN 'Middle East and North Africa' WHEN country_code IN ('BM', 'CA', 'US') THEN 'North America' WHEN country_code IN ('AD', 'AT', 'BE', 'CH', 'DE', 'DK', 'ES', 'FI', 'FR', 'FO', 'GB', 'GI', 'GL', 'IM', 'IE', 'IS', 'IT', 'LI', 'LU', 'MC', 'NL', 'NO', 'PT', 'SM', 'SE') THEN 'Northern and Western Europe' WHEN country_code IN ('AF', 'BD', 'BT', 'IN', 'LK', 'MV', 'NP', 'PK') THEN 'South Asia' WHEN country_code IN ('AO', 'BI', 'BJ', 'BF', 'BW', 'CF', 'CI', 'CM', 'CD', 'CG', 'KM', 'CV', 'ER', 'ET', 'GA', 'GH', 'GN', 'GM', 'GW', 'GQ', 'KE', 'LR', 'LS', 'MG', 'ML', 'MZ', 'MR', 'MU', 'MW', 'NA', 'NE', 'NG', 'RW', 'SD', 'SN', 'SL', 'SO', 'SS', 'ST', 'SZ', 'SC', 'TD', 'TG', 'TZ', 'UG', 'ZA', 'ZM', 'ZW') THEN 'Sub-Saharan Africa' ELSE 'NULL' END) AS "virtual_table" WHERE "users_are_anonymous" = false AND "region" = 'Sub-Saharan Africa' GROUP BY date_trunc('month', CAST("date" AS TIMESTAMP)) LIMIT 50000;

As expected things are faster with the joal table (less splits to compute). With automatic join it seems to improve performance on the wmf based query while note having same impact on the joal based one. At least it doesn't have bad impacts so will add this to the config

• nfraison updated the task description. (Show Details)Feb 15 2023, 4:58 PM
• nfraison updated the task description. (Show Details)
• nfraison updated the task description. (Show Details)Feb 15 2023, 5:05 PM
• nfraison updated the task description. (Show Details)Feb 15 2023, 5:11 PM
• nfraison updated the task description. (Show Details)Feb 15 2023, 5:31 PM
gerritbot added a comment.Feb 15 2023, 5:35 PM

Change 889580 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] chore(presto): remove useless gc tag PrintGCApplicationConcurrentTime

https://gerrit.wikimedia.org/r/889580

gerritbot added a project: Patch-For-Review.Feb 15 2023, 5:35 PM

Change 889581 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] chore(presto): remove configuration tuning added while trying to increase cluster size

https://gerrit.wikimedia.org/r/889581

gerritbot added a comment.Feb 15 2023, 5:35 PM

Change 889582 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] perf(presto): add join-distribution-type to config

https://gerrit.wikimedia.org/r/889582

gerritbot added a comment.Feb 16 2023, 9:30 AM

Change 889756 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] feat(presto): export splits and thread metrics

https://gerrit.wikimedia.org/r/889756

MoritzMuehlenhoff mentioned this in T329831: KDC performance tuning for TCP requests.Feb 16 2023, 12:04 PM
gerritbot added a comment.Feb 16 2023, 12:47 PM

Change 889800 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] feat(kerberos): rely on tcp first to query kdc

https://gerrit.wikimedia.org/r/889800

• nfraison updated the task description. (Show Details)Feb 16 2023, 1:14 PM
• nfraison added a comment.EditedFeb 16 2023, 1:28 PM

Result of test performed with @MoritzMuehlenhoff:

  • Increase number of kdc workers to 8 while still relying on UDP from presto nodes

The number of UDP error decrease by almost 3 but still there and still observing slow queries

Screenshot from 2023-02-16 14-15-29.png (2,485×1,102 px, 46 KB)

  • Increasing to 16 workers doesn't improve the behavior
  • Running kdc with 8 workers and relying on TCP from presto nodes

With that configuration no more timeout are observed from coordinator and workers (with 1 workers there was still few timeouts)
Also the query duration is improved compare to the duration with 1 worker and TCP usage
But we are observing some TCP: request_sock_TCP: Possible SYN flooding on port 88. Sending cookies. Check SNMP counters.

  • Running kdc with 8 workers and tunning kernel parameter net.core.somaxconn from 1024 to 16384 + rely on TCP from presto nodes

Same behavior than previously from presto + not facing anymore the SYN flooding
Will go to that configuration

Here are some query measure

  • 1 worker + TCP + somaxconn: 1024

image.png (939×1,222 px, 260 KB)

  • 8 worker + TCP + somaxconn: 16384

8.workers.tcp.somaxconn.16k.best.behavior.png (942×1,266 px, 259 KB)

Some of the query are at least 2 times faster

MoritzMuehlenhoff mentioned this in T329839: Migrate Kerberos clients towards TCP.Feb 16 2023, 2:06 PM
gerritbot added a comment.Feb 16 2023, 2:42 PM

Change 889800 merged by Nicolas Fraison:

[operations/puppet@production] feat(kerberos): add feature to rely on tcp first to query kdc

https://gerrit.wikimedia.org/r/889800

gerritbot added a comment.Feb 16 2023, 4:43 PM

Change 889580 merged by Nicolas Fraison:

[operations/puppet@production] chore(presto): remove useless gc tag PrintGCApplicationConcurrentTime

https://gerrit.wikimedia.org/r/889580

• nfraison updated the task description. (Show Details)Feb 17 2023, 9:04 AM
gerritbot added a comment.Feb 17 2023, 12:09 PM

Change 889756 merged by Nicolas Fraison:

[operations/puppet@production] feat(presto): export splits and thread metrics

https://gerrit.wikimedia.org/r/889756

gerritbot added a comment.Feb 17 2023, 2:40 PM

Change 889994 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] presto: add 5 nodes to the prod cluster

https://gerrit.wikimedia.org/r/889994

gerritbot added a comment.Feb 17 2023, 2:40 PM

Change 889995 had a related patch set uploaded (by Nicolas Fraison; author: Nicolas Fraison):

[operations/puppet@production] presto: add last 5 nodes to prod cluster

https://gerrit.wikimedia.org/r/889995

gerritbot added a comment.Feb 20 2023, 11:06 AM

Change 889581 merged by Nicolas Fraison:

[operations/puppet@production] perf(presto): remove some configuration tuning

https://gerrit.wikimedia.org/r/889581

MoritzMuehlenhoff closed subtask T329831: KDC performance tuning for TCP requests as Resolved.Feb 20 2023, 11:39 AM
gerritbot added a comment.Feb 20 2023, 12:15 PM

Change 889582 merged by Nicolas Fraison:

[operations/puppet@production] perf(presto): add join-distribution-type to config

https://gerrit.wikimedia.org/r/889582

• nfraison updated the task description. (Show Details)Feb 20 2023, 12:16 PM
Stashbot added a comment.Feb 20 2023, 12:29 PM

Mentioned in SAL (#wikimedia-analytics) [2023-02-20T12:29:08Z] <nfraison> restart presto coordinator on an-coord1001 to take in account new configs T329525

Stashbot added a comment.Feb 20 2023, 12:32 PM

Mentioned in SAL (#wikimedia-analytics) [2023-02-20T12:32:33Z] <nfraison> roll-restart presto workers on an-presto100[1-5] to take in account new configs T329525

gerritbot added a comment.Feb 20 2023, 12:44 PM

Change 889994 merged by Nicolas Fraison:

[operations/puppet@production] presto: add 5 nodes to the prod cluster

https://gerrit.wikimedia.org/r/889994

JArguello-WMF moved this task from In Progress to Done on the Shared-Data-Infrastructure (2022-23 Q4 Wrap up) board.Feb 20 2023, 1:37 PM
gerritbot added a comment.Feb 21 2023, 9:34 AM

Change 889995 merged by Nicolas Fraison:

[operations/puppet@production] presto: add last 5 nodes to prod cluster

https://gerrit.wikimedia.org/r/889995

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2023, 10:10 AM
JArguello-WMF closed this task as Resolved.Feb 21 2023, 1:48 PM
xcollazo awarded a token.Feb 21 2023, 4:12 PM
MoritzMuehlenhoff mentioned this in T331123: Tweak Kerberos auth logging.Mar 3 2023, 12:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits