KDC performance tuning for TCP requests
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MoritzMuehlenhoff
	Feb 16 2023, 12:04 PM

Description

See T329525 for context. @nfraison and myself did some tests and going forward we should

Increase the amount of KDC worker daemons
Bump net.core.somaxconn to cope with the bursty nature of tgs-req requests issued by Presto queries

Details

Subject	Repo	Branch	Lines +/-
Adjust monitoring for KDC processes if worker threads are in use	operations/puppet	production	+8 -1
Adapt KDC monitoring to dynamic KDC worker count	operations/puppet	production	+1 -1
Tweak scalability of KDC requests (v2)	operations/puppet	production	+30 -2
Revert "Tweak scalability of KDC requests"	operations/puppet	production	+2 -30
Tweak scalability of KDC requests	operations/puppet	production	+30 -2

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	BUG REPORT	• nfraison	T325809 Presto is unstable with more than 5 worker nodes
Resolved		• nfraison	T329525 Create Presto test clusters with 10 new nodes and try reproduce issue
Resolved		MoritzMuehlenhoff	T329831 KDC performance tuning for TCP requests

Event Timeline

MoritzMuehlenhoff claimed this task.Feb 16 2023, 12:04 PM

MoritzMuehlenhoff triaged this task as Medium priority.

MoritzMuehlenhoff created this task.

Change 889971 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Tweak scalability of KDC requests

https://gerrit.wikimedia.org/r/889971

gerritbot added a project: Patch-For-Review.Feb 17 2023, 12:01 PM

Change 889971 merged by Muehlenhoff:

[operations/puppet@production] Tweak scalability of KDC requests

https://gerrit.wikimedia.org/r/889971

Change 890388 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert "Tweak scalability of KDC requests"

https://gerrit.wikimedia.org/r/890388

Change 890388 merged by Muehlenhoff:

[operations/puppet@production] Revert "Tweak scalability of KDC requests"

https://gerrit.wikimedia.org/r/890388

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2023, 10:29 AM

Change 890389 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Tweak scalability of KDC requests (v2)

https://gerrit.wikimedia.org/r/890389

gerritbot added a project: Patch-For-Review.Feb 20 2023, 10:32 AM

Change 890389 merged by Muehlenhoff:

[operations/puppet@production] Tweak scalability of KDC requests (v2)

https://gerrit.wikimedia.org/r/890389

Change 890393 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Adapt KDC monitoring to dynamic KDC worker count

https://gerrit.wikimedia.org/r/890393

Change 890393 merged by Muehlenhoff:

[operations/puppet@production] Adapt KDC monitoring to dynamic KDC worker count

https://gerrit.wikimedia.org/r/890393

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2023, 11:10 AM

The amount of KDC workers is now configurable via the new profile::kerberos::kdc::workers Hiera setting and has been raised to 8. In the addition net.core.somaxconn sysctl was bumped to 16k. This fixes the request burst issues observed in T329525

Change 891310 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Adjust monitoring for KDC processes if worker threads are in use

https://gerrit.wikimedia.org/r/891310

gerritbot added a project: Patch-For-Review.Feb 22 2023, 3:15 PM

Change 891310 merged by Muehlenhoff:

[operations/puppet@production] Adjust monitoring for KDC processes if worker threads are in use

https://gerrit.wikimedia.org/r/891310

Maintenance_bot removed a project: Patch-For-Review.Feb 23 2023, 9:11 AM

BTullis mentioned this in T358196: [Presto] Use JWT authentication instead of Kerberos for cluster-internal communication.Feb 22 2024, 10:32 AM

BTullis mentioned this in T294772: Superset Timeout Logging.Mar 22 2024, 11:58 AM