a few appservers at a time suffer mcrouter backlogs, leading to high latency
Closed, ResolvedPublic
Actions

Edit Task
Edit Related Tasks...
Create Subtask
Edit Parent Tasks
Edit Subtasks
Merge Duplicates In
Close As Duplicate
Edit Related Objects...
Edit Commits
Edit Mocks
Mute Notifications
Protect as security issue
Award Token
Flag For Later

Assigned To

Authored By

	CDanis
	Dec 11 2019, 12:48 AM

Description

@Joe and @RLazarus have been investigating this.

Evidence of the correlation between latency and mcrouter backlogs https://w.wiki/DeT

Said appservers seem to saturate on CPU (consumption flattops at about 85%).

I tried decreasing the weight of a few as they were in the flattop state, and it did not seem to have any immediate effect.

Event Timeline

CDanis created this task.Dec 11 2019, 12:48 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2019, 12:48 AM

We've changed the niceness of mcrouter to be on par with the niceness of php-fpm; since then, we didn't see such huge lags anymore. Marking this as resolved.

a few appservers at a time suffer mcrouter backlogs, leading to high latencyClosed, ResolvedPublicActions

Description

Event Timeline

a few appservers at a time suffer mcrouter backlogs, leading to high latency
Closed, ResolvedPublic
Actions