Page MenuHomePhabricator

Renderer running hot
Closed, DeclinedPublic

Description

Renderer has scaled out to 6 (it's maximum) and is eating up most of the cluster's cpu. Getting a lot of:

Tue Jan 17 19:57:28 2023 - *** uWSGI listen queue of socket "0.0.0.0:8000" (fd: 3) full !!! (101/100) ***
Tue Jan 17 19:57:29 2023 - *** uWSGI listen queue of socket "0.0.0.0:8000" (fd: 3) full !!! (101/100) ***
Tue Jan 17 19:57:30 2023 - *** uWSGI listen queue of socket "0.0.0.0:8000" (fd: 3) full !!! (101/100) ***
Tue Jan 17 19:57:31 2023 - *** uWSGI listen queue of socket "0.0.0.0:8000" (fd: 3) full !!! (101/100) ***

Related Objects

Event Timeline

Such logs only appear to be coming out of the pods that are running 4 cpu. Those running 2 or close to none (there seem to be three groups 4, 2 or almost 0) do not seem to have this message. Deleting the 4 cpu usage pods seems to have cleared it, in that the pods that replace them do not have the errors or the cpu usage.

the 2 core pods also seemed to continue running at 2 cores. It seems as though renderer revs up, but doesn't slow back down when it is done. Removing the 2 core using systems seemed to drop renderer back down to minimal usage. Possibly this is something about a deployment just starting, as it did just this morning with the magnum cutover (T326554). Though that sounds unlikely. Further investigation on this deployment in what makes it run hot and how to have it drop when it is done is perhaps reasonable.

Haven't seen this in awhile. May still be an issue, but is hard to track with a lack of instances of it.