Page MenuHomePhabricator

High CPU usage from swift-proxy on frontend machines
Closed, DeclinedPublic

Description

Today while having ms-fe1001 depooled for T155095 I noticed its CPU usage was still 50% even with the machine depooled. Turns out some (stray?) swift-proxy processes were consuming one core each. I've collected a bt full from gdb in P4792 and P4791 from one of the processes. I suspect whatever gdb sees might be corrupted because the addresses seem too low to make sense.
Anyways all these processes were the ones left behind by a swift-init proxy restart to support graceful restarts. The fresh swift-proxy processes don't seem to be affected.
perf top reports as much

46.76%  python2.7              [.] 0x00000000000b7671
40.32%  python2.7              [.] _PyTrash_thread_destroy_chain
 7.13%  python2.7              [.] _PyTrash_thread_deposit_object

Event Timeline

fgiunchedi renamed this task from high CPU usage from swift-proxy on frontend machines to High CPU usage from swift-proxy on frontend machines.Jan 24 2017, 4:07 PM
fgiunchedi triaged this task as Medium priority.

Mentioned in SAL (#wikimedia-operations) [2017-01-24T16:12:32Z] <godog> kill stray swift-proxy processes from ms-fe1* T156143

Hasn't reoccurred through multiple depool cycles, in the meantime swift has been upgraded too, declining