High CPU usage from swift-proxy on frontend machines
Open, NormalPublic


Today while having ms-fe1001 depooled for T155095 I noticed its CPU usage was still 50% even with the machine depooled. Turns out some (stray?) swift-proxy processes were consuming one core each. I've collected a bt full from gdb in P4792 and P4791 from one of the processes. I suspect whatever gdb sees might be corrupted because the addresses seem too low to make sense.
Anyways all these processes were the ones left behind by a swift-init proxy restart to support graceful restarts. The fresh swift-proxy processes don't seem to be affected.
perf top reports as much

46.76%  python2.7              [.] 0x00000000000b7671
40.32%  python2.7              [.] _PyTrash_thread_destroy_chain
 7.13%  python2.7              [.] _PyTrash_thread_deposit_object
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2017, 4:05 PM
fgiunchedi changed the title from "high CPU usage from swift-proxy on frontend machines" to "High CPU usage from swift-proxy on frontend machines".Jan 24 2017, 4:07 PM
fgiunchedi triaged this task as "Normal" priority.

Mentioned in SAL (#wikimedia-operations) [2017-01-24T16:12:32Z] <godog> kill stray swift-proxy processes from ms-fe1* T156143