While looking at CPU usage of instances on labs, I found out that `tools-webgrid-lighttpd-*` have high user CPU usage being reported. That seems to be leaking over time as shown for 5 months (CPU usage is number of CPU being busy):
{F11189248}
Over a week:
{F11189256}
It seems we have the equivalent of 90 CPU being stuck at 100%.
**[[ https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?orgId=1&var-project=tools&var-server=tools-webgrid-lighttpd-1401&var-server=tools-webgrid-lighttpd-1402&var-server=tools-webgrid-lighttpd-1403&var-server=tools-webgrid-lighttpd-1404&var-server=tools-webgrid-lighttpd-1405&var-server=tools-webgrid-lighttpd-1406&var-server=tools-webgrid-lighttpd-1407&var-server=tools-webgrid-lighttpd-1408&var-server=tools-webgrid-lighttpd-1409&var-server=tools-webgrid-lighttpd-1410&var-server=tools-webgrid-lighttpd-1411&var-server=tools-webgrid-lighttpd-1412&var-server=tools-webgrid-lighttpd-1413&var-server=tools-webgrid-lighttpd-1414&var-server=tools-webgrid-lighttpd-1415&var-server=tools-webgrid-lighttpd-1416&var-server=tools-webgrid-lighttpd-1417&var-server=tools-webgrid-lighttpd-1418&var-server=tools-webgrid-lighttpd-1419&var-server=tools-webgrid-lighttpd-1420&var-server=tools-webgrid-lighttpd-1421&var-server=tools-webgrid-lighttpd-1422&var-server=tools-webgrid-lighttpd-1424&var-server=tools-webgrid-lighttpd-1425&var-server=tools-webgrid-lighttpd-1426&var-server=tools-webgrid-lighttpd-1427&var-server=tools-webgrid-lighttpd-1428&from=now-30d&to=now | View of CPU usage for tools-webrid-lighttpd-* instances ]]**.
Based on:
```
lang=sh
# List proces per cputime, percent cpu, command and username
# Filter on cputime of 1 day or more and current cpu being at least 10%
cmd="ps -e -o cputime,pcpu,cmd,user|grep -P '^\d+-.* \d\d\.'|sort -n|sed -e 's%\(^\|\s\+\)% | %g'"
for i in $(seq 1401 1428); do
ssh "tools-webgrid-lighttpd-$i.tools.eqiad.wmflabs" "$cmd" | sed -e "s%^%| $i %"
done
```
The top offender seems to be `tools.jembot`:
| Instance | Cumulative time | Utilization | Command | User
|--|--|--|--
| 1401 | 2-13:21:42 | 32.5 | /usr/bin/php-cgi | tools.jembot
| 1401 | 2-13:25:16 | 32.5 | /usr/bin/php-cgi | tools.jembot
| 1401 | 6-08:01:50 | 80.5 | /usr/bin/php-cgi | tools.jembot
| 1401 | 35-05:49:02 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1401 | 35-14:02:15 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1401 | 150-13:44:30 | 94.9 | /usr/bin/php-cgi | tools.blockcalc
| 1402 | 34-07:53:25 | 72.1 | /usr/bin/php-cgi | tools.jembot
| 1402 | 34-07:55:13 | 72.1 | /usr/bin/php-cgi | tools.jembot
| 1402 | 34-07:59:00 | 72.1 | /usr/bin/php-cgi | tools.jembot
| 1402 | 34-07:59:33 | 72.1 | /usr/bin/php-cgi | tools.jembot
| 1402 | 115-16:53:51 | 99.5 | /usr/bin/php-cgi | tools.jembot
| 1404 | 142-06:41:59 | 89.7 | /usr/bin/php-cgi | tools.wam
| 1405 | 51-18:15:59 | 72.5 | /usr/bin/php-cgi | tools.jembot
| 1405 | 51-18:22:31 | 72.5 | /usr/bin/php-cgi | tools.jembot
| 1405 | 51-18:30:20 | 72.5 | /usr/bin/php-cgi | tools.jembot
| 1405 | 51-18:39:53 | 72.5 | /usr/bin/php-cgi | tools.jembot
| 1405 | 146-08:42:45 | 92.2 | /usr/bin/php-cgi | tools.dupdet
| 1407 | 46-08:18:50 | 72.0 | /usr/bin/php-cgi | tools.jembot
| 1407 | 46-08:22:04 | 72.0 | /usr/bin/php-cgi | tools.jembot
| 1407 | 46-08:24:32 | 72.0 | /usr/bin/php-cgi | tools.jembot
| 1407 | 46-08:30:14 | 72.0 | /usr/bin/php-cgi | tools.jembot
| 1407 | 129-05:07:39 | 99.4 | /usr/bin/php-cgi | tools.wsexport
| 1408 | 2-01:53:59 | 39.4 | /usr/bin/php-cgi | tools.jembot
| 1408 | 2-01:54:32 | 39.4 | /usr/bin/php-cgi | tools.jembot
| 1408 | 38-11:48:46 | 86.0 | /usr/bin/php-cgi | tools.spellcheck
| 1408 | 41-14:47:46 | 86.6 | /usr/bin/php-cgi | tools.spellcheck
| 1408 | 49-02:31:07 | 97.5 | /usr/bin/php-cgi | tools.pinyin-wiki
| 1408 | 50-21:48:15 | 94.8 | /usr/bin/php-cgi | tools.spellcheck
| 1409 | 15-09:04:39 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1409 | 15-09:06:07 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1409 | 15-09:06:18 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1409 | 15-09:06:45 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1409 | 129-19:34:52 | 99.5 | /usr/bin/php-cgi | tools.jembot
| 1409 | 129-19:35:24 | 99.5 | /usr/bin/php-cgi | tools.jembot
| 1410 | 43-16:32:51 | 73.3 | /usr/bin/php-cgi | tools.jembot
| 1410 | 43-16:38:04 | 73.3 | /usr/bin/php-cgi | tools.jembot
| 1410 | 43-16:44:53 | 73.3 | /usr/bin/php-cgi | tools.jembot
| 1410 | 43-16:47:58 | 73.3 | /usr/bin/php-cgi | tools.jembot
| 1410 | 157-06:37:30 | 99.1 | /usr/bin/php-cgi | tools.supercount
| 1411 | 10-18:32:39 | 93.8 | /usr/bin/php-cgi | tools.dupdet
| 1411 | 92-23:08:14 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1411 | 92-23:08:18 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1411 | 92-23:08:35 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1411 | 92-23:17:04 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1412 | 54-14:33:15 | 52.6 | /usr/bin/php-cgi | tools.jembot
| 1412 | 54-14:36:12 | 52.6 | /usr/bin/php-cgi | tools.jembot
| 1412 | 54-14:36:59 | 52.6 | /usr/bin/php-cgi | tools.jembot
| 1412 | 54-14:40:08 | 52.6 | /usr/bin/php-cgi | tools.jembot
| 1412 | 89-09:05:38 | 74.6 | /usr/bin/php-cgi | tools.sowhy
| 1412 | 89-10:01:35 | 74.7 | /usr/bin/php-cgi | tools.sowhy
| 1413 | 35-06:21:19 | 97.3 | /usr/bin/php-cgi | tools.jembot
| 1413 | 35-06:28:46 | 97.3 | /usr/bin/php-cgi | tools.jembot
| 1413 | 71-13:12:44 | 97.5 | /usr/bin/php-cgi | tools.croptool
| 1413 | 73-18:54:20 | 98.3 | /usr/bin/php-cgi | tools.wsexport
| 1414 | 33-06:35:23 | 72.6 | /usr/bin/php-cgi | tools.jembot
| 1414 | 33-06:36:01 | 72.6 | /usr/bin/php-cgi | tools.jembot
| 1414 | 33-06:39:23 | 72.6 | /usr/bin/php-cgi | tools.jembot
| 1414 | 33-06:48:28 | 72.6 | /usr/bin/php-cgi | tools.jembot
| 1414 | 92-12:56:13 | 99.4 | /usr/bin/php-cgi | tools.jembot
| 1415 | 35-18:27:11 | 97.8 | /usr/bin/php-cgi | tools.jembot
| 1415 | 35-18:32:21 | 97.8 | /usr/bin/php-cgi | tools.jembot
| 1415 | 81-22:56:38 | 94.5 | /usr/bin/php-cgi | tools.dupdet
| 1415 | 109-01:52:29 | 99.1 | /usr/bin/php-cgi | tools.wsexport
| 1416 | 13-09:42:14 | 76.7 | /usr/bin/php-cgi | tools.wsexport
| 1416 | 37-13:49:31 | 89.9 | /usr/bin/php-cgi | tools.jembot
| 1416 | 37-13:58:05 | 89.9 | /usr/bin/php-cgi | tools.jembot
| 1416 | 83-14:53:33 | 95.1 | /usr/bin/php-cgi | tools.jembot
| 1416 | 83-14:54:22 | 95.1 | /usr/bin/php-cgi | tools.jembot
| 1417 | 2-12:47:18 | 45.0 | /usr/bin/php-cgi | tools.jembot
| 1417 | 2-13:00:44 | 45.2 | /usr/bin/php-cgi | tools.jembot
| 1417 | 2-13:23:16 | 45.5 | /usr/bin/php-cgi | tools.jembot
| 1417 | 2-13:42:51 | 45.7 | /usr/bin/php-cgi | tools.jembot
| 1417 | 34-14:41:33 | 93.6 | /usr/bin/php-cgi | tools.jembot
| 1417 | 34-14:44:43 | 93.6 | /usr/bin/php-cgi | tools.jembot
| 1417 | 77-00:25:16 | 95.5 | /usr/bin/php-cgi | tools.wsexport
| 1418 | 133-15:05:09 | 97.9 | /usr/bin/php-cgi | tools.jembot
| 1418 | 133-15:05:37 | 97.9 | /usr/bin/php-cgi | tools.jembot
| 1418 | 133-15:07:07 | 97.9 | /usr/bin/php-cgi | tools.jembot
| 1418 | 133-15:09:27 | 97.9 | /usr/bin/php-cgi | tools.jembot
| 1419 | 1-09:20:21 | 48.6 | /usr/bin/php-cgi | tools.jembot
| 1419 | 1-09:20:27 | 48.6 | /usr/bin/php-cgi | tools.jembot
| 1419 | 1-09:20:41 | 48.7 | /usr/bin/php-cgi | tools.jembot
| 1419 | 1-09:21:04 | 48.7 | /usr/bin/php-cgi | tools.jembot
| 1419 | 140-04:27:25 | 96.6 | /usr/bin/php-cgi | tools.jembot
| 1419 | 140-04:29:50 | 96.7 | /usr/bin/php-cgi | tools.jembot
| 1419 | 140-04:31:18 | 96.7 | /usr/bin/php-cgi | tools.jembot
| 1419 | 140-04:37:20 | 96.7 | /usr/bin/php-cgi | tools.jembot
| 1420 | 30-08:16:38 | 72.6 | /usr/bin/php-cgi | tools.jembot
| 1420 | 30-08:23:05 | 72.6 | /usr/bin/php-cgi | tools.jembot
| 1420 | 30-08:38:47 | 72.7 | /usr/bin/php-cgi | tools.jembot
| 1420 | 30-08:41:00 | 72.7 | /usr/bin/php-cgi | tools.jembot
| 1420 | 81-13:19:28 | 99.6 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:43:15 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:44:37 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:46:35 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:46:44 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:47:00 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:47:20 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:47:32 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1421 | 42-10:49:49 | 48.9 | /usr/bin/php-cgi | tools.jembot
| 1422 | 52-14:34:38 | 73.5 | /usr/bin/php-cgi | tools.jembot
| 1422 | 52-14:40:13 | 73.5 | /usr/bin/php-cgi | tools.jembot
| 1422 | 52-14:55:12 | 73.5 | /usr/bin/php-cgi | tools.jembot
| 1422 | 52-14:57:34 | 73.6 | /usr/bin/php-cgi | tools.jembot
| 1422 | 137-02:57:29 | 99.8 | /usr/bin/php-cgi | tools.jembot
| 1425 | 2-19:30:04 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1425 | 2-19:30:22 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1425 | 2-19:30:57 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1425 | 2-19:31:18 | 48.8 | /usr/bin/php-cgi | tools.jembot
| 1425 | 106-11:44:56 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1425 | 106-11:55:05 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1425 | 106-11:57:24 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1425 | 106-11:59:18 | 95.3 | /usr/bin/php-cgi | tools.jembot
| 1426 | 26-11:08:42 | 49.2 | /usr/bin/php-cgi | tools.jembot
| 1426 | 26-11:08:51 | 49.2 | /usr/bin/php-cgi | tools.jembot
| 1426 | 26-11:09:03 | 49.2 | /usr/bin/php-cgi | tools.jembot
| 1426 | 26-11:11:33 | 49.2 | /usr/bin/php-cgi | tools.jembot
| 1426 | 124-20:24:19 | 99.2 | /usr/bin/php-cgi | tools.jembot
| 1426 | 124-20:28:51 | 99.2 | /usr/bin/php-cgi | tools.jembot
| 1427 | 132-22:04:21 | 97.4 | /usr/bin/php-cgi | tools.jembot
| 1427 | 132-22:05:57 | 97.4 | /usr/bin/php-cgi | tools.jembot
| 1427 | 132-22:06:18 | 97.4 | /usr/bin/php-cgi | tools.jembot
| 1427 | 132-22:13:33 | 97.4 | /usr/bin/php-cgi | tools.jembot
| 1428 | 7-09:35:19 | 75.5 | /usr/bin/php-cgi | tools.iabot
| 1428 | 52-05:51:51 | 95.4 | /usr/bin/php-cgi | tools.jembot
| 1428 | 52-05:58:25 | 95.4 | /usr/bin/php-cgi | tools.jembot
| 1428 | 105-22:16:38 | 97.3 | /usr/bin/php-cgi | tools.wsexport