Despite configuration it looks like the new version of diamond logs as DEBUG level with --log-stdout --foreground, spamming syslog
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| diamond: ship systemd override file | operations/puppet | production | +20 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T97635 Update diamond to latest upstream version | |||
| Resolved | fgiunchedi | T171580 Diamond log level set to DEBUG spams syslog |
Event Timeline
Comment Actions
Migrating my comment from T171638, as it may be useful for debugging:
I believe it is on stretch, because I have not seen it on other hosts, but it could be because a newer package or other things:
Jun 22 10:10:46 db2072 python[20568]: 1498126246.73 [CPUCollector:21046:DEBUG] Collection took 3 ms Jun 22 10:10:50 db2072 python[20568]: 1498126250.08 [DiskUsageCollector:20948:DEBUG] Collection took 6 ms Jun 22 10:10:51 db2072 python[20568]: 1498126251.08 [NetworkCollector:20869:DEBUG] Collection took 14 msWhile it is not a huge problem, it puts a lot of garbage on the syslog (making more difficult to debug, and probably not useful at all unless they are errors). Specially because I saw it in some cases (servers) complain about special devices not being able to show available disk space (so that is 20 lines every minute):
Jul 12 11:17:15 dbstore2002 python[777]: 1499858235.21 [NetworkCollector:1124:DEBUG] Collection took 19 ms Jul 12 11:17:15 dbstore2002 python[777]: 1499858235.21 [NetworkCollector:1124:DEBUG] Collection took 19 ms Jul 12 11:17:15 dbstore2002 python[777]: 1499858235.21 [NetworkCollector:1124:DEBUG] Collection took 19 ms Jul 12 11:17:16 dbstore2002 python[777]: 1499858236.15 [CPUCollector:1387:DEBUG] Collection took 3 ms Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys since it is of typ Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /proc since it is of ty Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /dev since it is of typ Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /dev/pts since it is of Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /run since it is of typ Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/kernel/security si Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.85 [DiskSpaceCollector:1448:DEBUG] Ignoring /dev/shm since it is of Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /run/lock since it is o Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup since it Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/systemd Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/pstore since it Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/perf_eve Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/cpu,cpua Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/memory s Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/freezer Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/net_cls, Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/pids sin Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/devices Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/cpuset s Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/fs/cgroup/blkio si Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /sys/kernel/debug since Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /dev/hugepages since it Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /proc/sys/fs/binfmt_mis Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /dev/mqueue since it is Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Ignoring /proc/sys/fs/binfmt_mis Jul 12 11:17:17 dbstore2002 python[777]: 1499858237.86 [DiskSpaceCollector:1448:DEBUG] Collection took 7 ms Jul 12 11:17:18 dbstore2002 python[777]: 1499858238.99 [LoadAverageCollector:1420:DEBUG] Collection took 3 ms Jul 12 11:17:22 dbstore2002 python[777]: 1499858242.74 [DiskUsageCollector:1451:DEBUG] Collection took 9 ms Jul 12 11:17:27 dbstore2002 prometheus-mysqld-exporter[790]: time="2017-07-12T11:17:27Z" level=error msg="Error pinging mysqld: dial Jul 12 11:17:33 dbstore2002 python[777]: 1499858253.52 [TCPCollector:1476:DEBUG] Collection took 18 msThe collectors on stretch already had to be tuned before, so my thesis would be more config that is different from jessie: https://gerrit.wikimedia.org/r/361878
Comment Actions
Change 368177 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] diamond: ship systemd override file
Comment Actions
Change 368177 merged by Filippo Giunchedi:
[operations/puppet@production] diamond: ship systemd override file