Page MenuHomePhabricator

Handle unknown stats in rsyslog_exporter
Closed, ResolvedPublic

Description

This shows up in the logs on central syslog hosts:

Nov 22 10:49:31 wezen rsyslog_exporter[21742]: 2018/11/22 10:49:31 error handling stats line: unknown pstat type: 0, line was: 2018-11-22T10:49:31.413068+00:00 wezen rsyslogd-pstats: { "name": "imudp(w0)", "origin": "imudp", "called.recvmmsg": 158, "called.recvmsg": 0, "msgs.received": 81 }

See also https://github.com/filippog/rsyslog_exporter/issues/1

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 22 2018, 10:49 AM
fgiunchedi moved this task from Inbox to Up next on the observability board.Feb 8 2019, 11:38 AM
jijiki added a subscriber: jijiki.

This is polluting our messages on mw* servers as well:

Oct 20 06:20:19 mw1333 rsyslog_exporter[44310]: 2019/10/20 06:20:19 error handling stats line: unknown pstat type: 0, line was: 2019-10-20T06:20:19.674293+00:00 mw1333 rsyslogd-pstats: { "name": "imudp(w0)
", "origin": "imudp", "called.recvmmsg": 5443193, "called.recvmsg": 0, "msgs.received": 2724803 }
jijiki moved this task from Incoming 🐫 to Unsorted on the serviceops board.Aug 17 2020, 11:47 PM
lmata moved this task from Up next to Backlog on the observability board.Sep 21 2020, 8:27 PM
Volans added a subscriber: Volans.Mon, Oct 5, 5:43 PM

Any news on this by any chance? On this random host I was checking today it taking up 1/3rd of syslog.

 netbox-dev2001  0 ~$ grep -c rsyslog_exporter /var/log/syslog
6365
 netbox-dev2001  0 ~$ wc -l /var/log/syslog
18953 /var/log/syslog

Mentioned in SAL (#wikimedia-operations) [2020-10-08T17:16:32Z] <shdubsh> install prometheus-rsyslog-exporter_0.0.0+git20201008 on centrallog1001 - T210137

colewhite claimed this task.Thu, Oct 8, 5:17 PM
colewhite triaged this task as Low priority.

Found a new upstream and have deployed it to netbox-dev2001 and centrallog1001 to run for a few days. If all checks out, we'll roll it to the rest of the fleet.

colewhite moved this task from Backlog to In progress on the observability board.Thu, Oct 8, 5:19 PM

Change 634112 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] prometheus: ensure new prometheus-rsyslog-exporter version

https://gerrit.wikimedia.org/r/634112

Change 634112 merged by Cwhite:
[operations/puppet@production] prometheus: ensure new prometheus-rsyslog-exporter version

https://gerrit.wikimedia.org/r/634112

colewhite closed this task as Resolved.Thu, Oct 15, 8:46 PM

Updated prometheus-rsyslog-exporter deployed to the fleet. If the log message comes up again please let us know.