Page MenuHomePhabricator

prometheus-pdns-exporter log noise about unexpected metrics
Open, Needs TriagePublic

Description

Seen in the syslog output on cloudservices1003:

Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-answers'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-changes'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-queries'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-refused'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'rd-queries'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'recursion-unanswered'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'security-status'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'udp-answers-bytes'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'udp-do-queries'

Event Timeline

bd808 created this task.Jul 7 2019, 4:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 7 2019, 4:31 PM

Change 521042 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/debs/prometheus-pdns-exporter@master] Add dnsupdate, rd, recursion, security, and udp metrics

https://gerrit.wikimedia.org/r/521042

Change 521042 merged by Muehlenhoff:
[operations/debs/prometheus-pdns-exporter@master] Add dnsupdate, rd, recursion, security, and udp metrics

https://gerrit.wikimedia.org/r/521042

Mentioned in SAL (#wikimedia-operations) [2019-07-08T19:23:37Z] <moritzm> uploaded prometheus-pdns-exporter 0.4.1 to stretch-wikimedia T227411

New and improved "unknown metric" spam discovered in /var/log/syslog on cloudservices2002-dev.wikimedia.org after updating to prometheus-pdns-exporter 0.4.1. This seems to be a result of a much newer version of pdns-server being installed there (4.0.3-1+deb9u4) compared to the version on cloudservices1003.wikimedia.org (3.4.1-4+deb8u8):

Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'incoming-notifications'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'overload-drops'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'signatures'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp4-answers'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp4-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp4-queries'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp6-answers'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp6-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp6-queries'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp4-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp6-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'fd-usage'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'key-cache-size'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'meta-cache-size'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'real-memory-usage'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'signature-cache-size'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'sys-msec'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-in-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-noport-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-recvbuf-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-sndbuf-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'uptime'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'user-msec'

Example sudo pdns_control list output from the newer version of pdns-server:

$ sudo pdns_control list
corrupt-packets=3,deferred-cache-inserts=0,deferred-cache-lookup=0,dnsupdate-answers=0,dnsupdate-changes=0,dnsupdate-queries=0,dnsupdate-refused=0,incoming-notifications=0,overload-drops=0,packetcache-hit=0,packetcache-miss=51,packetcache-size=57,query-cache-hit=0,query-cache-miss=173,rd-queries=96,recursing-answers=0,recursing-questions=0,recursion-unanswered=0,security-status=0,servfail-packets=0,signatures=0,tcp-answers=6,tcp-answers-bytes=382,tcp-queries=9,tcp4-answers=6,tcp4-answers-bytes=382,tcp4-queries=9,tcp6-answers=0,tcp6-answers-bytes=0,tcp6-queries=0,timedout-packets=0,udp-answers=65,udp-answers-bytes=2918,udp-do-queries=0,udp-queries=65,udp4-answers=61,udp4-answers-bytes=2744,udp4-queries=61,udp6-answers=4,udp6-answers-bytes=174,udp6-queries=4,fd-usage=110,key-cache-size=0,latency=132,meta-cache-size=0,qsize-q=0,real-memory-usage=6258688,signature-cache-size=0,sys-msec=21512,udp-in-errors=0,udp-noport-errors=1,udp-recvbuf-errors=0,udp-sndbuf-errors=0,uptime=450802,user-msec=4240,

Mentioned in SAL (#wikimedia-operations) [2019-07-08T20:53:39Z] <bd808> Upgraded prometheus-pdns-exporter to 0.4.1 on cloudservices1003.wikimedia.org (T227411)

Mentioned in SAL (#wikimedia-operations) [2019-07-08T20:57:07Z] <bd808> Upgraded prometheus-pdns-exporter to 0.4.1 on cloudservices1004.wikimedia.org (T227411)