Page MenuHomePhabricator
Create Task
Maniphest T227411

prometheus-pdns-exporter log noise about unexpected metrics
Closed, ResolvedPublic
Actions

Description

Seen in the syslog output on cloudservices1003:

Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-answers'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-changes'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-queries'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'dnsupdate-refused'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'rd-queries'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'recursion-unanswered'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'security-status'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'udp-answers-bytes'
Jul  7 06:24:35 cloudservices1003 prometheus-pdns-exporter[774]: WARNING:__main__:Unknown metric 'udp-do-queries'

Details

SubjectRepoBranchLines +/-
Bump changelog for pdns4 supportoperations/debs/prometheus-pdns-exportermaster+6 -0
Add support for PDNS 4operations/debs/prometheus-pdns-exportermaster+24 -0
Add dnsupdate, rd, recursion, security, and udp metricsoperations/debs/prometheus-pdns-exportermaster+9 -0
Customize query in gerrit

Related Objects

Event Timeline

bd808 created this task.Jul 7 2019, 4:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 7 2019, 4:31 PM
gerritbot added a comment.Jul 7 2019, 5:15 PM

Change 521042 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[operations/debs/prometheus-pdns-exporter@master] Add dnsupdate, rd, recursion, security, and udp metrics

https://gerrit.wikimedia.org/r/521042

gerritbot added a project: Patch-For-Review.Jul 7 2019, 5:15 PM
Malyacko subscribed.Jul 7 2019, 7:08 PM
gerritbot added a comment.Jul 8 2019, 6:18 PM

Change 521042 merged by Muehlenhoff:
[operations/debs/prometheus-pdns-exporter@master] Add dnsupdate, rd, recursion, security, and udp metrics

https://gerrit.wikimedia.org/r/521042

Maintenance_bot removed a project: Patch-For-Review.Jul 8 2019, 7:10 PM
Stashbot added a comment.Jul 8 2019, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-08T19:23:37Z] <moritzm> uploaded prometheus-pdns-exporter 0.4.1 to stretch-wikimedia T227411

bd808 added a comment.Jul 8 2019, 8:32 PM

New and improved "unknown metric" spam discovered in /var/log/syslog on cloudservices2002-dev.wikimedia.org after updating to prometheus-pdns-exporter 0.4.1. This seems to be a result of a much newer version of pdns-server being installed there (4.0.3-1+deb9u4) compared to the version on cloudservices1003.wikimedia.org (3.4.1-4+deb8u8):

Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'incoming-notifications'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'overload-drops'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'signatures'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp4-answers'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp4-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp4-queries'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp6-answers'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp6-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'tcp6-queries'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp4-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp6-answers-bytes'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'fd-usage'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'key-cache-size'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'meta-cache-size'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'real-memory-usage'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'signature-cache-size'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'sys-msec'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-in-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-noport-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-recvbuf-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'udp-sndbuf-errors'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'uptime'
Jul  8 20:13:33 cloudservices2002-dev prometheus-pdns-exporter[3911]: WARNING:__main__:Unknown metric 'user-msec'

Example sudo pdns_control list output from the newer version of pdns-server:

$ sudo pdns_control list
corrupt-packets=3,deferred-cache-inserts=0,deferred-cache-lookup=0,dnsupdate-answers=0,dnsupdate-changes=0,dnsupdate-queries=0,dnsupdate-refused=0,incoming-notifications=0,overload-drops=0,packetcache-hit=0,packetcache-miss=51,packetcache-size=57,query-cache-hit=0,query-cache-miss=173,rd-queries=96,recursing-answers=0,recursing-questions=0,recursion-unanswered=0,security-status=0,servfail-packets=0,signatures=0,tcp-answers=6,tcp-answers-bytes=382,tcp-queries=9,tcp4-answers=6,tcp4-answers-bytes=382,tcp4-queries=9,tcp6-answers=0,tcp6-answers-bytes=0,tcp6-queries=0,timedout-packets=0,udp-answers=65,udp-answers-bytes=2918,udp-do-queries=0,udp-queries=65,udp4-answers=61,udp4-answers-bytes=2744,udp4-queries=61,udp6-answers=4,udp6-answers-bytes=174,udp6-queries=4,fd-usage=110,key-cache-size=0,latency=132,meta-cache-size=0,qsize-q=0,real-memory-usage=6258688,signature-cache-size=0,sys-msec=21512,udp-in-errors=0,udp-noport-errors=1,udp-recvbuf-errors=0,udp-sndbuf-errors=0,uptime=450802,user-msec=4240,
Stashbot added a comment.Jul 8 2019, 8:53 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-08T20:53:39Z] <bd808> Upgraded prometheus-pdns-exporter to 0.4.1 on cloudservices1003.wikimedia.org (T227411)

Stashbot added a comment.Jul 8 2019, 8:57 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-08T20:57:07Z] <bd808> Upgraded prometheus-pdns-exporter to 0.4.1 on cloudservices1004.wikimedia.org (T227411)

Andrew merged a task: T238859: prometheus-pdns-exporter complaining about many unknown metrics.Nov 21 2019, 8:11 PM
Andrew subscribed.
Andrew added a subscriber: MoritzMuehlenhoff.Nov 21 2019, 8:15 PM

I upgraded pdns to version 4 yesterday and now there's a lot more of this. I don't see the metrics being complained about defined in prometheus-pdns-exporter so I'm not sure how to address this -- @MoritzMuehlenhoff if you want to point me in the right direction I'm happy to do the coding.

MoritzMuehlenhoff added a project: SRE.Nov 22 2019, 8:03 AM
MoritzMuehlenhoff added a comment.EditedNov 22 2019, 8:19 AM

The metrics below were added in PDNS 4. which were not yet present in 3. They need to be added to the metrics dictionary of the exporter in operations/debs/prometheus-pdns-exporter
You can find docs on the specific metrics at https://doc.powerdns.com/authoritative/performance.html (from a quick glance they should all be of GaugeMetricFamily)

'incoming-notifications'
'overload-drops'
'signatures'
'tcp-answers-bytes'
'tcp4-answers'
'tcp4-answers-bytes'
'tcp4-queries'
'tcp6-answers'
'tcp6-answers-bytes'
'tcp6-queries'
'udp4-answers-bytes'
'udp6-answers-bytes'
'fd-usage'
'key-cache-size'
'meta-cache-size'
'real-memory-usage'
'signature-cache-size'
'sys-msec'
'udp-in-errors'
'udp-noport-errors'
'udp-recvbuf-errors'
'udp-sndbuf-errors'
'uptime'
'user-msec'
gerritbot added a comment.Nov 22 2019, 8:33 AM

Change 552467 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/debs/prometheus-pdns-exporter@master] Add support for PDNS 4

https://gerrit.wikimedia.org/r/552467

gerritbot added a project: Patch-For-Review.Nov 22 2019, 8:33 AM
MoritzMuehlenhoff added a comment.Nov 22 2019, 8:35 AM

@Andrew : I created an (untested) patch which should fix this, can you take it from here?

Andrew added a comment.Nov 22 2019, 3:00 PM
In T227411#5683732, @MoritzMuehlenhoff wrote:

@Andrew : I created an (untested) patch which should fix this, can you take it from here?

yes, thank you!

Andrew added a comment.Nov 22 2019, 3:39 PM

That patch seems to quiet the alerts; I'll see about building and deploying

gerritbot added a comment.Nov 22 2019, 3:40 PM

Change 552467 merged by Andrew Bogott:
[operations/debs/prometheus-pdns-exporter@master] Add support for PDNS 4

https://gerrit.wikimedia.org/r/552467

gerritbot added a comment.Nov 22 2019, 3:43 PM

Change 552531 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/debs/prometheus-pdns-exporter@master] Bump changelog for pdns4 support

https://gerrit.wikimedia.org/r/552531

gerritbot added a comment.Nov 22 2019, 3:45 PM

Change 552531 merged by Andrew Bogott:
[operations/debs/prometheus-pdns-exporter@master] Bump changelog for pdns4 support

https://gerrit.wikimedia.org/r/552531

Andrew closed this task as Resolved.Nov 22 2019, 4:03 PM
Andrew claimed this task.

done -- logs are nice and quiet now.

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2019, 4:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL