Page MenuHomePhabricator

newer version of nagios-nrpe-plugin nrpe (check_nrpe) with fixed logging issue on stretch icinga
Closed, ResolvedPublic

Description

As heavy users of check_nrpe from nagios-nrpe-plugin we are suffering from this bug:

"lots of check_nrpe entries logged on server. How to turn off"
https://support.nagios.com/forum/viewtopic.php?f=7&t=41115

<shdubsh> check_nrpe is 53% of logs

In that ticket the fix is to edit " lines 171 - 172 in nrpe-3.0.1/src/check_nrpe.c.".

And also:

<shdubsh> They fixed it in 3.2 https://github.com/NagiosEnterprises/nrpe/commit/b8ee0362614a1ca49d8269165ba2140f2c903dd0#diff-57ad8ac241e8ef08e403026d56a74927

and:

<shdubsh> 3.2.1 is in stable backports

<shdubsh> gah, what are the chances of getting a newer version in debian?

<mutante> let's see who the maintainer is..
<mutante> Debian Nagios Maintainer Group (QA Page, Mail Archive)
<mutante> Bas Couwenberg (QA Page)

have: https://packages.debian.org/stretch/nagios-nrpe-plugin

want: https://packages.debian.org/stretch-backports/nagios-nrpe-plugin
@Muehlenhoff @fgiunchedi Recommendations how to handle it it best? Get package from stable-backports on apt.wm.org? Ask the upstream maintainer? Other options?

Event Timeline

Dzahn renamed this task from newer version of nrpe (check_nrpe) with fixed logging issue on stretch icinga to newer version of nagios-nrpe-plugin nrpe (check_nrpe) with fixed logging issue on stretch icinga.Oct 23 2018, 6:34 PM
Dzahn updated the task description. (Show Details)

Change 469337 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] icinga: tell rsyslog to discard logs from check_nrpe

https://gerrit.wikimedia.org/r/469337

We enable backports by default nowadays, so pinning said package in puppet to backports will DTRT I think.

We enable backports by default nowadays, so pinning said package in puppet to backports will DTRT I think.

The risk being of course that a later version upgrade will bring an even newer version, safest is to import the package internally.

We enable backports by default nowadays, so pinning said package in puppet to backports will DTRT I think.

The risk being of course that a later version upgrade will bring an even newer version, safest is to import the package internally.

Yeah. We also have that with e.g. the app servers; Lilypond missed stretch and we're now using it from stretch-backports, but every 1-2 months a new version arrives there, while we don't need/use any of the new features.

But the even better solution would be to backport the https://github.com/NagiosEnterprises/nrpe/commit/b8ee0362614a1ca49d8269165ba2140f2c903dd0#diff-57ad8ac241e8ef08e403026d56a74927 fix to the stretch version and open a bug against nagios-nrpe with the patch?

The maintainer already made a stretch update in the past (for https://packages.qa.debian.org/n/nagios-nrpe/news/20170715T221717Z.html) and if that's a bug which affects large Icinga installations on stretch, it would be best to fix it in Stretch (and totally matches the criteria for bugfixes in stable updates).

Good idea! I think upstream fe006d2 and 08425ff are the fixes for this particular issue and they seem to apply cleanly on top of 3.0.1 with 1 file changed, 5 insertions(+), 3 deletions(-), so trivial enough.

b8ee036 is a bigger/wider fix that is probably not stretch material I'd say.

Good idea! I think upstream fe006d2 and 08425ff are the fixes for this particular issue and they seem to apply cleanly on top of 3.0.1 with 1 file changed, 5 insertions(+), 3 deletions(-), so trivial enough.

Ack, these two look good.

Daniel/Cole: I think procedure-wise it would be best to first build a fixed package internally and if we can confirm that it works fine and fixes the issue, submit it to the Debian BTS. Let me know if you have any questions wrt patch application etc.

Mentioned in SAL (#wikimedia-operations) [2018-10-25T16:24:49Z] <shdubsh> installed patched nagios-nrpe-plugin and nagios-nrpe-server on icinga1001 - T207775

The patched source package has been imported into apt.wikimedia.org

[install1002:~] $ sudo -i reprepro ls nagios-nrpe
nagios-nrpe | 3.0.1-3+deb9u1+wmf1 | stretch-wikimedia | source

But the binary packages coming from it, -plugin and -server are not there:

[install1002:~] $ sudo -i reprepro ls nagios-nrpe-plugin
[install1002:~] $

@Muehlenhoff We are looking for the missing step there ^

Change 469337 abandoned by Dzahn:
icinga: on stretch, tell rsyslog to discard logs from check_nrpe

https://gerrit.wikimedia.org/r/469337

It has been fixed by @colewhite and by adding the binaries like so:

17:05 < moritzm> reprepro -C main includedeb stretch-wikimedia $NAME-OF-NAGIOS-NRPE-PLUGIN-DEB
17:05 < moritzm> reprepro -C main includedeb stretch-wikimedia $NAME-OF-NAGIOS-NRPE-SERVER-DEB

Mostly resolved but we might want to keep it open for the "suggest to upstream maintainer" part too.

Dzahn triaged this task as Medium priority.Oct 25 2018, 11:53 PM

If we can confirm that this works fine on icinga1001, we can report it to Debian by using the reportbug tool:
Running "reportbug nagios-nrpe-server" will ask for the necessary information. When asked for the version for which this bug applies to use "3.0.1-3+deb9u1" to indicate that this applies to stable. When asked for the severity "important" sounds like an adequate choice, as the logs in big installations become quite spammy. Reportbug will spawn an editor with the problem description, make sure to be as verbose as possible here and suggest that this could be fixed in a stable point release. When you save the editor window, reportbug will show a line like "Submit this report to (..) with some selections. There you can use "a" to attach the two patches which we applied to our internal installation.

We have deployed the patches internally and the fix appears to work. Unfortunately upstream does not want the patches.

Marking resolved.

While nagios-nrpe-plugin has been upgraded on the Icinga hosts, the nagios-nrpe source package also builds the nagios-nrpe-server binary, which should also be upgraded for consistency: https://debmonitor.wikimedia.org/packages/nagios-nrpe-server