Icinga has httpauth on (not accessible for public)
Open, NormalPublic

Description

Just noticed that Icinga apparently has httpauth on, so it's not accessible for me (the public).
Is there a security issue or other reason that forced this?

also http://status.wikimedia.org/ reports it as down for several days.


Version: wmf-deployment
Severity: normal
URL: https://icinga.wikimedia.org/icinga
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=54713
https://rt.wikimedia.org/Ticket/Display.html?id=6838

Details

Reference
bz60112
bzimport raised the priority of this task from to Normal.
bzimport set Reference to bz60112.
bzimport added a subscriber: Unknown Object (MLST).
Se4598 created this task.Jan 15 2014, 11:27 PM

Logging in works for me with my a Labs / wikitech.wikimedia.org account, but that might just be because I'm in a specific LDAP group, like bug 54713.

(In reply to comment #1)
yep, logging in with wikitech-acc doesn't work for me.

Basically all I expect as answer here is a information why it currently on and when it is expected to be disabled again.
(icinga is on neon and this has nothing to do with graphite's apparently pending security review, right? bug 54713#c5)

scfc added a comment.Jan 23 2014, 4:05 AM

RobH said in #wikimedia-operations that "there are security issues with icinga iirc".

Ok. So we used to have Nagios which anyone could have a look at to see what's wrong. Someone decided to switch to another tool (Icinga). Now it turns out that that tool has security issues and public access got disabled? Way to go.....

It's been so since December. Originally I understood it was a matter of days...

2013-12-20 12.31 < whym> icinga.wikimedia.org now requirs authorization from me. Is this how it's intended to be?
2013-12-20 12.39 < paravoid> whym: there are a couple of security vulnerabilities for icinga in the wild, so we've temporarily locked public access

https://gerrit.wikimedia.org/r/#/c/100989/

(In reply to comment #4)

Ok. So we used to have Nagios which anyone could have a look at to see what's
wrong. Someone decided to switch to another tool (Icinga). Now it turns out
that that tool has security issues and public access got disabled? Way to
go.....

IIRC nagois had security issues as well.

Filed for ops as RT #6838

Yes, there are security issues with Icinga that forced us to lock it down temporarily back in December 12th.

These are CVE-2013-7106, CVE-2013-7107 & CVE-2013-7108. They are still unfixed in Ubuntu precise (LTS); Icinga is in the universe section, so the Ubuntu security team deals with them on a "best effort" basis (i.e. they might not even update it, at all).

The vulnerability status per Ubuntu distribution can be tracked at:
http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-7106.html
http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-7107.html
http://people.canonical.com/~ubuntu-security/cve/2013/CVE-2013-7108.html
respectively. Note how they decided to ignore the first one (a CSRF), which shows IMHO a poor judgement from their part.

I don't think we can take the time to do a major Icinga version upgrade right now, nor to backport the fixes ourselves. Our current strategy is "wait for Ubuntu", but if anyone wants to help the backporting process (and optionally engage with the Ubuntu security team so others can benefit from that) that'd be awesome.

scfc added a comment.Feb 14 2014, 10:06 AM

Created attachment 14586
Backport fix for CVE-2013-7106.

Attached:

scfc added a comment.Feb 14 2014, 10:07 AM

Created attachment 14587
Backport fix for CVE-2013-7108.

Attached:

scfc added a comment.Feb 14 2014, 11:09 AM

Created attachment 14588
Backport fix for CVE-2013-7107.

Attached:

Hey, that's good stuff! Thanks! Would you mind terribly contacting the Ubuntu security team to offer these code backports? Their usual response is "you're on your own", but if you attach code they might treat it differently, who knows :)

scfc added a comment.Feb 14 2014, 12:50 PM

No, I don't mind, but I need to test it first at least once :-). I've asked petan for access to the Nagios project on Labs, will set up a new instance there and see if the package I baked works.

(Ceterum censeo Debian packaging esse delendam. I simply love Fedora (and other RPM distros) for its cleanliness; on Debian I'm never sure what patches and files end up in the (source) package.)

Hey Tim, have you contacted the Ubuntu security team? Anything we can do to help?

scfc added a comment.Apr 17 2014, 12:38 PM

*argl* Forgot to test it; now I see the bugs have expired. I'll test it Real Soon Now(TM) and get back to you if there's anything unsurmountable.

ricky.wikitech wrote:

Been a few months - any update here? Or anything I (as a community member) can do to help with moving this along? :)

this could get fixed also during the debian migration, I believe icinga jessie packages would do

scfc added a comment.Mar 7 2015, 9:06 PM

IIRC, it could also be fixed by upgrading neon (?) to Trusty, and as those upgrades always seem imminent and I think they have more value than shepherding those patches back to Ubuntu Precise, I didn't spend any time on the latter as testing them for upstream is too complex in my setup.

IIRC, it could also be fixed by upgrading neon (?) to Trusty

Yep, neon is the icinga host.

Krenair added a subtask: Restricted Task.May 30 2015, 1:59 PM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 3 2015, 9:47 AM
Sitic added a subscriber: Sitic.Jul 7 2015, 11:00 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 26 2015, 9:30 AM
akosiaris closed subtask Restricted Task as Resolved.Nov 2 2016, 1:33 PM

Although we do have a security-support version of Icinga running now, I'm against opening up the access to the public. Icinga/Nagios have a non-ideal security history, the CGIs are written in C and the monitoring hosts have privileged access to our production networks. (I'm fine with group-based access for selected, trusted service owners).

Although we do have a security-support version of Icinga running now, I'm against opening up the access to the public. Icinga/Nagios have a non-ideal security history, the CGIs are written in C and the monitoring hosts have privileged access to our production networks. (I'm fine with group-based access for selected, trusted service owners).

What about putting a reverse proxy in front of it with some strict filtering so that only a small part is viewable without logging in? You could just call that one https://icinga-public.wikimedia.org/

Did something changed here in over two years since icinga is login-only?

Did something changed here in over two years since icinga is login-only?

No, and given @MoritzMuehlenhoff 's answer above I don't think there will be any soon. . I know @Multichill's idea sounds like a good compromise but unfortunately it is not very feasible technically. Most of the icinga interface is practically the output of 1 CGI script (status.cgi) which alters it's behaviour based on the HTTP GET params it receives. That makes that approach technically difficult and most importantly quite error prone. It would also probably mean that the end result would not differ much from https://status.wikimedia.org/ (which already exist and provides status functionality) making that approach kind of a moot point.

My current inclination is, unfortunately, to close this task as declined