Page MenuHomePhabricator
Create Task
Maniphest T114059

ssl expiry tracking in icinga - we don't monitor that many domains
Closed, ResolvedPublic
Actions

Description

With the unplanned expiration of a couple of SSL certificates, T112542 was generated. Since then, we've listed all of the domains we purchased certificates from.

Now we need to decide if we are going to put in icinga checks for all of them, or just some, and how to differentiate.

In T112542#1640176, @BBlack wrote:

We only have that icinga check on the primary unified cert, which covers the production endpoints for:

  • wikipedia.org
  • mediawiki.org
  • wikibooks.org
  • wikidata.org
  • wikimediafoundation.org
  • wikimedia.org
  • wikinews.org
  • wikiquote.org
  • wikisource.org
  • wikiversity.org
  • wikivoyage.org
  • wiktionary.org

... and all of their mobile subdomains and whatnot. It's a pretty verbose check, validates functional SSL for all of the SAN domains, checks the cert expiry, etc.

But we don't have any kind of checking in place for the various other misc certs we own that are deployed for smaller or one-off services, or deployed to third parties (or in some cases, rare today but important later - not deployed at all but still critical). Just looking at puppet's files/ssl/ today, that list is something like:

archiva.wikimedia.org.crt
blog.wikimedia.org.crt
dumps.wikimedia.org.crt
ecc-star.wmfusercontent.org.crt
eventdonations.wikimedia.org.crt
ganglia.wikimedia.org.crt
gerrit.wikimedia.org.crt
icinga.wikimedia.org.crt
labvirt-star.eqiad.wmnet.crt
ldap-codfw.wikimedia.org.crt
ldap-eqiad.wikimedia.org.crt
ldap-mirror.wikimedia.org.crt
librenms.wikimedia.org.crt
lists.wikimedia.org.crt
policy.wikimedia.org.crt
rt.wikimedia.org.crt
star.planet.wikimedia.org.crt
star.wmflabs.org.crt
star.wmfusercontent.org.crt
stream.wikimedia.org.crt
tendril.wikimedia.org.crt
ticket.wikimedia.org.crt
toolserver.org.crt
virt-star.eqiad.wmnet.crt
wikitech.wikimedia.org.crt

Of those, I can see in our icinga config direct expiry checks only for:

lists.wikimedia.org
ticket.wikimedia.org
ldap-codfw.wikimedia.org
ldap-eqiad.wikimedia.org

Additionally: https://docs.google.com/a/wikimedia.org/spreadsheets/d/1yT5rvoEEUHhNeJAQRVamr8ECqN3TLsMaO8N_At4Ki3I/edit?usp=sharing lists all the certificates and expiry info.

We need to determine which of these will get icinga checks.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+26 -0icinga: ssl cert monitoring for external services
operations/puppetproduction+4 -0planet: add ssl cert expiry check
operations/puppetproduction+5 -0dumps: add cert expiry check
operations/puppetproduction+5 -0icinga: add cert expiry check for icinga itself
operations/puppetproduction+5 -0wikitech: add SSL cert expiry monitoring
operations/puppetproduction+5 -0gerrit: add cert expiry check
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Sep 28 2015, 11:22 PM
RobH raised the priority of this task from to Needs Triage.
RobH updated the task description. (Show Details)
RobH added a project: acl*sre-team.
RobH added subscribers: RobH, BBlack.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptSep 28 2015, 11:22 PM
Dzahn added a subscriber: Dzahn.EditedOct 8 2015, 10:47 PM

The following are covered already:

  • lists
  • otrs

because they use the check_ssl_http check_command which is defined in modules/nagios_common/files/check_commands/check_ssl.cfg and set to warn at 60 and go CRIT at 30 days before expiry, it uses check_ssl, written by Faidon, which has a sub ssl_expiry_check.

i'm going to upload a change or more to add checks for other services using the same method, in a monitoring::service in the role class.

gerritbot added a subscriber: gerritbot.Oct 8 2015, 11:00 PM

Change 244610 had a related patch set uploaded (by Dzahn):
wikitech: add SSL cert expiry monitoring

https://gerrit.wikimedia.org/r/244610

gerritbot added a project: Patch-For-Review.Oct 8 2015, 11:00 PM
gerritbot added a comment.Oct 8 2015, 11:05 PM

Change 244614 had a related patch set uploaded (by Dzahn):
icinga: add ssl cert expiry for icinga itself

https://gerrit.wikimedia.org/r/244614

gerritbot added a comment.Oct 8 2015, 11:08 PM

Change 244617 had a related patch set uploaded (by Dzahn):
dumps: add cert expiry check

https://gerrit.wikimedia.org/r/244617

gerritbot added a comment.Oct 8 2015, 11:10 PM

Change 244618 had a related patch set uploaded (by Dzahn):
gerrit: add cert expiry check

https://gerrit.wikimedia.org/r/244618

Dzahn triaged this task as High priority.Oct 19 2015, 11:22 PM
gerritbot added a comment.Oct 20 2015, 12:02 AM

Change 244618 merged by Dzahn:
gerrit: add cert expiry check

https://gerrit.wikimedia.org/r/244618

Dzahn added a comment.EditedOct 20 2015, 12:33 AM

check for gerrit cert added:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=ytterbium&service=HTTPS

"SSL OK - Certificate gerrit.wikimedia.org valid until 2018-05-25 21:30:06 +0000 (expires in 948 days)"

Dzahn claimed this task.Oct 20 2015, 12:34 AM
gerritbot added a comment.Oct 20 2015, 2:00 PM

Change 244610 merged by Faidon Liambotis:
wikitech: add SSL cert expiry monitoring

https://gerrit.wikimedia.org/r/244610

Dzahn added a comment.Oct 20 2015, 2:42 PM

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=silver&service=HTTPS

SSL OK - Certificate wikitech.wikimedia.org valid until 2016-01-25 04:44:13 +0000 (expires in 96 days)

gerritbot added a comment.Oct 20 2015, 6:04 PM

Change 244614 merged by Dzahn:
icinga: add cert expiry check for icinga itself

https://gerrit.wikimedia.org/r/244614

Dzahn added a comment.Oct 20 2015, 7:28 PM

and let's also have meta monitoring. icinga itself should have a working cert :)

added:

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=neon&service=HTTPS

SSL OK - Certificate icinga.wikimedia.org valid until 2016-01-08 03:00:46 +0000 (expires in 79 days)

Dzahn added a comment.Oct 20 2015, 8:28 PM

progress tracking on etherpad now:

https://etherpad.wikimedia.org/p/T114059

gerritbot added a comment.Oct 20 2015, 9:59 PM

Change 244617 merged by Dzahn:
dumps: add cert expiry check

https://gerrit.wikimedia.org/r/244617

gerritbot added a comment.Oct 20 2015, 10:12 PM

Change 247744 had a related patch set uploaded (by Dzahn):
planet: add ssl cert expiry check

https://gerrit.wikimedia.org/r/247744

gerritbot added a comment.Oct 20 2015, 10:15 PM

Change 247744 merged by Dzahn:
planet: add ssl cert expiry check

https://gerrit.wikimedia.org/r/247744

Dzahn added a comment.Oct 20 2015, 11:05 PM

dumps: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=dataset1001&service=HTTPS
OTRS: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=iodine&service=HTTPS
lists: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fermium&service=HTTPS
icinga: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=neon&service=HTTPS
gerrit: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=ytterbium&service=HTTPS
RT: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=magnesium&service=HTTPS
planet: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=planet1001&service=HTTPS (BROKEN, this needs to be on the cp boxes, special case! )
librenms: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=netmon1001&service=HTTPS

gerritbot added a comment.Oct 21 2015, 7:45 PM

Change 247905 had a related patch set uploaded (by Dzahn):
icinga: ssl cert monitoring for external services

https://gerrit.wikimedia.org/r/247905

gerritbot added a comment.Oct 21 2015, 8:13 PM

Change 247905 merged by Dzahn:
icinga: ssl cert monitoring for external services

https://gerrit.wikimedia.org/r/247905

Dzahn added a comment.Oct 21 2015, 8:18 PM

archiva: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=titanium&service=HTTPS
ganglia: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=uranium&service=HTTPS
rcstream: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=rcs1001&service=HTTPS

Dzahn added a comment.Oct 21 2015, 10:17 PM

policy: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=policy.wikimedia.org
blog: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=blog.wikimedia.org&service=HTTPS-blog
toolserver: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=www.toolserver.org&service=HTTPS-toolserver
eventdonations: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=eventdonations.wikimedia.org&service=HTTPS-eventdonations (HOST down due to filtered ICMP?)

Dzahn added a comment.Oct 21 2015, 11:21 PM

planet: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=1&host=en.planet.wikimedia.org
labs / tools.wmflabs.org https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=tools.wmflabs.org&service=HTTPS-wmflabs
ldap-codfw: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=nembus&service=LDAP-SSL
ldap-eqiad: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=neptunium&service=LDAP-SSL

Dzahn added a comment.EditedOct 21 2015, 11:23 PM

ok, all done, except these remnants.

can you help me here? what's the status?

ecc-star.wmfusercontent.org.crt
labvirt-star.eqiad.wmnet.crt
ldap-mirror.wikimedia.org.crt
star.wmfusercontent.org.crt
virt-star.eqiad.wmnet.crt

@Andrew @chasemp @yuvipanda can you tell me if these certs above are actually used and on which FQDN they are expected?

Dzahn added subscribers: Andrew, • yuvipanda, • chasemp.Oct 21 2015, 11:28 PM
Dzahn added a comment.Oct 21 2015, 11:41 PM

skip ldap-mirror. that should be this https://gerrit.wikimedia.org/r/#/c/247936/1/manifests/role/openldap.pp

Dzahn added a comment.Oct 22 2015, 12:25 AM

wmfusercontent done: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=phab.wmfusercontent.org&service=HTTPS-wmfusercontent

LDAP (corp/OIT) mirror (has issues):

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=plutonium&service=SSL-LDAP
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pollux&service=SSL-LDAP

virt-star: ?

Dzahn added a comment.Oct 22 2015, 12:32 AM

re: LDAP-mirror: nothing to monitor here, because Icinga said:

"SSL CRITICAL - failed to connect or SSL handshake:IO::Socket::SSL: connect: Connection refused "

and puppet code says:

port => '389', # Yes, explicitly not supporting LDAPS (port 636)

(but we still have the cert for it? oh well)

Dzahn added a comment.EditedOct 22 2015, 1:17 AM

@Andrew which service on which port uses the virt-star cert? i see virt100x compute nodes had it, and exist in site.pp but _not in DNS_ and labvirt1001 should have it now but none of the services on thse high ports 59xx ? seem to respond to openssl

Krenair added a subscriber: Krenair.Oct 22 2015, 1:20 AM
Andrew added a comment.EditedOct 22 2015, 1:57 PM

virt-star is used by the nova-compute services to talk to each other, for example when migrating instances from one place to another. It's almost certainly self-signed. I'll try to figure out what port you can test it on.

(edit: labvirt-star is the interesting one, virt-star isn't used in production.)

Dzahn added a comment.Oct 22 2015, 10:00 PM

We agreed to split this special case into a non-blocking subtask -> T116332

Dzahn closed this task as Resolved.Oct 22 2015, 10:08 PM

All the certs in the original list in the ticket are covered now. The only exception is the special case for labvirtstar.

Dzahn edited projects, added observability, Icinga, HTTPS; removed Patch-For-Review.Oct 22 2015, 10:09 PM
Dzahn set Security to None.

https://etherpad.wikimedia.org/p/T114059

Dzahn closed subtask T116332: monitor expiration of labvirt-star SSL cert as Resolved.Oct 28 2015, 12:37 AM
Dzahn added a subtask: T120662: Track/alert cassandra certs expiration.Dec 15 2015, 6:09 PM
Dzahn mentioned this in T130366: Should we have a specific check for SSL certificate expiration on elasticsearch.Mar 23 2016, 4:29 PM
Dzahn added a subtask: T130366: Should we have a specific check for SSL certificate expiration on elasticsearch.
Restricted Application added a project: Traffic. · View Herald TranscriptMar 23 2016, 4:30 PM
Deskana closed subtask T130366: Should we have a specific check for SSL certificate expiration on elasticsearch as Resolved.Mar 24 2016, 4:28 PM
fgiunchedi closed subtask T120662: Track/alert cassandra certs expiration as Resolved.Aug 30 2016, 8:43 AM
Phabricator_maintenance removed a subscriber: • yuvipanda.Jun 7 2017, 6:47 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL