Add better monitoring for Analytics UIs
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	Mar 18 2021, 6:53 AM

Description

We need to have better monitoring for Superset and Turnilo:

monitoring::service { 'superset':
    description   => 'superset',
    check_command => "check_tcp!${::superset::port}",
    require       => Class['::superset'],
    notes_url     => 'https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset',
}

    monitoring::service { 'turnilo':
        description   => 'turnilo',
        check_command => "check_tcp!${port}",
        contact_group => $contact_group,
        notes_url     => 'https://wikitech.wikimedia.org/wiki/Analytics/Systems/Turnilo-Pivot',
    }

These are effective only if the daemon is down, since the port is not available anymore, but not if the daemon is up but responding with errors (like happened this morning).

Superset is tricky due to the auth scheme, maybe we could have a test user and use something like https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset#Test_as_different_user_on_staging ? Turnilo should be easy.

Details

Subject	Repo	Branch	Lines +/-
turnilo: Fix port variable dererence for monitor	operations/puppet	production	+1 -1
turnilo: add monitoring for node application	operations/puppet	production	+2 -2
superset: Enable superset http check	operations/puppet	production	+0 -1
superset: use different user headers for staging and production	operations/puppet	production	+10 -3
superset: ensure http check absent until it is working	operations/puppet	production	+8 -7
superset: comment out check that isn't working as intended	operations/puppet	production	+10 -9
superset: put puppet:// resource in files/	operations/puppet	production	+0 -0
superset: check http server following redirects with curl	operations/puppet	production	+31 -0

Customize query in gerrit

Event Timeline

elukey triaged this task as High priority.Mar 18 2021, 6:53 AM

elukey created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 18 2021, 6:53 AM

• fdans assigned this task to • razzi.Mar 18 2021, 4:54 PM

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.

Change 673556 had a related patch set uploaded (by Razzi; owner: Razzi):
[operations/puppet@production] turnilo: add monitoring for http

https://gerrit.wikimedia.org/r/673556

gerritbot added a project: Patch-For-Review.Mar 19 2021, 6:15 PM

@elukey I made a patch for turnilo, should be straightforward as you said.

For superset, if monitoring check come from the host itself, we should be able to use the user header and can either create a new user, or just use "admin".

Here's a curl command that checks if superset is running:

curl -L 'http://localhost:9080/login/' -H 'X-Remote-User: admin' -c cookiejar-$RANDOM

For Nagios to implement this, we'll have to ensure the following:

redirects are enabled (-L in curl) since /login will set a session cookie and redirect to /superset/welcome/
cookies are preserved between redirects (-c cookiejar-$RANDOM above, the $RANDOM is there to get a fresh cookie jar each time)

and of course we'll have to pass the X-Remote-User header. Apparently X-Forwarded-Proto is not needed here.

Apparently check_http does not support following redirects with cookies: https://github.com/monitoring-plugins/monitoring-plugins/issues/1423

• razzi added a project: Analytics-Kanban.Apr 8 2021, 6:42 PM

Alright, nagios doesn't support it directly, but we can use a shell script, like modules/profile/files/eventstreams/check_eventstreams.sh

Change 678044 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] superset: check http server following redirects with curl

https://gerrit.wikimedia.org/r/678044

• razzi moved this task from Next Up to In Code Review on the Analytics-Kanban board.Apr 8 2021, 8:04 PM

Change 678044 merged by Razzi:

[operations/puppet@production] superset: check http server following redirects with curl

https://gerrit.wikimedia.org/r/678044

• razzi moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Apr 8 2021, 10:07 PM

Seeing this error when running puppet on an-tool1010:

Error: /Stage[main]/Profile::Superset/File[/usr/local/bin/check_superset_http]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/superset/check_superset_http.sh
Notice: /Stage[main]/Profile::Superset/Nrpe::Monitor_service[check_superset_http]/Nrpe::Check[check_check_superset_http]/File[/etc/nagios/nrpe.d/check_check_superset_http.cfg]: Dependency File[/usr/local/bin/check_superset_http] has failures: true
Warning: /Stage[main]/Profile::Superset/Nrpe::Monitor_service[check_superset_http]/Nrpe::Check[check_check_superset_http]/File[/etc/nagios/nrpe.d/check_check_superset_http.cfg]: Skipping because of failed dependencies
Warning: /Stage[main]/Nrpe/Base::Service_unit[nagios-nrpe-server]/Service[nagios-nrpe-server]: Skipping because of failed dependencies

Change 678109 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] superset: put puppet:// resource in files/

https://gerrit.wikimedia.org/r/678109

Change 678109 merged by Razzi:

[operations/puppet@production] superset: put puppet:// resource in files/

https://gerrit.wikimedia.org/r/678109

Fixed the puppet:/// resource error, now getting curl code 47, too many redirects

Ok, interesting, the error is happening only on an-tool1010, on an-tool1005 it works fine. I'll roll the check back for now while I look into a command that works on both machines.

Change 678113 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] superset: comment out check that isn't working as intended

https://gerrit.wikimedia.org/r/678113

Change 678113 merged by Razzi:

[operations/puppet@production] superset: comment out check that isn't working as intended

https://gerrit.wikimedia.org/r/678113

Change 678130 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] superset: ensure http check absent until it is working

https://gerrit.wikimedia.org/r/678130

Change 678130 merged by Razzi:

[operations/puppet@production] superset: ensure http check absent until it is working

https://gerrit.wikimedia.org/r/678130

Change 678966 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] superset: use different user headers for staging and production

https://gerrit.wikimedia.org/r/678966

Change 678966 merged by Razzi:

[operations/puppet@production] superset: use different user headers for staging and production

https://gerrit.wikimedia.org/r/678966

I implemented a check that works on both staging and production, using the appropriate header for production (x-cas-uid rather than x-remote-user). I re-enabled alarms and everything looks good here.

Change 679985 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] superset: Enable superset http check

https://gerrit.wikimedia.org/r/679985

Change 679985 merged by Razzi:

[operations/puppet@production] superset: Enable superset http check

https://gerrit.wikimedia.org/r/679985

• fdans closed this task as Resolved.Apr 29 2021, 2:56 PM

Change 673556 merged by Razzi:

[operations/puppet@production] turnilo: add monitoring for node application

https://gerrit.wikimedia.org/r/673556

Change 800231 had a related patch set uploaded (by Dzahn; author: BCornwall):

[operations/puppet@production] turnilo: Fix port variable dererence for monitor