Page MenuHomePhabricator

Beta cluster logstash down
Closed, ResolvedPublic

Description

Originally, it was returning 502 errors:
https://logstash-beta.wmflabs.org/
502 Bad Gateway
nginx/1.13.6

Currently, it is "up" in the sense that it can be reached, but no events are visible ("No results found")

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
herron claimed this task.
herron added a subscriber: herron.

Apache2 on deployment-logstash03 was erroring with [auth_cas:error] [pid 18928:tid 139767719112768] MOD_AUTH_CAS: CASLoginURL or CASValidateURL not defined.

I've disabled the auth_cas module with # a2dismod auth_cas, and after restarting apache2 https://logstash-beta.wmflabs.org is accessible again.

@herron its no longer crashing, but the last event shown was on 11/16, and no more recent events are displayed

DannyS712 triaged this task as High priority.

Reset assignee since it was automatically set when @herron closed as resolved, but the issue isn't resolved, and @herron does not appear to still be working on this (please correct me if I'm wrong)

colewhite claimed this task.
colewhite added a subscriber: colewhite.

Hi @DannyS712

As of this writing, logs are flowing in deployment-prep again.

Hi @DannyS712

As of this writing, logs are flowing in deployment-prep again.

I'm still not seeing any mediawiki debug logs though (type: "mediawiki")

I'm still not seeing any mediawiki debug logs though (type: "mediawiki")

You're right. I found an incorrect password preventing those logs from getting in. Looks better now to me.

I'm still not seeing any mediawiki debug logs though (type: "mediawiki")

You're right. I found an incorrect password preventing those logs from getting in. Looks better now to me.

Indeed! But, looking at the errors dashboard (https://logstash-beta.wmflabs.org/app/kibana#/dashboard/mediawiki-errors) there is Could not locate that visualization (id: Trending-Backtrace-File) - is this related?

Indeed! But, looking at the errors dashboard (https://logstash-beta.wmflabs.org/app/kibana#/dashboard/mediawiki-errors) there is Could not locate that visualization (id: Trending-Backtrace-File) - is this related?

It looks like someone may have accidentally deleted that visualization at some point. It's unrelated to how logs get into the system, but is clearly not ideal.

I rebuilt that visualization and repaired the mediawiki-errors dashboard.