Page MenuHomePhabricator
Create Task
Maniphest T268200

Beta cluster logstash down
Closed, ResolvedPublic
Actions

Description

Originally, it was returning 502 errors:
https://logstash-beta.wmflabs.org/
502 Bad Gateway
nginx/1.13.6

Currently, it is "up" in the sense that it can be reached, but no events are visible ("No results found")

Related Objects

Event Timeline

DannyS712 created this task.Nov 19 2020, 1:45 AM
Restricted Application added a project: User-DannyS712. · View Herald TranscriptNov 19 2020, 1:45 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
DannyS712 moved this task from Unsorted to Reports on the User-DannyS712 board.Nov 19 2020, 1:46 AM
DannyS712 added a project: Release-Engineering-Team.
herron closed this task as Resolved.Nov 20 2020, 3:34 PM
herron claimed this task.
herron subscribed.

Apache2 on deployment-logstash03 was erroring with [auth_cas:error] [pid 18928:tid 139767719112768] MOD_AUTH_CAS: CASLoginURL or CASValidateURL not defined.

I've disabled the auth_cas module with # a2dismod auth_cas, and after restarting apache2 https://logstash-beta.wmflabs.org is accessible again.

DannyS712 reopened this task as Open.Nov 22 2020, 2:32 AM

@herron its no longer crashing, but the last event shown was on 11/16, and no more recent events are displayed

DannyS712 removed herron as the assignee of this task.Dec 8 2020, 5:19 AM
DannyS712 triaged this task as High priority.

Reset assignee since it was automatically set when @herron closed as resolved, but the issue isn't resolved, and @herron does not appear to still be working on this (please correct me if I'm wrong)

DannyS712 mentioned this in T268210: User-defined project domains should be validated.Dec 20 2020, 7:28 AM
Volans added a project: observability.Dec 21 2020, 10:05 AM
DannyS712 updated the task description. (Show Details)Dec 21 2020, 10:10 AM
colewhite closed this task as Resolved.Dec 22 2020, 4:31 PM
colewhite claimed this task.
colewhite subscribed.

Hi @DannyS712

As of this writing, logs are flowing in deployment-prep again.

DannyS712 added a comment.EditedDec 22 2020, 11:58 PM
In T268200#6708351, @colewhite wrote:

Hi @DannyS712

As of this writing, logs are flowing in deployment-prep again.

I'm still not seeing any mediawiki debug logs though (type: "mediawiki")

colewhite added a comment.Dec 23 2020, 5:27 AM
In T268200#6709451, @DannyS712 wrote:

I'm still not seeing any mediawiki debug logs though (type: "mediawiki")

You're right. I found an incorrect password preventing those logs from getting in. Looks better now to me.

DannyS712 added a comment.Dec 23 2020, 6:31 AM
In T268200#6709792, @colewhite wrote:
In T268200#6709451, @DannyS712 wrote:

I'm still not seeing any mediawiki debug logs though (type: "mediawiki")

You're right. I found an incorrect password preventing those logs from getting in. Looks better now to me.

Indeed! But, looking at the errors dashboard (https://logstash-beta.wmflabs.org/app/kibana#/dashboard/mediawiki-errors) there is Could not locate that visualization (id: Trending-Backtrace-File) - is this related?

colewhite added a comment.Dec 23 2020, 3:53 PM
In T268200#6709830, @DannyS712 wrote:

Indeed! But, looking at the errors dashboard (https://logstash-beta.wmflabs.org/app/kibana#/dashboard/mediawiki-errors) there is Could not locate that visualization (id: Trending-Backtrace-File) - is this related?

It looks like someone may have accidentally deleted that visualization at some point. It's unrelated to how logs get into the system, but is clearly not ideal.

I rebuilt that visualization and repaired the mediawiki-errors dashboard.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL