Maniphest T211984

Logstash in beta fails periodically
Open, Needs TriagePublic
Actions

Description

Today there were no logs again in beta, presumably because logstash was "stuck" (i.e. a logstash restart fixed it). Similar to T205863: Logstash in beta doesn't have any logs but no obvious configuration issue afaics.

Setup monitoring and alerting at least for logstash and elasticsearch
Increase logstash logging verbosity

Related Objects

Mentioned In: T233134: logstash-beta.wmflabs.org does not receive any mediawiki events
T239868: Logstash in Beta Cluster stopped ingesting messages from MediaWiki
T215204: deployment prep's logstash-beta.wmflabs.org reports no logs since Jan 14th
T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14
Mentioned Here: T205863: Logstash in beta doesn't have any logs

Event Timeline

fgiunchedi created this task.Dec 14 2018, 4:12 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2018, 4:12 PM

fgiunchedi merged a task: T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14.Jan 8 2019, 12:06 PM

fgiunchedi added subscribers: Jdforrester-WMF, Smalyshev.

fgiunchedi mentioned this in T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14.Jan 8 2019, 12:11 PM

EBernhardson mentioned this in T215204: deployment prep's logstash-beta.wmflabs.org reports no logs since Jan 14th.Feb 4 2019, 9:22 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:26 PM

Jdforrester-WMF mentioned this in T239868: Logstash in Beta Cluster stopped ingesting messages from MediaWiki.Dec 4 2019, 10:14 PM

fgiunchedi moved this task from Inbox to Radar on the observability board.Dec 9 2019, 11:36 AM

Krinkle mentioned this in T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.Jan 14 2020, 5:08 PM