Page MenuHomePhabricator
Create Task
Maniphest T211984

Logstash in beta fails periodically
Open, Needs TriagePublic
Actions

Description

Today there were no logs again in beta, presumably because logstash was "stuck" (i.e. a logstash restart fixed it). Similar to T205863: Logstash in beta doesn't have any logs but no obvious configuration issue afaics.

  • Setup monitoring and alerting at least for logstash and elasticsearch
  • Increase logstash logging verbosity

Related Objects

Event Timeline

fgiunchedi created this task.Dec 14 2018, 4:12 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2018, 4:12 PM
fgiunchedi merged a task: T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14.Jan 8 2019, 12:06 PM
fgiunchedi added subscribers: Jdforrester-WMF, Smalyshev.
fgiunchedi mentioned this in T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14.Jan 8 2019, 12:11 PM
EBernhardson mentioned this in T215204: deployment prep's logstash-beta.wmflabs.org reports no logs since Jan 14th.Feb 4 2019, 9:22 PM
fgiunchedi added a project: observability.Aug 19 2019, 2:26 PM
Jdforrester-WMF mentioned this in T239868: Logstash in Beta Cluster stopped ingesting messages from MediaWiki.Dec 4 2019, 10:14 PM
fgiunchedi moved this task from Inbox to Radar on the observability board.Dec 9 2019, 11:36 AM
Krinkle mentioned this in T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.Jan 14 2020, 5:08 PM
Krinkle mentioned this in T254174: 1.35.0-wmf.37 deployment blockers.Jun 16 2020, 4:02 PM
Krinkle mentioned this in T274593: Logstash beta is not getting any events.Feb 11 2021, 9:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL