Maniphest T211984

Logstash in beta fails periodically
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Dec 14 2018, 4:12 PM

Tags

Referenced Files

None

Subscribers

Jdforrester-WMF

Description

Today there were no logs again in beta, presumably because logstash was "stuck" (i.e. a logstash restart fixed it). Similar to T205863: Logstash in beta doesn't have any logs but no obvious configuration issue afaics.

Setup monitoring and alerting at least for logstash and elasticsearch
Increase logstash logging verbosity

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		colewhite	T211984 Logstash in beta fails periodically
		Resolved		colewhite	T233134 logstash-beta.wmflabs.org does not receive any mediawiki events
		Resolved		taavi	T283013 Migrate beta cluster to ELK7
		Resolved	BUG REPORT	colewhite	T288989 beta logstash servers run out of disk space
		Resolved		bd808	T294978 Redirect logstash-beta.wmflabs.org to beta-logs.wmcloud.org

Event Timeline

fgiunchedi created this task.Dec 14 2018, 4:12 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2018, 4:12 PM

fgiunchedi merged a task: T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14.Jan 8 2019, 12:06 PM

fgiunchedi added subscribers: Jdforrester-WMF, Smalyshev.

fgiunchedi mentioned this in T213129: Beta Cluster ("deployment-prep") logstash has gone silent since 2019-01-02Z01:14:14.Jan 8 2019, 12:11 PM

EBernhardson mentioned this in T215204: deployment prep's logstash-beta.wmflabs.org reports no logs since Jan 14th.Feb 4 2019, 9:22 PM

fgiunchedi added a project: observability.Aug 19 2019, 2:26 PM

Jdforrester-WMF mentioned this in T239868: Logstash in Beta Cluster stopped ingesting messages from MediaWiki.Dec 4 2019, 10:14 PM

fgiunchedi moved this task from Inbox to Radar on the observability board.Dec 9 2019, 11:36 AM

Krinkle mentioned this in T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.Jan 14 2020, 5:08 PM

Krinkle mentioned this in T254174: 1.35.0-wmf.37 deployment blockers.Jun 16 2020, 4:02 PM

Krinkle mentioned this in T274593: Logstash beta is not getting any events.Feb 11 2021, 9:44 PM

thcipriani mentioned this in T288989: beta logstash servers run out of disk space.Aug 16 2021, 7:36 PM

Maintenance_bot added a project: SRE Observability.Aug 16 2021, 7:45 PM

lmata edited projects, added SRE Observability (FY2021/2022-Q2); removed SRE Observability.Aug 17 2021, 3:15 PM

Krinkle added a subtask: T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.Aug 17 2021, 4:59 PM

lmata triaged this task as Medium priority.Sep 30 2021, 9:49 PM

colewhite claimed this task.Oct 27 2021, 10:48 PM

colewhite moved this task from Backlog to Up next on the Wikimedia-Logstash board.Oct 27 2021, 11:29 PM

colewhite closed subtask T233134: logstash-beta.wmflabs.org does not receive any mediawiki events as Resolved.Nov 3 2021, 10:32 PM

colewhite closed this task as Resolved.Nov 17 2021, 10:22 PM