Page MenuHomePhabricator
Create Task
Maniphest T236343

Alert on logstash index failures on too many fields
Closed, ResolvedPublic
Actions

Description

This is a followup from T234564: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array and specifically about getting alerted when we're hitting elasticsearch's per-index field limits, which in turn usually indicates a "fields explosion" problem.

Details

SubjectRepoBranchLines +/-
prometheus: lower threshold for logstash indexing failuresoperations/puppetproduction+2 -2
logstash: move ingestion alerts to be site-localoperations/puppetproduction+13 -12
logstash: alert on indexing failuresoperations/puppetproduction+12 -0
mtail: export logstash ES index failure detailsoperations/puppetproduction+8 -0
prometheus: collect logstash mtail metricsoperations/puppetproduction+7 -0
hieradata: fix mtail::logs location for logstash roleoperations/puppetproduction+3 -2
profile: add mtail to logstashoperations/puppetproduction+14 -0
mtail: add logstash programoperations/puppetproduction+64 -0
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Oct 24 2019, 8:04 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2019, 8:04 AM
fgiunchedi mentioned this in T234564: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array.Oct 24 2019, 8:05 AM
fgiunchedi added a project: User-fgiunchedi.Oct 25 2019, 1:17 PM
fgiunchedi moved this task from Inbox to Up next on the observability board.Oct 28 2019, 2:16 PM
fgiunchedi added a project: Wikimedia-Logstash.Oct 28 2019, 2:26 PM
herron subscribed.Oct 28 2019, 7:31 PM
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Oct 31 2019, 12:27 PM
fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Nov 4 2019, 1:42 PM
gerritbot added a comment.Nov 4 2019, 3:03 PM

Change 548280 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] mtail: add logstash program

https://gerrit.wikimedia.org/r/548280

gerritbot added a project: Patch-For-Review.Nov 4 2019, 3:03 PM

Change 548281 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] profile: add mtail to logstash

https://gerrit.wikimedia.org/r/548281

gerritbot added a comment.Nov 6 2019, 9:40 AM

Change 548280 merged by Filippo Giunchedi:
[operations/puppet@production] mtail: add logstash program

https://gerrit.wikimedia.org/r/548280

gerritbot added a comment.Nov 6 2019, 9:40 AM

Change 548281 merged by Filippo Giunchedi:
[operations/puppet@production] profile: add mtail to logstash

https://gerrit.wikimedia.org/r/548281

gerritbot added a comment.Nov 6 2019, 9:48 AM

Change 548975 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: fix mtail::logs location for logstash role

https://gerrit.wikimedia.org/r/548975

gerritbot added a comment.Nov 6 2019, 9:49 AM

Change 548975 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: fix mtail::logs location for logstash role

https://gerrit.wikimedia.org/r/548975

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2019, 10:10 AM
gerritbot added a comment.Nov 12 2019, 11:06 AM

Change 550446 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: collect logstash mtail metrics

https://gerrit.wikimedia.org/r/550446

gerritbot added a project: Patch-For-Review.Nov 12 2019, 11:06 AM
gerritbot added a comment.Nov 12 2019, 11:12 AM

Change 550446 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: collect logstash mtail metrics

https://gerrit.wikimedia.org/r/550446

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2019, 12:10 PM
gerritbot added a comment.Nov 12 2019, 2:11 PM

Change 550471 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: alert on indexing failures

https://gerrit.wikimedia.org/r/550471

gerritbot added a project: Patch-For-Review.Nov 12 2019, 2:11 PM
gerritbot added a comment.Nov 13 2019, 9:17 AM

Change 550640 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] mtail: export logstash ES index failure details

https://gerrit.wikimedia.org/r/550640

gerritbot added a comment.Nov 13 2019, 9:28 AM

Change 550640 merged by Filippo Giunchedi:
[operations/puppet@production] mtail: export logstash ES index failure details

https://gerrit.wikimedia.org/r/550640

gerritbot added a comment.Nov 13 2019, 10:46 AM

Change 550471 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: alert on indexing failures

https://gerrit.wikimedia.org/r/550471

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2019, 11:10 AM
fgiunchedi closed this task as Resolved.Nov 13 2019, 11:22 AM
fgiunchedi claimed this task.

This is completed, surges of indexing errors will result in an alert now. Unfortunately the thresholds are a little higher than I expected because of background noise of errors/conflicts (tracked in T238196: Logging fields conflicts (tracking))

gerritbot added a comment.Nov 13 2019, 1:30 PM

Change 550678 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: move ingestion alerts to be site-local

https://gerrit.wikimedia.org/r/550678

gerritbot added a project: Patch-For-Review.Nov 13 2019, 1:30 PM
gerritbot added a comment.Nov 13 2019, 1:33 PM

Change 550678 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: move ingestion alerts to be site-local

https://gerrit.wikimedia.org/r/550678

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2019, 2:10 PM
fgiunchedi mentioned this in T238344: MediaWiki Math invalid JSON in logs on Restbase server error.Nov 14 2019, 4:04 PM
gerritbot added a comment.Nov 22 2019, 12:17 PM

Change 552492 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: lower threshold for logstash indexing failures

https://gerrit.wikimedia.org/r/552492

gerritbot added a project: Patch-For-Review.Nov 22 2019, 12:17 PM
gerritbot added a comment.Nov 22 2019, 12:21 PM

Change 552492 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: lower threshold for logstash indexing failures

https://gerrit.wikimedia.org/r/552492

Maintenance_bot removed a project: Patch-For-Review.Nov 22 2019, 1:10 PM
fgiunchedi mentioned this in T150106: Type collisions in log events causing indexing failures in ELK Elasticsearch.Nov 25 2019, 1:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL