Page MenuHomePhabricator

Add logstash diagnostics
Open, LowPublic

Description

The logging pipeline is getting many features we want to reliably work. In order to better diagnose its behavior, deep diagnostic data is needed. Some of this data was historically put into the tags field which would serve us better as a human interface rather than a diagnostic interface.

  1. Must integrate with testing
  2. Must be easily enabled/disabled
  3. Can provide deep information about how the pipeline is working
    1. Which input did <event> come from
    2. Which filters did <event> hit?
    3. Which output was <event> assigned to?
    4. Which bucket was <event> assigned?
    5. Which partition (if any) was <event> assigned?

Event Timeline

Change 775375 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add pipeline diagnostics

https://gerrit.wikimedia.org/r/775375

Change 778353 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reprioritize dlq filter

https://gerrit.wikimedia.org/r/778353

Change 778353 merged by Cwhite:

[operations/puppet@production] logstash: reprioritize dlq filter

https://gerrit.wikimedia.org/r/778353

Change 775375 merged by Cwhite:

[operations/puppet@production] logstash: populate target index format and add pipeline diagnostics

https://gerrit.wikimedia.org/r/775375

Change 822451 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add target index validation to tests

https://gerrit.wikimedia.org/r/822451

Change 822451 merged by Cwhite:

[operations/puppet@production] logstash: add target index validation to tests

https://gerrit.wikimedia.org/r/822451

lmata triaged this task as Medium priority.
lmata moved this task from Backlog to Prioritized on the Observability-Logging board.
colewhite lowered the priority of this task from Medium to Low.Sep 21 2022, 10:38 AM
colewhite moved this task from Prioritized to Backlog on the Observability-Logging board.

Can provide deep information about how the pipeline is working
Which input did <event> come from
Which filters did <event> hit?
...

Logstash does not currently support it without a lot of manual configuration. Possibly explore adding a feature to add the plugin id to an array for all filters that evaluated successfully (possibly @metadata).