Page MenuHomePhabricator

Emergency response to logstash being backlogged
Open, Needs TriagePublic

Description

During the row D blip and D2 failure logstash developed a bit of backlog due to a large amount of logs being generated. In such occasions it might be needed to clear the backlog so that recent logs can be seen again to e.g. debug still ongoing issues, see also incident report

Outline of what is needed:

  • Spicerack cookbook to stop logstash + reset kafka offsets (for a subset of topics if desired) + start logstash
  • The above is the bare minimum, which would work but has the downside of effectively losing the backlog, thus as an improvement the previous and new kafka offsets should be persisted somewhere. Then we can go back later and replay the topics from/to those offsets to backfill the backlog (assuming this happens before the kafka retention expires)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 24 2019, 4:30 PM
fgiunchedi updated the task description. (Show Details)Sep 24 2019, 4:31 PM