Page MenuHomePhabricator

Experiment with automating error log triage
Open, Needs TriagePublic3 Estimated Story Points

Description

We should mess around with building (or finding) something that sees errors in logstash, dedupes them, and turns them into Phabricator tasks with far less manual intervention and general toil than the current painstaking and error-prone process.

We've discussed a lot of ideas for this sort of thing during triage sessions, train deploys, and casual ranting during offsites. Let's just experiment.

A few notes from the last time this came up:

  • Fully automated pipelining of tasks into Phabricator is probably difficult because of potential security issues, etc. On the other hand, it ought to be possible to get a list of errors and click a button to say either "make this a task" or "this is associated with the existing errors". Phatality but more so, essentially.
  • Kibana is extremely painful to use. Can we just cut it out of the interaction loop entirely (but maybe keep filters in sync)?
  • Or can we just replace the new errors dashboard entirely?

Event Timeline

brennen renamed this task from Automate error log triage to Experiment with automating error log triage.Sep 2 2021, 4:31 PM
brennen set the point value for this task to 3.

Hi @brennen there is a lot to unpack here. Maybe we can have a quick chat between teams and understand a bit more about the specific pain points. I'll reach out offline to coordinate some time to talk.

@lmata - yeah, would be great to have a chat. The task description here mostly just captures a recurring conversation between train deployers and other log triagers.

@brennen I've sent an invite with proposed timing for the 20th. Please feel free to propose alternate schedules and include additional folks from your team.

@brennen I've sent an invite with proposed timing for the 20th. Please feel free to propose alternate schedules and include additional folks from your team.

Brennen is on vacation this week. If you add me to the invite I can do the requisite wrangling :)