Page MenuHomePhabricator

Add the schema name to the EL EventError topic [8 pts]
Closed, ResolvedPublic

Description

EventLogging's validation errors are shown live here:
https://logstash.wikimedia.org/#/dashboard/elasticsearch/eventlogging-errors

It would be nice to add one column with the name of the schema and another with revision, so that error logs can be easily identified and grouped by schema. Those infos are already in the raw log, but logstash has difficulties in parsing it. So, we'd need to:

  • create a new version of the EventError schema with 2 new fields: schema and revision
  • modify eventlogging processor to include these infos in the validation errors, and to use the new version of the EventError schema
  • modify the logstash filter in operations/puppet to flatten the new fields?
  • Modify the Kibana dashboard to include the new fields

Event Timeline

mforns created this task.Oct 9 2015, 3:32 PM
mforns raised the priority of this task from to Needs Triage.
mforns updated the task description. (Show Details)
mforns added a project: Analytics-Backlog.
mforns added a subscriber: mforns.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 9 2015, 3:32 PM
Ottomata renamed this task from Add the schema name to the EL EventError topic to Add the schema name to the EL EventError topic [5 pts].Oct 9 2015, 5:38 PM
Ottomata claimed this task.
Ottomata set Security to None.

I'll do the first 2! :)

Change 244729 had a related patch set uploaded (by Ottomata):
Include schema and revision of errored event in EventError if it can be parsed

https://gerrit.wikimedia.org/r/244729

Ottomata renamed this task from Add the schema name to the EL EventError topic [5 pts] to Add the schema name to the EL EventError topic [8 pts].Oct 9 2015, 7:09 PM
Ottomata moved this task from Next Up to In Code Review on the Analytics-Kanban board.

Change 244729 merged by Ottomata:
Include schema and revision of errored event in EventError if it can be parsed

https://gerrit.wikimedia.org/r/244729

Nuria closed this task as Resolved.Oct 22 2015, 4:05 PM