Page MenuHomePhabricator
Create Task
Maniphest T115121

Add the schema name to the EL EventError topic [8 pts]
Closed, ResolvedPublic
Actions

Description

EventLogging's validation errors are shown live here:
https://logstash.wikimedia.org/#/dashboard/elasticsearch/eventlogging-errors

It would be nice to add one column with the name of the schema and another with revision, so that error logs can be easily identified and grouped by schema. Those infos are already in the raw log, but logstash has difficulties in parsing it. So, we'd need to:

  • create a new version of the EventError schema with 2 new fields: schema and revision
  • modify eventlogging processor to include these infos in the validation errors, and to use the new version of the EventError schema
  • modify the logstash filter in operations/puppet to flatten the new fields?
  • Modify the Kibana dashboard to include the new fields

Event Timeline

mforns created this task.Oct 9 2015, 3:32 PM
mforns updated the task description. (Show Details)
mforns raised the priority of this task from to Needs Triage.
mforns added a project: Analytics-Backlog.
mforns added a subscriber: mforns.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 9 2015, 3:32 PM
Ottomata renamed this task from Add the schema name to the EL EventError topic to Add the schema name to the EL EventError topic [5 pts].Oct 9 2015, 5:38 PM
Ottomata claimed this task.
Ottomata added projects: Analytics-Kanban, Analytics-EventLogging.
Ottomata set Security to None.

I'll do the first 2! :)

gerritbot added a subscriber: gerritbot.Oct 9 2015, 6:55 PM

Change 244729 had a related patch set uploaded (by Ottomata):
Include schema and revision of errored event in EventError if it can be parsed

https://gerrit.wikimedia.org/r/244729

Ottomata renamed this task from Add the schema name to the EL EventError topic [5 pts] to Add the schema name to the EL EventError topic [8 pts].Oct 9 2015, 7:09 PM
Ottomata moved this task from Next Up to In Code Review on the Analytics-Kanban board.
gerritbot added a comment.Oct 19 2015, 6:32 PM

Change 244729 merged by Ottomata:
Include schema and revision of errored event in EventError if it can be parsed

https://gerrit.wikimedia.org/r/244729

Nuria closed this task as Resolved.Oct 22 2015, 4:05 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL