Page MenuHomePhabricator

Eventlogging mysql consumer restarting for several hours due to schema parsing errors
Open, HighPublic5 Story Points

Description

See : https://grafana.wikimedia.org/d/000000505/eventlogging?panelId=12&fullscreen&orgId=1&from=1552682997000&to=now

Error trace makes you think some events are sending text like "<div class="mw-body" role="main">" that parser cannot parse and mysql client is crashing:

Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 58, in compose_document
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.get_event()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 118, in get_event
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.current_event = self.state()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 193, in parse_document_end
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: token = self.peek_token()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 128, in peek_token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.fetch_more_tokens()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 257, in fetch_more_tokens
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: % ch.encode('utf-8'), self.get_mark())
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: ScannerError: while scanning for the next token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: found character '\t' that cannot start any token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: in "<unicode string>", line 28, column 1:
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: #011#011<div class="mw-body" role="main">

Event Timeline

Nuria created this task.Mar 20 2019, 10:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2019, 10:54 PM
Nuria set the point value for this task to 5.
Nuria renamed this task from Eventlogging mysql consumer restarting due to schema parsing errors to Eventlogging mysql consumer restarting for several hours due to schema parsing errors.
Nuria updated the task description. (Show Details)Mar 20 2019, 10:55 PM
Nuria moved this task from Done to In Code Review on the Analytics-Kanban board.Mar 20 2019, 11:31 PM

Change 497951 had a related patch set uploaded (by Nuria; owner: Nuria):
[eventlogging@master] Adding logging to help troubleshoot errors with mysql processor

https://gerrit.wikimedia.org/r/497951

Nuria triaged this task as High priority.
Nuria claimed this task.Fri, Mar 22, 4:14 PM
Nuria moved this task from In Code Review to Paused on the Analytics-Kanban board.Tue, Mar 26, 4:06 PM