Page MenuHomePhabricator

Eventlogging mysql consumer restarting for several hours due to schema parsing errors
Closed, DeclinedPublic5 Story Points

Description

See : https://grafana.wikimedia.org/d/000000505/eventlogging?panelId=12&fullscreen&orgId=1&from=1552682997000&to=now

Error trace makes you think some events are sending text like "<div class="mw-body" role="main">" that parser cannot parse and mysql client is crashing:

Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 58, in compose_document
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.get_event()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 118, in get_event
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.current_event = self.state()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 193, in parse_document_end
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: token = self.peek_token()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 128, in peek_token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.fetch_more_tokens()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 257, in fetch_more_tokens
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: % ch.encode('utf-8'), self.get_mark())
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: ScannerError: while scanning for the next token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: found character '\t' that cannot start any token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: in "<unicode string>", line 28, column 1:
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: #011#011<div class="mw-body" role="main">

Details

Related Gerrit Patches:

Event Timeline

Nuria created this task.Mar 20 2019, 10:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 20 2019, 10:54 PM
Nuria renamed this task from Eventlogging mysql consumer restarting due to schema parsing errors to Eventlogging mysql consumer restarting for several hours due to schema parsing errors.Mar 20 2019, 10:54 PM
Nuria added a project: Analytics.
Nuria set the point value for this task to 5.
Nuria updated the task description. (Show Details)Mar 20 2019, 10:55 PM
Nuria moved this task from Done to In Code Review on the Analytics-Kanban board.Mar 20 2019, 11:31 PM

Change 497951 had a related patch set uploaded (by Nuria; owner: Nuria):
[eventlogging@master] Adding logging to help troubleshoot errors with mysql processor

https://gerrit.wikimedia.org/r/497951

Nuria triaged this task as High priority.Mar 21 2019, 4:07 PM
Nuria moved this task from Incoming to Operational Excellence on the Analytics board.
Nuria claimed this task.Mar 22 2019, 4:14 PM
Nuria moved this task from In Code Review to Paused on the Analytics-Kanban board.Mar 26 2019, 4:06 PM
Nuria added a comment.Aug 25 2019, 8:51 AM

Closing this ticket as we will be deprecating mysql consumer by end of Q1/early Q2

Nuria closed this task as Resolved.Aug 25 2019, 8:51 AM
Nuria changed the task status from Resolved to Declined.