Page MenuHomePhabricator

Eventlogging mysql consumer restarting for several hours due to schema parsing errors
Closed, DeclinedPublic5 Estimated Story Points

Description

See : https://grafana.wikimedia.org/d/000000505/eventlogging?panelId=12&fullscreen&orgId=1&from=1552682997000&to=now

Error trace makes you think some events are sending text like "<div class="mw-body" role="main">" that parser cannot parse and mysql client is crashing:

Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 58, in compose_document
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.get_event()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 118, in get_event
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.current_event = self.state()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 193, in parse_document_end
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: token = self.peek_token()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 128, in peek_token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.fetch_more_tokens()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 257, in fetch_more_tokens
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: % ch.encode('utf-8'), self.get_mark())
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: ScannerError: while scanning for the next token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: found character '\t' that cannot start any token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: in "<unicode string>", line 28, column 1:
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: #011#011<div class="mw-body" role="main">

Event Timeline

Nuria renamed this task from Eventlogging mysql consumer restarting due to schema parsing errors to Eventlogging mysql consumer restarting for several hours due to schema parsing errors.Mar 20 2019, 10:54 PM
Nuria added a project: Analytics.
Nuria set the point value for this task to 5.

Change 497951 had a related patch set uploaded (by Nuria; owner: Nuria):
[eventlogging@master] Adding logging to help troubleshoot errors with mysql processor

https://gerrit.wikimedia.org/r/497951

Nuria triaged this task as High priority.Mar 21 2019, 4:07 PM
Nuria moved this task from Incoming to Operational Excellence on the Analytics board.

Closing this ticket as we will be deprecating mysql consumer by end of Q1/early Q2

Nuria changed the task status from Resolved to Declined.

Change 497951 abandoned by Nuria:
Adding logging to help troubleshoot errors with mysql processor

Reason:
...

https://gerrit.wikimedia.org/r/497951