See : https://grafana.wikimedia.org/d/000000505/eventlogging?panelId=12&fullscreen&orgId=1&from=1552682997000&to=now
Error trace makes you think some events are sending text like "<div class="mw-body" role="main">" that parser cannot parse and mysql client is crashing:
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/composer.py", line 58, in compose_document
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.get_event()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 118, in get_event
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.current_event = self.state()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/parser.py", line 193, in parse_document_end
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: token = self.peek_token()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 128, in peek_token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: self.fetch_more_tokens()
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: File "/usr/lib/python2.7/dist-packages/yaml/scanner.py", line 257, in fetch_more_tokens
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: % ch.encode('utf-8'), self.get_mark())
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: ScannerError: while scanning for the next token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: found character '\t' that cannot start any token
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: in "<unicode string>", line 28, column 1:
Mar 19 14:07:41 eventlog1002 eventlogging-consumer@mysql-m4-master-00[12267]: #011#011<div class="mw-body" role="main">
I added some logging to jrm.py (on the box, ahem) to see if i could figure out what schema might be causing errors but they might have disappeared entirely.