Hi everybody,
apologies if this is a duplicate, didn't find anything in Phabricator. After .wmf24 I have seen an increase in Grafana alerts due to HTTP 400s returned by Eventbus:
Apr 9 08:55:41 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) Failed processing event: Failed validating at path rev_id: <Event 3feb5920-5aa5-11e9-8692-14187761398e of schema (u'mediawiki/page/delete', 3)>. {} is not of type 'integer'
Apr 9 08:55:41 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 0 out of 1 events were accepted.
Apr 9 08:55:41 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 400 POST /v1/events (10.64.0.61) 3.95ms
Apr 9 08:55:54 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) Failed processing event: Failed validating at path rev_id: <Event 47aaf98c-5aa5-11e9-a5b6-1866da992497 of schema (u'mediawiki/page/delete', 3)>. {} is not of type 'integer'
Apr 9 08:55:54 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 0 out of 1 events were accepted.
Apr 9 08:55:54 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 400 POST /v1/events (10.64.32.49) 2.58ms
Apr 9 08:56:03 kafka1001 eventlogging-service-eventbus[8035]: (MainThread) Failed processing event: Failed validating at path rev_id: <Event 4d084a00-5aa5-11e9-ab1a-b083fecf0ff5 of schema (u'mediawiki/page/delete', 3)>. {} is not of type 'integer'
Apr 9 08:56:03 kafka1001 eventlogging-service-eventbus[8035]: (MainThread) 0 out of 1 events were accepted.
Apr 9 08:56:03 kafka1001 eventlogging-service-eventbus[8035]: (MainThread) 400 POST /v1/events (10.64.48.87) 3.39msThis error is not new, I can see occurrences of it on April 3rd and 4th, but judging from /srv/log/eventlogging/eventlogging-service-eventbus.failed_events.log it seems that some events have indeed a broken revId field.