Page MenuHomePhabricator

Eventbus errors: Failed processing event: Failed validating at path rev_id
Closed, ResolvedPublic

Description

Hi everybody,

apologies if this is a duplicate, didn't find anything in Phabricator. After .wmf24 I have seen an increase in Grafana alerts due to HTTP 400s returned by Eventbus:

Apr  9 08:55:41 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) Failed processing event: Failed validating at path rev_id: <Event 3feb5920-5aa5-11e9-8692-14187761398e of schema (u'mediawiki/page/delete', 3)>. {} is not of type 'integer'
Apr  9 08:55:41 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 0 out of 1 events were accepted.
Apr  9 08:55:41 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 400 POST /v1/events (10.64.0.61) 3.95ms
Apr  9 08:55:54 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) Failed processing event: Failed validating at path rev_id: <Event 47aaf98c-5aa5-11e9-a5b6-1866da992497 of schema (u'mediawiki/page/delete', 3)>. {} is not of type 'integer'
Apr  9 08:55:54 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 0 out of 1 events were accepted.
Apr  9 08:55:54 kafka1001 eventlogging-service-eventbus[8042]: (MainThread) 400 POST /v1/events (10.64.32.49) 2.58ms
Apr  9 08:56:03 kafka1001 eventlogging-service-eventbus[8035]: (MainThread) Failed processing event: Failed validating at path rev_id: <Event 4d084a00-5aa5-11e9-ab1a-b083fecf0ff5 of schema (u'mediawiki/page/delete', 3)>. {} is not of type 'integer'
Apr  9 08:56:03 kafka1001 eventlogging-service-eventbus[8035]: (MainThread) 0 out of 1 events were accepted.
Apr  9 08:56:03 kafka1001 eventlogging-service-eventbus[8035]: (MainThread) 400 POST /v1/events (10.64.48.87) 3.39ms

This error is not new, I can see occurrences of it on April 3rd and 4th, but judging from /srv/log/eventlogging/eventlogging-service-eventbus.failed_events.log it seems that some events have indeed a broken revId field.

Event Timeline

elukey triaged this task as High priority.Apr 9 2019, 8:57 AM
elukey created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 9 2019, 8:57 AM

Change 502533 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] icinga::monitoring::services: remove analytics from eventbus alerts

https://gerrit.wikimedia.org/r/502533

Change 502533 merged by Elukey:
[operations/puppet@production] icinga::monitoring::services: remove analytics from eventbus alerts

https://gerrit.wikimedia.org/r/502533

Change 500971 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/extensions/EventBus@master] Followup on EventFactory refactoring

https://gerrit.wikimedia.org/r/500971

Pchelolo edited projects, added Services (doing); removed Services.Apr 9 2019, 4:42 PM

Sorry about that. Fixed by above patch.

Change 500971 merged by jenkins-bot:
[mediawiki/extensions/EventBus@master] Followup on EventFactory refactoring

https://gerrit.wikimedia.org/r/500971