Ok, apparently @bd808 temporarily disabled logging in https://gerrit.wikimedia.org/r/#/c/320016/2 Can that be reverted now?

Have you done anything to change the code that was causing the failures as noted in T150106#2774165 and T150106#2777178? We did not come up with a generic solution for either problem where mixed scalar and object values are stored under the same field name.

Looks like https://phabricator.wikimedia.org/T150106#2777178 was improved by https://github.com/wikimedia/change-propagation/pull/133

Now the event property in ChangeProp is always a string, so it's consistent within the service, Event-Platform extension doesn't ever include the event entry any more, but I'm not sure what EventLogging Service does. @Ottomata ?

Ah ok, I remember what's going on here. So https://phabricator.wikimedia.org/T150106#2777178 is about eventlogging error event logs conflicting with change prop's event object. As @Pchelolo, if changeprop now emits event as a string, this shouldn't conflict.

I'm not sure how https://gerrit.wikimedia.org/r/#/c/320016/2/wmf-config/InitialiseSettings.php was related though, as I don't think the EventBus extension doesn't really emits log errors with an event field.

Can we reenable EventBus extension logstash stuff and see what happens?

I think the conflict was with the eventlogging_EventError topic. This data contains EventLogging Analytics events that did had errors, usually one that did not validate. EventLogging Analytics events are all wrapped in a capsule, with the actual event data contained in an event object.

{
  "wiki": "",
  "uuid": "3548417061a911e7860e90b11c2d80e4",
  "timestamp": 1499276503,
  "schema": "EventError",
  "revision": 14035058,
  "recvFrom": "eventlog1001.eqiad.wmnet",
  "event": {
    "schema": "MobileWikiAppFindInPage",
    "revision": 14586774,
    "rawEvent": "xxxxxxxx",
    "message": "findText is a required property",
    "code": "validation"
  }
}

Marcel had done some work in the past to ingest eventlogging_EventError from Kafka into logstash, but I'm not sure of the current status of that.