@Nuria: would love to get your thoughts on this

@mpopov there are two things here:

• send events in batches for better use of radio in mobile, that already happens (that is how mw,.track works). See: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/EventLogging/+/524575/ for inclusion of that change into eventlogging so the perf benefits from sending events "together" (not in an array but in batches) are unrelated to EventGate ability to process arrays

• eventgate adds the ability to put events on an array, that, per se, has no perf benefit unless a mechanism to enque the events exists in the client cause you can send an array of events 3 times in one second which will overuse the raido, makes sense?

In the eventgate pipeline the events in the array are unbundled and the pipeline can handle that w/o issues.

@Nuria: By batching I meant an array of events sent in a single request as one payload – as one batch of events. You're referring to a "burst" behavior where events are enqueued and then the queue is flushed, rapidly sending one event after another as separate requests. The patch you linked to – as I understand it and the way we've done it with EPC – is that rapid-fire, "burst" behavior.

In some situations, there might be a benefit of POSTing 1 big request containing an array (batch) of, say, 10 events, than individually POSTing 10 small requests in a rapid succession.

In some situations, there might be a benefit of POSTing 1 big request containing an array (batch) of, say, 10 events, than individually POSTing 10 small requests in a rapid succession.

In some maybe, in the majority of them, I doubt it. The bulk of your time and battery consumption in mobile goes into waking up the radio (as you mentioned earlier) which you have to do in either case. So the critical perf improvements (when it comes to the battery) are gained from walking up the radio less frequently.

rather than navigator.sendBeacon one at a time which negatively impacts next page load

This statement is , I think, incorrect. The SendBeacon API was designed for this use case in mind, to not affect the next page navigation. It comes to solve that very problem:
"By using the sendBeacon() method, the data is transmitted asynchronously to the web server when the User Agent has an opportunity to do so, without delaying the unload or affecting the performance of the next navigation". See: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon

an optimal batch size

The current (adjustable) limits are independent of the batch array length, but are instead about byte length of individual
messages, and also the total POST body byte length.

From https://github.com/wikimedia/operations-deployment-charts/blob/master/helmfile.d/services/eqiad/eventgate-logging-external/values.yaml#L3-L8

    # The request body is accepted as an array of events, each of which
    # will be an individual message in Kafka.  Each individual
    # message must be smaller than message.max.bytes, but EventGate
    # can accept multiple events at once in the request body.
    # Limit this to a smaller size for this externally accessible eventgate instance.
    max_body_size: 4mb
# ...
        # Enforce a smaller message size limit for this externally accessible eventgate instance.
        message.max.bytes: 1048576

So with current settings, each message must be smaller than 1MB, and the entire POST body must be smaller than 4MB.

So the critical perf improvements (when it comes to the battery) are gained from walking up the radio less frequently.

@Nuria even if this is true, is there a reason not to batch events together?

@Nuria even if this is true, is there a reason not to batch events together?

That is fine and dandy but the main argument to be made is, I think, not a performance one.

TODO: Verify that this is indeed happening in all consolidated libraries.

This task applies specifically to the Android and iOS (Java and Swift) libraries, so I've updated the title accordingly. It doesn't apply to PHP, and is handled in MediaWiki JS by the existing EventLogging modules.

This will depend on the integration work, so I'll add the Android and iOS MP client integration tasks as subtasks here.

For the record, I believe that the Android app is already sending batched events, but iOS is not.