Mentioned in SAL (#wikimedia-analytics) [2024-01-02T10:56:33Z] <brouberol> configuring [eqiad,codfw].mediawiki.cirrussearch.page_rerender.v1 as compacted topics on jumbo-eqiad - T353715

We can see the impact on the overall topic size

22% of the topic segments were compacted and deleted:

Interesting! Curious, so the reason for using compaction here is just to save space, not necessarily to keep the latest record per key forever?

300 / second is not nothing, but the total disk space used for this stream (~32G across 5 brokers?) isn't that much, and depending on how many distinct keys there, the total number of messages removed by compaction probably won't be that much?

I'm very interested in using compaction for keeping a full current state in Kafka, but I hadn't considered using it for saving disk space.

@Ottomata, yes, this was intended to a) save disk space and b) reduce the number of records that have to be processed in case of a back fill. Events representing "page X has been re-rendered" seemed a perfect use case where only the latest event is meaningful and worth to be kept.

Are you sure you want delete in the policy then? Perhaps you want to keep all the latest event per page forever, so you can backfill fully from the topic?

@Ottomata, we considered this but but decided against it since

a) page_rerender is only of currently five source topics we aggregate so this would only be complete if all those topics would retain records just as long as page_rerender
b) building a full index from scratch based on events is not a scenario we want to cover, mainly because it's a lot more time consuming than starting from an index snapshot and backfill only the delta (since the snapshot was taken)

+1 k!

For reference, here's a screenshot of more kafka metrics around enabling compaction:

{F41663744}