Per IRC conversation with @Joe , he is the right person to approve this change in the absence of @elukey . He'll review and make a decision on this as time permits.

It generally seems ok, but a few considerations:

• kafka-main is much smaller than kafka-jumbo, and critical to site operations
• The codfw.mediawiki.currussearch.page_rerender.v1 topic is pretty large at the moment, 292 GB in codfw and 149 GB in eqiad, while the corresponding eqiad topic is as expected tiny/irrelevant.

There is still the possibility that the initial compaction has a negative effect on the producers, which could degrade site performance and functionality.

I think that to play it fully safe we have two options:

• Limit impact by following traffic
  1. We perform the compaction now on the eqiad cluster, which is mostly unused at the moment for producing events as the main datacenter is codfw
  2. After the switchover on the week of the next equinox, so in march, we also do codfw

• Limit impact by reducing temporarily retention
  1. reduce retention to say 1 day, thus reducing the amount of data to compress to 1/7th of what it is now
  2. apply the change
  3. re-raise retention after compaction has happened

I'm pretty agnostic about which way we're going to go between the two.

I also want to note that this doesn't solve the long-standing issue of persisting topic configurations somehow.

I also want to note that this doesn't solve the long-standing issue of persisting topic configurations somehow.

As an aside, this is *very much* on my radar and something I'd like to fix in the near future if possible.

Discussion with @Joe : no objection to enabling compaction as long as we follow one of the option to reduce impact.

I'm going to go ahead, and go with solution 1. As there's no strong favor and support for solution 2, I'm going to implement the one in which there's no data loss.

Mentioned in SAL (#wikimedia-operations) [2024-01-31T14:53:39Z] <brouberol> I'm going to apply kafka log compaction for {eqiad,codfw}.mediawiki.currussearch.page_rerender.v1 on kafka-main-eqiad only (current replica) - T354794

Looking at the topic sizes, I'm going to change the config of the smallest topic (eqiad.mediawiki.cirrussearch.page_rerender.v1), wait a bit, then apply the same change on the bigger topic (codfw.mediawiki.cirrussearch.page_rerender.v1).

The topic is so small, the effect of compaction went completely unregistered.

brouberol@kafka-main1003:~$ kafka configs --entity-type topics --entity-name 'codfw.mediawiki.cirrussearch.page_rerender.v1' --alter --add-config 'cleanup.policy=[compact,delete]'
kafka-configs --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --entity-type topics --entity-name codfw.mediawiki.cirrussearch.page_rerender.v1 --alter --add-config cleanup.policy=[compact,delete]
Completed Updating config for entity: topic 'codfw.mediawiki.cirrussearch.page_rerender.v1'.

The compaction of the 160GB topic took about 30 minutes, and had a noticeable impact on R/W IOPS and disk time. The topic is now 25% smaller.

However, no producer impact was seen (there's no active consumer on that cluster)
{F41735142}

Moving to our backlog board, to be picked up again after March 20th 2024

Mentioned in SAL (#wikimedia-operations) [2024-03-26T08:31:55Z] <brouberol> I'm going to apply kafka log compaction for {eqiad,codfw}.mediawiki.currussearch.page_rerender.v1 on kafka-main-codfw only (current replica) - T354794

All done! It had quite the impact on eqiad.mediawiki.cirrussearch.page_rerender.v1 (36% size reduction), and a bit less on codfw.mediawiki.cirrussearch.page_rerender.v1 (15% size reduction).