Change Details

Hello ServiceOps, This is a follow-up to T353715 where we enabled kafka log compaction on kafka-jumbo. We are requesting to enable log compaction on the page_rerender topic on kafka-main as the next step on moving the Search Update Pipeline into production. @elukey mentioned his concerns about enabling log compaction [[ https://phabricator.wikimedia.org/T351503#9378280 | here ]] . I believe we have addressed the resource usage concern around log compaction/deletion by [[ https://phabricator.wikimedia.org/T353715 | enabling it on kafka-jumbo topics ]] (thanks @brouberol ! ) . The screenshots attached to T353715 give us a good idea of the resource impact we can expect when we enable it on the page_rerender topic on kafka-main. You can also see [[ https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&from=1704189600000&to=1704197999000&var-datasource=thanos&var-kafka_cluster=jumbo-eqiad&var-cluster=kafka_jumbo&var-kafka_broker=All&var-disk_device=All | kafka-jumbo's resource usage during the change in grafana ]] until it's aged out. Thanks for looking and please let us know if you need additional info around this request.

Hello ServiceOps, This is a follow-up to T353715 where we enabled kafka log compaction on kafka-jumbo. We are requesting to enable log compaction on the page_rerender topic on kafka-main as the next step on moving the Search Update Pipeline into production. Enabling log compaction will save disk space, but our main motivation is to optimize the backfilling process, as explained [[ https://phabricator.wikimedia.org/T351503#9378361 | here ]] . @elukey mentioned his concerns about enabling log compaction [[ https://phabricator.wikimedia.org/T351503#9378280 | here ]] . I believe we have addressed the resource usage concern around log compaction/deletion by [[ https://phabricator.wikimedia.org/T353715 | enabling it on kafka-jumbo topics ]] (thanks @brouberol ! ) . The screenshots attached to T353715 give us a good idea of the resource impact we can expect when we enable it on the page_rerender topic on kafka-main. You can also see [[ https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&from=1704189600000&to=1704197999000&var-datasource=thanos&var-kafka_cluster=jumbo-eqiad&var-cluster=kafka_jumbo&var-kafka_broker=All&var-disk_device=All | kafka-jumbo's resource usage during the change in grafana ]] until it's aged out. Thanks for looking and please let us know if you need additional info around this request.

Hello ServiceOps, This is a follow-up to T353715 where we enabled kafka log compaction on kafka-jumbo. We are requesting to enable log compaction on the page_rerender topic on kafka-main as the next step on moving the Search Update Pipeline into production. Enabling log compaction will save disk space, but our main motivation is to optimize the backfilling process, as explained [[ https://phabricator.wikimedia.org/T351503#9378361 | here ]] . @elukey mentioned his concerns about enabling log compaction [[ https://phabricator.wikimedia.org/T351503#9378280 | here ]] . I believe we have addressed the resource usage concern around log compaction/deletion by [[ https://phabricator.wikimedia.org/T353715 | enabling it on kafka-jumbo topics ]] (thanks @brouberol ! ) . The screenshots attached to T353715 give us a good idea of the resource impact we can expect when we enable it on the page_rerender topic on kafka-main. You can also see [[ https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&from=1704189600000&to=1704197999000&var-datasource=thanos&var-kafka_cluster=jumbo-eqiad&var-cluster=kafka_jumbo&var-kafka_broker=All&var-disk_device=All | kafka-jumbo's resource usage during the change in grafana ]] until it's aged out. Thanks for looking and please let us know if you need additional info around this request.