Compaction cleanup.policy=compact,delete was added to pageview.trending.relative.v1 but it is causing other issues. There is an alert GobblinKafkaRecordsExtractedNotEqualRecordsExpected that gets confused as the number of offsets doesn't match the number of records. We also think that enabling compaction removes some valuable data for the Datalake, so we can to disable compaction now that the topic is controlled.

Hi @Eevans, thanks for looking into it!

I changed a couple of configs. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157

I believe zstd compression cannot be enabled if we don't upgrade the Gobblin Kafka client: https://gitlab.wikimedia.org/repos/data-engineering/gobblin-wmf/-/blob/main/pom.xml#L472-476

We could reduce the size of events on pageview.v1 by removing many fields. I don't think many of the fields are used for Trending right now, but pageview.v1 looks like a nice reusable stream, it could be used for other use cases, specially as it's using the same schema as the batch process.
Maybe @Ottomata you have some ideas about this, I'm not sure how we can measure the importance of the topic versus how much storage it uses.

Thanks! I'll start the pageview.trending.relative.v1 application now and monitorize it during the day. Let's see if it grows that fast or not. Meanwhile I'm looking at some of the other options to reduce the size.

I think adding cleanup.policy = compact,delete to pageview.trending.relative.v1 is a quick win. I think it will reduce the data stored drastically. Do you think we could do it @brouberol?

Buckets created. The application is running now. I'll move the task to Done. Thanks!

After https://gitlab.wikimedia.org/repos/data-engineering/dbt-jobs/-/merge_requests/130

The backfill is running.

That sounds right! after 7 days probably Kafka will remove the .rc0 offsets.

I have doubts about it, because we checked https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/#starting-offset and OffsetsInitializer.earliest() is the default, not OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST).

As of today, the data is well balanced:

Through the task: https://phabricator.wikimedia.org/T429127 we stablished a process to decide partitions based on size. This topic is 30GB and the applications we were building are already working with 1 partition. I think we can close this task.

Do you know if Flink cirrus-streaming-updater is updated to the latest schema that allows negative namespace_ids? There's a checkbox on the task description, I'd like to mark it if it's ready.

I think we can move this task to "Done". There are a few additional tasks that we could do, but they have their own task, and this has been in production for several weeks now.