Hello SRE Team,
We are deploying a couple of new Flink applications and we'd like to configure some Kafka topics properly. The topics doesn't exist yet, but they will exist as soon as we deploy, so maybe it's easier to create them with the right config before?
These are the topics and needed configurations, all of them in Jumbo:
| Topic | Partitions | Extra configs |
|---|---|---|
| eqiad.pageview.v1 | 100 | |
| eqiad.pageview.stats.v1 | 1 | cleanup.policy=compact,delete; retention.ms=2419200000 |
| eqiad.pageview.trending.relative.v1 | 3 | |
- eqiad.pageview.v1: This topic receives around 8% of webrequest_frontent_text. 8% is around 4 ~ 5TB, but the message is smaller, so it's hard to estimate it properly. But maybe 100 is a good starting point? It's ok to use fewer partitions and change later too.
- eqiad.pageview.stats.v1: Is a lookup table, only 3 ~ 12 million records. Data is produced weekly here, so we want more retention than 7 days. Compacted topic, we only care about the latest value for each key.
- eqiad.pageview.trending.relative.v1: Relative trending indicator, it's very hard to estimate it's size, it produces a record every 5 minutes per page viewed. 1M pages viewed in 1h means 12M records in 1h. Maybe 3 is a good starting point, but we should monitor if it requires more. I don't think the default 1 is ok, at least to avoid sending all the data to only 3 brokers.
I believe these are the commands that should be run, but I cannot do it myself:
sudo -E kafka topics --create --topic eqiad.pageview.v1 --partitions 100
sudo -E kafka topics --create --topic eqiad.pageview.stats.v1 --partitions 1 \
--config "cleanup.policy=compact,delete" --config retention.ms=2419200000
sudo -E kafka topics --create --topic eqiad.pageview.trending.relative.v1 --partitions 3