Change Details

We have started 2 Flink applications producing data into `pageview.v1` and `pageview.trending.relative.v1` topics. These applications use `webrequest_frontend_text` as source and they are producing a lot of data, which might lead to issues in our infrastructure. - Refinery Hourly started failing with `pageview.v1`, we increased the Spark job and seems stable now. - After 1 day, `pageview.v1` is already 4TB of data. It's supposed to have around 8% of `webrequest_frontend_text`, which is 52TB, but messages are bigger. It looks like it could reach **28 ~ 30TB in 7 days**. - `pageview.trending.relative.v1` reached 3TB in around 12hours. We stopped the application to prevent issues. First question to answer would be, can we afford having 2 new topics with around 30TB each? Some ideas that we could implement to reduce the data: * `pageview.v1` ** [] Reduce retention? maybe 5 days is enough, or even less. The data is meant to be used by the relative trending pipeline. ** [] Reduce message size? We were using the [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Webrequest#Current_Schema | schema used in batch ]], to be as close as possible to it, but we don't really use all that data. *** [x] Removing `x_analytics` field looks a quick win. The data is parsed and stored in their own fields, keeping the raw field is redundant, and it's a big one. *** [x] Remove `tls` field. It's parsed in `tls_map` and both are kept. (Deployed from tag v1.59.0-dev1 - https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/156) ** [] Enable zstd compression? * `pageview.trending.relative.v1` ** [] Reduce Window sliding for now. Data is produced per Window, the Window slides every 5 minutes now. Moving to 10 minutes will reduce the data to half. ** [] Reduce retention ** [x] Enable compaction. We only care about the last status of trending, and as each window produces data, a lot of data repeats the same key. Enabling compaction could reduce the data drastically. ** [] Enable zstd compression. ** Avoid producing useless messages: *** [x] Many pages got less than 5 views per hour, maybe we could avoid producing them. (Dropping all pages with less than 50 hourly views for now. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157 ) *** [] We could consider producing only the change in trending. If a page is not trending, it won't produce new messages. This will reduce the data drastically, but it requires development, keeping more state in memory (possible issues in K8s) and won't allow analyze the data properly. About Hive ingestion. [] Should we create a dedicated process of the large Spark job is enough?

We have started 2 Flink applications producing data into `pageview.v1` and `pageview.trending.relative.v1` topics. These applications use `webrequest_frontend_text` as source and they are producing a lot of data, which might lead to issues in our infrastructure. - Refinery Hourly started failing with `pageview.v1`, we increased the Spark job and seems stable now. - After 1 day, `pageview.v1` is already 4TB of data. It's supposed to have around 8% of `webrequest_frontend_text`, which is 52TB, but messages are bigger. It looks like it could reach **28 ~ 30TB in 7 days**. - `pageview.trending.relative.v1` reached 3TB in around 12hours. We stopped the application to prevent issues. First question to answer would be, can we afford having 2 new topics with around 30TB each? Some ideas that we could implement to reduce the data: * `pageview.v1` ** [x] ~~Reduce retention? maybe 5 days is enough, or even less. The data is meant to be used by the relative trending pipeline.~~ (1 day retention for now) ** [x] Reduce message size? We were using the [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Webrequest#Current_Schema | schema used in batch ]], to be as close as possible to it, but we don't really use all that data. *** [x] Removing `x_analytics` field looks a quick win. The data is parsed and stored in their own fields, keeping the raw field is redundant, and it's a big one. *** [x] Remove `tls` field. It's parsed in `tls_map` and both are kept. (Deployed from tag v1.59.0-dev1 - https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/156) ** [] ~~Enable zstd compression?~~ (Not doing it for now) * `pageview.trending.relative.v1` ** [] ~~Reduce Window sliding for now. Data is produced per Window, the Window slides every 5 minutes now. Moving to 10 minutes will reduce the data to half.~~ (Not doing it, not needed) ** [x] Reduce retention (1 day for now) ** [] ~~Enable compaction. We only care about the last status of trending, and as each window produces data, a lot of data repeats the same key. Enabling compaction could reduce the data drastically~~ (Compaction removed, it removes valuable data for the Datalake ** [] ~~Enable zstd compression.~~ (Not doing it yet) ** Avoid producing useless messages: *** [x] Many pages got less than 5 views per hour, maybe we could avoid producing them. (Dropping all pages with less than 50 hourly views for now. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157 ) *** [] ~~We could consider producing only the change in trending. If a page is not trending, it won't produce new messages. This will reduce the data drastically, but it requires development, keeping more state in memory (possible issues in K8s) and won't allow analyze the data properly.~~ (Not doing it) About Hive ingestion. [] ~~Should we create a dedicated process of the large Spark job is enough?~~ Solved with bigger spark jobs configured in the stream config.

We have started 2 Flink applications producing data into `pageview.v1` and `pageview.trending.relative.v1` topics. These applications use `webrequest_frontend_text` as source and they are producing a lot of data, which might lead to issues in our infrastructure. - Refinery Hourly started failing with `pageview.v1`, we increased the Spark job and seems stable now. - After 1 day, `pageview.v1` is already 4TB of data. It's supposed to have around 8% of `webrequest_frontend_text`, which is 52TB, but messages are bigger. It looks like it could reach **28 ~ 30TB in 7 days**. - `pageview.trending.relative.v1` reached 3TB in around 12hours. We stopped the application to prevent issues. First question to answer would be, can we afford having 2 new topics with around 30TB each? Some ideas that we could implement to reduce the data: * `pageview.v1` ** [] x] ~~Reduce retention? maybe 5 days is enough, or even less. The data is meant to be used by the relative trending pipeline. ~~ (1 day retention for now) ** [x] Reduce message size? We were using the [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Webrequest#Current_Schema | schema used in batch ]], to be as close as possible to it, but we don't really use all that data. *** [x] Removing `x_analytics` field looks a quick win. The data is parsed and stored in their own fields, keeping the raw field is redundant, and it's a big one. *** [x] Remove `tls` field. It's parsed in `tls_map` and both are kept. (Deployed from tag v1.59.0-dev1 - https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/156) ** [] ~~Enable zstd compression?~~ (Not doing it for now) * `pageview.trending.relative.v1` ** [] ~~Reduce Window sliding for now. Data is produced per Window, the Window slides every 5 minutes now. Moving to 10 minutes will reduce the data to half. ~~ (Not doing it, not needed) ** [x] Reduce retentionon (1 day for now) ** [x] ] ~~Enable compaction. We only care about the last status of trending, and as each window produces data, a lot of data repeats the same key. Enabling compaction could reduce the data drastically~~ (Compaction removed, Enabling compaction could reduce the data drastically.it removes valuable data for the Datalake ** [] ~~Enable zstd compression.~~ (Not doing it yet) ** Avoid producing useless messages: *** [x] Many pages got less than 5 views per hour, maybe we could avoid producing them. (Dropping all pages with less than 50 hourly views for now. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157 ) *** [] ~~We could consider producing only the change in trending. If a page is not trending, it won't produce new messages. This will reduce the data drastically, but it requires development, keeping more state in memory (possible issues in K8s) and won't allow analyze the data properly.~~ (Not doing it) About Hive ingestion. [] ~~Should we create a dedicated process of the large Spark job is enough?~~ Solved with bigger spark jobs configured in the stream config.