Page MenuHomePhabricator
Create Task
Maniphest T432204

Review `pageview` and `pageview.trending.relative` Kafka topic sizes.
Open, Needs TriagePublic
Actions

Description

We have started 2 Flink applications producing data into pageview.v1 and pageview.trending.relative.v1 topics. These applications use webrequest_frontend_text as source and they are producing a lot of data, which might lead to issues in our infrastructure.

  • Refinery Hourly started failing with pageview.v1, we increased the Spark job and seems stable now.
  • After 1 day, pageview.v1 is already 4TB of data. It's supposed to have around 8% of webrequest_frontend_text, which is 52TB, but messages are bigger. It looks like it could reach 28 ~ 30TB in 7 days.
  • pageview.trending.relative.v1 reached 3TB in around 12hours. We stopped the application to prevent issues.

First question to answer would be, can we afford having 2 new topics with around 30TB each?

Some ideas that we could implement to reduce the data:

  • pageview.v1
    • Reduce retention? maybe 5 days is enough, or even less. The data is meant to be used by the relative trending pipeline. (1 day retention for now)
    • Reduce message size? We were using the schema used in batch, to be as close as possible to it, but we don't really use all that data.
    • Enable zstd compression? (Not doing it for now)
  • pageview.trending.relative.v1
    • Reduce Window sliding for now. Data is produced per Window, the Window slides every 5 minutes now. Moving to 10 minutes will reduce the data to half. (Not doing it, not needed)
    • Reduce retention (1 day for now)
    • Enable compaction. We only care about the last status of trending, and as each window produces data, a lot of data repeats the same key. Enabling compaction could reduce the data drastically (Compaction removed, it removes valuable data for the Datalake
    • Enable zstd compression. (Not doing it yet)
    • Avoid producing useless messages:
      • Many pages got less than 5 views per hour, maybe we could avoid producing them. (Dropping all pages with less than 50 hourly views for now. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157 )
      • We could consider producing only the change in trending. If a page is not trending, it won't produce new messages. This will reduce the data drastically, but it requires development, keeping more state in memory (possible issues in K8s) and won't allow analyze the data properly. (Not doing it)

About Hive ingestion.

  • Should we create a dedicated process of the large Spark job is enough? Solved with bigger spark jobs configured in the stream config.

Details

Other Assignee
brouberol
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
stream: pageview.trending.relative.v1JavierMontonoperations/mediawiki-configmaster+6 -0
stream: pageview-trending-relative + webrequest-page-viewJavierMontonoperations/deployment-chartsmaster+5 -2
stream: pageview-trending-relativeJavierMontonoperations/deployment-chartsmaster+15 -3
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Trending - Reduce data producedrepos/data-engineering/mediawiki-event-enrichment!157javiermontonfeature/reduce-data-producedmain
Pageview: Filter out null page_idsrepos/data-engineering/mediawiki-event-enrichment!156javiermontonfeature/filterout-null-page-idmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

JMonton-WMF created this task.Wed, Jul 15, 8:36 AM

I think adding cleanup.policy = compact,delete to pageview.trending.relative.v1 is a quick win. I think it will reduce the data stored drastically. Do you think we could do it @brouberol?

brouberol added a comment.Wed, Jul 15, 8:44 AM

First question to answer would be, can we afford having 2 new topics with around 30TB each?

I'm a "strong no" on this one. Not that we can't afford it, but I'd rather we do everything we can to avoid it.

I think adding cleanup.policy = compact,delete to pageview.trending.relative.v1 is a quick win

Yep, let's try this first

brouberol added a comment.Wed, Jul 15, 8:45 AM
brouberol@kafka-jumbo1010:~$ sudo -E kafka configs --alter --entity-type topics --entity-name eqiad.pageview.trending.relative.v1 --add-config cleanup.policy='[compact,delete]'
Root user detected, using the broker's super user auth config.
Completed updating config for topic eqiad.pageview.trending.relative.v1.
brouberol@kafka-jumbo1010:~$ kafka topics --describe --topic eqiad.pageview.trending.relative.v1
Topic: eqiad.pageview.trending.relative.v1	TopicId: IhgY5dT-SxaflaFuH9D0xw	PartitionCount: 3	ReplicationFactor: 3	Configs: min.insync.replicas=1,cleanup.policy=compact,delete,max.message.bytes=20971520,message.timestamp.type=CreateTime
	Topic: eqiad.pageview.trending.relative.v1	Partition: 0	Leader: 1015	Replicas: 1015,1012,1018	Isr: 1015,1012,1018
	Topic: eqiad.pageview.trending.relative.v1	Partition: 1	Leader: 1016	Replicas: 1016,1015,1018	Isr: 1016,1015,1018
	Topic: eqiad.pageview.trending.relative.v1	Partition: 2	Leader: 1017	Replicas: 1017,1016,1018	Isr: 1017,1016,1018
brouberol updated Other Assignee, added: brouberol.Wed, Jul 15, 8:46 AM
JMonton-WMF added a comment.Wed, Jul 15, 8:47 AM

Thanks! I'll start the pageview.trending.relative.v1 application now and monitorize it during the day. Let's see if it grows that fast or not. Meanwhile I'm looking at some of the other options to reduce the size.

JMonton-WMF added a comment.Wed, Jul 15, 8:52 AM

We could reduce the size of events on pageview.v1 by removing many fields. I don't think many of the fields are used for Trending right now, but pageview.v1 looks like a nice reusable stream, it could be used for other use cases, specially as it's using the same schema as the batch process.
Maybe @Ottomata you have some ideas about this, I'm not sure how we can measure the importance of the topic versus how much storage it uses.

For now, I'll remove the x_analytics field, is a big one and it's redundant, it's also parsed and stored.

JMonton-WMF updated the task description. (Show Details)Wed, Jul 15, 8:53 AM
JMonton-WMF updated the task description. (Show Details)Wed, Jul 15, 8:58 AM
JMonton-WMF added a comment.Wed, Jul 15, 9:23 AM

I believe zstd compression cannot be enabled if we don't upgrade the Gobblin Kafka client: https://gitlab.wikimedia.org/repos/data-engineering/gobblin-wmf/-/blob/main/pom.xml#L472-476

JMonton-WMF updated the task description. (Show Details)Wed, Jul 15, 11:12 AM
CodeReviewBot added a project: Patch-For-Review.Wed, Jul 15, 12:03 PM

javiermonton opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157

Trending - Reduce data produced

gerritbot added a comment.Wed, Jul 15, 12:11 PM

Change #1311043 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: pageview-trending-relative

https://gerrit.wikimedia.org/r/1311043

CodeReviewBot added a comment.Wed, Jul 15, 12:12 PM

javiermonton updated https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/156

Pageview: Filter out null page_ids

gerritbot added a comment.Wed, Jul 15, 2:12 PM

Change #1311043 merged by jenkins-bot:

[operations/deployment-charts@master] stream: pageview-trending-relative

https://gerrit.wikimedia.org/r/1311043

JMonton-WMF updated the task description. (Show Details)Wed, Jul 15, 3:06 PM

I changed a couple of configs. https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157

Now trending is not considered trending if there are less than 200 views on 1h. Pages with less than 50 views per hour are dropped completely. This is reducing the traffic by x100 or more. We should review this, but maybe this is a better approach, start working with low throughput, increase it eventually.

CodeReviewBot added a comment.Thu, Jul 16, 7:47 AM

javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/157

Trending - Reduce data produced

CodeReviewBot added a comment.Thu, Jul 16, 7:47 AM

javiermonton merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/156

Pageview: Filter out null page_ids

gerritbot added a comment.Thu, Jul 16, 10:22 AM

Change #1311418 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] stream: pageview-trending-relative + webrequest-page-view

https://gerrit.wikimedia.org/r/1311418

gerritbot added a comment.Thu, Jul 16, 5:58 PM

Change #1311418 merged by jenkins-bot:

[operations/deployment-charts@master] stream: pageview-trending-relative + webrequest-page-view

https://gerrit.wikimedia.org/r/1311418

JMonton-WMF moved this task from Next Up to In progress on the Data-Engineering (Q1 FS26/27 July 1st - September 30th) board.Fri, Jul 17, 1:43 PM
Ottomata added a comment.Mon, Jul 20, 5:57 PM

First question to answer would be, can we afford having 2 new topics with around 30TB each?

I'm a "strong no" on this one. Not that we can't afford it, but I'd rather we do everything we can to avoid it.

@brouberol I agree we should reduce storage if we can, but: kafka jumbo is supposed to be "jumbo". :)

I understand the current problems with hardware prices, but I think we should be willing to scale Kafka jumbo hardware if we need to. Do we need to here? Maybe not!

pageview.trending.relative.v1 should be manageable, and it looks like its size is being reduced.

pageview.v1 will always be quite large, and I don't think reducing message content is the best way to mitigate its size. A rich dataset is more useful than a sparse one!

But, we have a batch computed pageview dataset in the data lake. It might eventually be worth thinking if a pageview stream is useful. I think it is, but I am not sure.

Perhaps pageview events are more generally useful in a stream than webrequest is? If so, maybe we can reduce webrequest retention by a day or two, effectively reducing the total size taken up by 'webrequest' related data in Kafka?

JMonton-WMF added a comment.EditedMon, Aug 3, 2:58 PM

Compaction cleanup.policy=compact,delete was added to pageview.trending.relative.v1 but it is causing other issues. There is an alert GobblinKafkaRecordsExtractedNotEqualRecordsExpected that gets confused as the number of offsets doesn't match the number of records. We also think that enabling compaction removes some valuable data for the Datalake, so we can to disable compaction now that the topic is controlled.

There is also a possible issue in retention, I see it is 8640000 and I guess it was meant to be 86400000 (one more 0). It's 2,4h rather than 24h right now.

@brouberol, could you please change cleanup.policy to delete and retention.ms to 86400000 on `eqiad.pageview.trending.relative.v1? The topic is 6GB now, I guess it will grow to 60GB by the retention, and a maybe a few more GB by the compaction.

I can create a separate ticket if needed.

Changes needed:

  • eqiad.pageview.trending.relative.v1 -> clenaup.policy = delete, retention.ms=86400000 (expected 60GB storage after changes)
  • eqiad.pageview.v1 -> retention.ms=86400000 (expected ~4TB storage after changes)

We can consider not increasing that much the eqiad.pageview.v1 if needed

gerritbot added a comment.Tue, Aug 4, 8:43 AM

Change #1320757 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/mediawiki-config@master] stream: pageview.trending.relative.v1

https://gerrit.wikimedia.org/r/1320757

gerritbot added a comment.Tue, Aug 4, 1:48 PM

Change #1320757 merged by jenkins-bot:

[operations/mediawiki-config@master] stream: pageview.trending.relative.v1

https://gerrit.wikimedia.org/r/1320757

Stashbot mentioned this in T433816: UserInfoCard: Include target user name in server-side instrumentation.Tue, Aug 4, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-04T14:05:02Z] <mszwarc@deploy1003> Started scap sync-world: Backport for [[gerrit:1320757|stream: pageview.trending.relative.v1 (T432204)]], [[gerrit:1319086|WikimediaAntiAbuse: Document required load order after Echo (T432452)]], [[gerrit:1320781|UIC: Add user name to server-side instrumentation events (T433816)]], [[gerrit:1320782|UIC: Add user name to server-side instrumentation events (T433816)]]

Stashbot mentioned this in T432452: Echo notifications when new items are flagged.Tue, Aug 4, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-04T14:07:05Z] <mszwarc@deploy1003> javiermonton, mszwarc, mpostoronca: Backport for [[gerrit:1320757|stream: pageview.trending.relative.v1 (T432204)]], [[gerrit:1319086|WikimediaAntiAbuse: Document required load order after Echo (T432452)]], [[gerrit:1320781|UIC: Add user name to server-side instrumentation events (T433816)]], [[gerrit:1320782|UIC: Add user name to server-side instrumentation events (T433816)]] synced to the testser

Stashbot added a comment.Tue, Aug 4, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-04T14:13:00Z] <mszwarc@deploy1003> Finished scap sync-world: Backport for [[gerrit:1320757|stream: pageview.trending.relative.v1 (T432204)]], [[gerrit:1319086|WikimediaAntiAbuse: Document required load order after Echo (T432452)]], [[gerrit:1320781|UIC: Add user name to server-side instrumentation events (T433816)]], [[gerrit:1320782|UIC: Add user name to server-side instrumentation events (T433816)]] (duration: 07m 58s)

brouberol added a comment.Wed, Aug 5, 8:34 AM

All good for eqiad.pageview.trending.relative.v1

brouberol@kafka-jumbo1010:~$ kafka topics --describe --topic eqiad.pageview.trending.relative.v1
Topic: eqiad.pageview.trending.relative.v1	TopicId: IhgY5dT-SxaflaFuH9D0xw	PartitionCount: 3	ReplicationFactor: 3	Configs: min.insync.replicas=1,cleanup.policy=delete,retention.ms=86400000,max.message.bytes=20971520,message.timestamp.type=CreateTime
	Topic: eqiad.pageview.trending.relative.v1	Partition: 0	Leader: 1015	Replicas: 1015,1012,1018	Isr: 1012,1015,1018
	Topic: eqiad.pageview.trending.relative.v1	Partition: 1	Leader: 1016	Replicas: 1016,1015,1018	Isr: 1015,1016,1018
	Topic: eqiad.pageview.trending.relative.v1	Partition: 2	Leader: 1017	Replicas: 1017,1016,1018	Isr: 1016,1017,1018

All good for eqiad.pageview.v1

brouberol@kafka-jumbo1010:~$ kafka topics --describe --topic eqiad.pageview.v1
Topic: eqiad.pageview.v1	TopicId: w2dagFt3T-GK5iinVnhvOw	PartitionCount: 20	ReplicationFactor: 3	Configs: min.insync.replicas=1,retention.ms=86400000,max.message.bytes=20971520,message.timestamp.type=CreateTime
	Topic: eqiad.pageview.v1	Partition: 0	Leader: 1013	Replicas: 1013,1011,1018	Isr: 1011,1013,1018
	Topic: eqiad.pageview.v1	Partition: 1	Leader: 1011	Replicas: 1011,1014,1018	Isr: 1011,1014,1018
	...
	Topic: eqiad.pageview.v1	Partition: 18	Leader: 1013	Replicas: 1013,1018,1010	Isr: 1010,1013,1018
	Topic: eqiad.pageview.v1	Partition: 19	Leader: 1011	Replicas: 1011,1013,1018	Isr: 1011,1013,1018
JMonton-WMF updated the task description. (Show Details)Wed, Aug 5, 9:13 AM
JMonton-WMF moved this task from In progress to In Review on the Data-Engineering (Q1 FS26/27 July 1st - September 30th) board.Wed, Aug 5, 9:53 AM
JMonton-WMF moved this task from In Review to Done on the Data-Engineering (Q1 FS26/27 July 1st - September 30th) board.Fri, Aug 7, 11:12 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits