Mentioned in SAL (#wikimedia-analytics) [2020-05-27T13:42:02Z] <ottomata> increased Kafka topic retention in jumbo-eqiad to 31 days for (eqiad|codfw).mediawiki.revision-create - T253753

Did 31 days:

$ kafka topics --alter --topic eqiad.mediawiki.revision-create --config retention.ms=2678400000
$ kafka topics --alter --topic codfw.mediawiki.revision-create --config retention.ms=2678400000

$ kafka topics --describe --topic '^(eqiad|codfw).mediawiki.revision-create$'
Topic:codfw.mediawiki.revision-create	PartitionCount:1	ReplicationFactor:3	Configs:retention.ms=2678400000
	Topic: codfw.mediawiki.revision-create	Partition: 0	Leader: 1002	Replicas: 1002,1003,1004	Isr: 1004,1002,1003
Topic:eqiad.mediawiki.revision-create	PartitionCount:1	ReplicationFactor:3	Configs:retention.ms=2678400000
	Topic: eqiad.mediawiki.revision-create	Partition: 0	Leader: 1003	Replicas: 1003,1002,1005	Isr: 1002,1005,1003

An idea: How about sending back to kafka the update stream and make THAT one retention higher?
Moving retention to 30 days for revision-create will make a lot of data stay that wouldn't be necessary (about half of the data), while keeping only the updates should be enough.
Just an idea :)

@JAllemandou I think that is an option as well, the thing is that is it is transitional to help to bootstrap a test of the full pipeline. In the end we won't be using jumbo and thus won't be able to rely on a 30days retention on main so hopefully we'll be able to reset the retention back to 7days once we're done with the test.
To circumvent this particular problem (time to make the dumps available > retention) we could either:

• send back the events that matter back to kafka and have higher retention like you suggest
• create a dedicated job running on the analytics network to read the events stored in HDFS and figure out a way to make the resulting data available in kafka main

Given that retention is not on puppet is this a setting that is communicated to a new node when it joins the cluster by the leader of the partition or similar?