Page MenuHomePhabricator

Increase retention to 4 days for structured-data.articles.v1 topic
Open, LowPublic5 Estimated Story Points

Description

In order to allow the client to catch up after a disconnect, we need to retain the events for a minimum of 4 days.

To do

  • add retention to "aws.structured-data.articles.v1" topic in msk.tf
{
  "config" : {
     "cleanup.policy" : "delete",
      "retention.ms" : "345,600,000 "    // 4 days
   }
}
  • After 4 days of deployment, go on kafka-ui dev. From this topic, pick few events with partition, offset and ROWTIME.
  • Run the wme client parallel-realtime example using offsets and then since_per_partition. Measure how long approx. it takes to fetch the first message for the partition for which we set offset/since_per_partition. Do this for few messages.

Acceptance criteria
Able to receive messages from partition with offset and since_per_partition as in the last item above.

Related Objects

StatusSubtypeAssignedTask
InvalidNone
OpenNone

Event Timeline

@prabhat when you are back from PTO, please let me know if this should roll over to the next sprint (50) or if we should include it in the current one.