JobQueue has 2 methods of dedupication: individual job deduplication and root-job deduplication. Here’s how both work:
- Individual job deduplication: each job has an sha1 associated with it and a boolean flag whether to deduplicate. On each job push it checks with Redis whether a job with the same identifier was already pushed, if yes - it rejects the job.
- Root job deduplication. Each job has identifiers of it root job - the one that’s created it, and a timestamp of the root job. After a newer root job is pushed, the latest timestamp is updated in Redis. When the leaf job is pulled, it’s root job timestamp is checked against latest and it’s potentially deduplicated.
We can create an absolutely matching algorithm in ChangeProp by adding some new fields to our events.
This is only possible after we add some persistency discussed in T157089