Page MenuHomePhabricator

Use RichAsyncFunction for GenerateEntityDiffPatchOperation
Closed, ResolvedPublic5 Estimated Story Points

Description

Doing blocking calls in a classic operation is causing nasty side-effects, the most annoying one is that the checkpoint barriers have to wait behing the blocking calls.
RichAsyncFunction has been added to flink to circumvent this, it allows flink to save the state of current pending blocking calls while still allowing the checkpoint barriers to flow to the next operator.

AC:

  • Checkpoints no longer times out with default settings when backfilling
  • The order of input events must be kept

Event Timeline

dcausse created this task.Jul 23 2020, 9:25 AM
Restricted Application added a project: Wikidata. · View Herald TranscriptJul 23 2020, 9:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse claimed this task.Aug 3 2020, 8:05 AM
dcausse triaged this task as Medium priority.
dcausse set the point value for this task to 5.
dcausse moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
dcausse moved this task from Scaling to Current work on the Wikidata-Query-Service board.

Change 618321 had a related patch set uploaded (by DCausse; owner: DCausse):
[wikidata/query/rdf@master] Switch GenerateEntityDiffPatchOperation to RichAsyncFunction

https://gerrit.wikimedia.org/r/618321

Change 618321 merged by jenkins-bot:
[wikidata/query/rdf@master] Switch GenerateEntityDiffPatchOperation to RichAsyncFunction

https://gerrit.wikimedia.org/r/618321

Gehel closed this task as Resolved.Sep 1 2020, 12:25 PM