Page MenuHomePhabricator

Determine and control cirrus streaming updater's usage of MWAPI resources
Closed, InvalidPublic3 Estimated Story Points

Description

Per today's meeting, @dcausse , @EBernhardson , @pfischer and myself discussed the resource usage of the new cirrus streaming updater. The updater will consume MWAPI, so it's important we determine the extent of cirrus streaming updater's usage of MWAPI under various scenarios (spikes, day-to-day, backfill, etc). T346456 introduced some config options for rdf-streaming-updater that should apply here as well.

Creating this ticket to:

  • Estimate approximate MWAPI usage
  • Explore/document ways to throttle from the flink/cirrus streaming updater side
  • Discuss the above with ServiceOps and adjust according to their resource constraints.

Event Timeline

bking updated the task description. (Show Details)
Gehel set the point value for this task to 3.Nov 6 2023, 4:43 PM

It's hard to resource usage estimate until we've done more testing. Consensus seems to be that MWAPI usage will not be a problem, but we can always re-open later if problems surface.