Page MenuHomePhabricator

Improve concurrency limits configuration of the wdqs updater
Closed, ResolvedPublic3 Estimated Story Points

Description

The WDQS updater have several config options to reduce the concurrency at which it calls the MW api.

The config option wikibase_repo_thread_pool_size controls the size of the thread pool running HTTP requests.

During a test of this application with zookeeper and the flink-k8s-operator we had to backfill around 2weeks of updates and this caused a massive load on the mw-api-int cluster.

We lowered this value from 30 to 5 expecting to see a 1/6 fold reduction but this reduction was nowhere near what was expected, we barely saw the impact suggesting that the current limits are already too high and the system is limited by the endpoint capacity not by itself.

Looking a the code this limits is imposed on the HTTP thread pool that is attached to a job task, given that we run at a parallelism of 12 this means that the actual number of concurrent requests is parallelism * wikibase_repo_thread_pool_size.

So we went from 30*12=360 to 5*12=60.

We should definitely change how this is configured to take the flink parallelism into account.

AC:

  • the updater should have a single option to control the MW requests concurrency
  • we should probably not run the AsyncOp over all the 12 tasks

Event Timeline

EBernhardson set the point value for this task to 3.Sep 18 2023, 3:48 PM

Change 960097 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Rework how concurrency limits are configured

https://gerrit.wikimedia.org/r/960097

Change 960097 abandoned by DCausse:

[wikidata/query/rdf@master] [WIP] Rework how concurrency limits are configured

Reason:

will restore if actually needed

https://gerrit.wikimedia.org/r/960097

Change 960554 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/rdf@master] Add mediawiki_max_concurrent_requests

https://gerrit.wikimedia.org/r/960554

Change 961020 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/alerts@master] search: simplify flink parallelism configuration

https://gerrit.wikimedia.org/r/961020

Change 961024 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: simplify parallelism configuration

https://gerrit.wikimedia.org/r/961024

Change 960554 merged by jenkins-bot:

[wikidata/query/rdf@master] Add mediawiki_max_concurrent_requests

https://gerrit.wikimedia.org/r/960554

Change 961138 had a related patch set uploaded (by DCausse; author: DCausse):

[wikidata/query/deploy@master] flink: simplify parallelism setup

https://gerrit.wikimedia.org/r/961138

Change 963105 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] rdf-streaming-updater: simplify parallelism and use newer kafka APIs

https://gerrit.wikimedia.org/r/963105

Change 963105 merged by Bking:

[operations/deployment-charts@master] rdf-streaming-updater: simplify parallelism and use newer kafka APIs

https://gerrit.wikimedia.org/r/963105

Change 961020 merged by Bking:

[operations/alerts@master] search: simplify flink parallelism configuration

https://gerrit.wikimedia.org/r/961020

Change 961024 abandoned by DCausse:

[operations/deployment-charts@master] rdf-streaming-updater: simplify parallelism configuration

Reason:

superseded by Id9a9ec3873f1eebfdc53a97218cef417dcc2e8bd

https://gerrit.wikimedia.org/r/961024

Change 961138 abandoned by DCausse:

[wikidata/query/deploy@master] flink: simplify parallelism setup

Reason:

hopefully won't be needed as we move away from this deployment process

https://gerrit.wikimedia.org/r/961138