Change Details

The dumps v1 tasks can put strain on the databases they query, as well as on the ES (external storage) servers. Running too many dumps tasks in parallel can have a dramatic effect on production if we overload these ES databases, as they store the actual wiki contents for all wikis. We need to perform some gradual load tests to assess what would be good parallelism settings for the dumps pools. Reminder: we're planning to setup 2 pools. One for the misc wiki dumps and one for the large wiki dumps. This way, we can adjust parallelism on the fly, as well as ensure that the large wiki dumps start as soon as possible.

The dumps v1 tasks can put strain on the databases they query, as well as on the ES (external storage) servers. Running too many dumps tasks in parallel can have a dramatic effect on production if we overload these ES databases, as they store the actual wiki contents for all wikis. We need to perform some gradual load tests to assess what would be good parallelism settings for the dumps pools. (cc @Ladsgroup ) Reminder: we're planning to setup 2 pools. One for the misc wiki dumps and one for the large wiki dumps. This way, we can adjust parallelism on the fly, as well as ensure that the large wiki dumps start as soon as possible.

The dumps v1 tasks can put strain on the databases they query, as well as on the ES (external storage) servers. Running too many dumps tasks in parallel can have a dramatic effect on production if we overload these ES databases, as they store the actual wiki contents for all wikis. We need to perform some gradual load tests to assess what would be good parallelism settings for the dumps pools. (cc @Ladsgroup ) Reminder: we're planning to setup 2 pools. One for the misc wiki dumps and one for the large wiki dumps. This way, we can adjust parallelism on the fly, as well as ensure that the large wiki dumps start as soon as possible.