Change Details

We've observed a pretty linear increase between the number of wikis handled by the dumps v1 DAG and the scheduler loop duration (mainly spent checking task dependencies). We could experiment with having a DAG per wiki first letter. This way, each DAG would have about 1000 tasks, instead of a single DAG with ~30000 tasks. As we're limited by the number of pool slots and overall parallelism, it might actually be lest costly to shard tasks to more DAGs? {F59041617} //A dumps DAG run with 16 wikis (left) and 32 wikis (right)//

We've observed a pretty linear increase between the number of wikis handled by the dumps v1 DAG and the scheduler loop duration (mainly spent checking task dependencies). We could experiment with having a DAG per wiki first letter, or simply a maximum of X wikis per. DAG. This way, each DAG would have about 1000 tasks, instead of a single DAG with ~30000 tasks. As we're limited by the number of pool slots and overall parallelism, it might actually be lest costly to shard tasks to more DAGs? {F59041617} //A dumps DAG run with 16 wikis (left) and 32 wikis (right)//

We've observed a pretty linear increase between the number of wikis handled by the dumps v1 DAG and the scheduler loop duration (mainly spent checking task dependencies). We could experiment with having a DAG per wiki first letter, or simply a maximum of X wikis per. DAG. This way, each DAG would have about 1000 tasks, instead of a single DAG with ~30000 tasks. As we're limited by the number of pool slots and overall parallelism, it might actually be lest costly to shard tasks to more DAGs? {F59041617} //A dumps DAG run with 16 wikis (left) and 32 wikis (right)//