This is our baseline of running dumps for 32 wikis with max_active_tasks=16: it takes about 1h20min to process through 99% of the tasks.

FWIW, the dump process puts a bit of CPU pressure on the k8s masters, but nothing dramatic.

We're interested in the ability of increasing our scheduling throughput, so the first step is to increase max_active_tasks from 16 to 32. This might collide with the scheduler.max_tis_per_query=16 airflow config, especially as we increase the wikis count. We'll see. Same remark for kubernetes_executor.worker_pods_creation_batch_size: 16.

To prepare for 32 concurrent tasks executions, I'm going to also increase the mediawiki-dumps-legacy-regular pool slot count to 32.

I'm seeing a lot of

Forbidden: exceeded quota: quota-compute-resources, requested: limits.memory=4896Mi, used: limits.memory=150676Mi, limited: limits.memory=150Gi"

If we want to run at least 32 parallel dump jobs, we need to either reduce the ns memory limit, or increase the quota.

Looking at the following graph, we seem to be requesting a lot of CPUs.

And ideed, for each dump task, we actually trigger 2 pods:

• the KubernetesPodOperator, in charge of creating and managing the actual dump pod
• the actual dump pod

The first one is created with the same resource as any other pod task, ie

worker:
  resources:
    requests:
      cpu: 1000m
      memory: 1500Mi
    limits:
      cpu: 2000m
      memory: 3Gi

which is waaay too much for following kubernetes events and sending an API request.

Looking at the pod details dashboard, we see that the pod barely uses any CPU at all, and about 300MB of memory.

We need to adjust these request/limits if we want to make a better use of the available resources.

Change #1136706 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: ensure the pod running in the KubernetesPodOperator itself gets low resources

https://gerrit.wikimedia.org/r/1136706

Change #1136706 merged by Brouberol:

[operations/deployment-charts@master] airflow: ensure the pod running in the KubernetesPodOperator itself gets low resources

https://gerrit.wikimedia.org/r/1136706

Change #1136726 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: hotfix: only assign low resources to kubernetes pod operator pods

https://gerrit.wikimedia.org/r/1136726

Change #1136726 merged by Brouberol:

[operations/deployment-charts@master] airflow: hotfix: only assign low resources to kubernetes pod operator pods

https://gerrit.wikimedia.org/r/1136726

I've run a dump of 64 wikis, with 32 concurrent tasks. I aborted it as I was seeing many memory limits quota errors, as show on the following graph:

We'll probably need to increase the memory limit to 200GB if we want to be able to run 32 parallel dumps.

By the way, our effort to reduce resource requests/limits cluster-wide were successful! By downsizing 2 flink apps from 20 to 2 (which should have been applied months ago but was not), as well as reducing the amount of resources allotted to the canary events, we've been able to significantly increase available resources: {F59288535}

We can get to 32 parallel dump tasks by setting kubernetes_executor.worker_pods_creation_batch_size: 32 and max_active_tasks: 32 on the DAG, as well as 32 pool slots.

We've been able to dump 64 wikis while running 32 concurrent jobs and pretty much saturating the associated pool, in about 3h.

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1255

test_k8s/dumpsv1: increase max_active_tasks to 32 for regular dags

Change #1138739 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-test-k8s: allow 32 pods to be created in a single executor batch

https://gerrit.wikimedia.org/r/1138739

Thanks to the new quotas, we never even approached resource quota saturation.

Change #1138739 merged by Brouberol:

[operations/deployment-charts@master] airflow-test-k8s: allow 32 pods to be created in a single executor batch

https://gerrit.wikimedia.org/r/1138739

We've been able to increase parallelism to 32, and be pretty efficient at saturating the pool. I'm considering this task done.