Update on what I've learned so far. The tasks are numbered as follows: 1) Dump, 2) Quality check, 3) Spark processing, 4) Index, 5) Orchestrate reload

To share data among tasks in a DAG, one needs a persistent volume claim (PVC), which can be backed by CephFS or some other kind of storage.

Here are the things we need, according to my current understanding:

• New CephFS-backed PVC in the "wikidata" airflow instance Here is an example of a PVC being created in the "sre" airflow instance. We will need one of these in our "wikidata" helm chart. It will hold the output of task 3, the spark data-processing jobs.

• Access to a PVC in the mediawiki-dumps-legacy namespace We also need a place to hold the wikibase dump in task 1.Since the dump needs to run in a different K8s namespace (the one that already does all the dumps), I hope that we can merely reuse the existing PVC that exists there. Otherwise we will also need a new PVC in mediawiki-dumps-legacy. I have a question out to @BTullis about this now.

• An S3 bucket for wikidata Finally, we will need an S3 bucket to write the output of task 1 to (the point of this is to allow it to be accessed by tasks 2 and 3, which run in our own namespace). We file a ticket to got this set up.

I filed T425973 for the S3 bucket bullet above.

Update:

• we likely will not need our own PVC in our namespace. We can probably use S3 for all cross-task storage in our own DAG tasks.
• We have an S3 bucket!
• @BTullis will create a PVC for us in the mediawiki-dumps-legacy to use for the dump

We have a PVC now in mediawiki-dumps-legacy. It is called wdqs-update-pipeline
I have successfully interacted with this PVC in a test airflow instance (created a directory and performed a wikibase lexemes dump)

I also manually created a test S3 bucket using the creds and s3cmd and wrote a file and read it.

Because we have an S3 bucket and a PVC, and we've interacted with both successfully, I consider this task done.