Last week in the collab we decided on a path structure for the data reload pipeline in the S3 bucket, s3://wdqs-update-pipeline.

The relevant directories in the bucket that we decided on are:

• /wikidata-rdf-export to hold the dumps being written by the Airflow DAG (needs to be updated - currently the dumps go to /wikibase-dumps)
• /temp/ which holds its own subdir /munged-n3 to hold the NT files output by the data processing stage of the pipeline (and might hold other data that only needs to live for the lifetime of the DAG)
• /qlever-index to hold the final indices (further split out by graph and date/timestamp) and metadata files read by the deployment

To prevent further contamination of the bucket, the data_reload DAG needs to be updated to write dumps to the new path.

AC:

• Everything else has been deleted from the bucket
• data_reload DAG has been updated to write the export files to the new directory, /wikidata-rdf-export