Update: I've indexed a shard of the import_wikidata_ttl DAG's output and put it in our S3 bucket, along with a tiny metadata file. This was a manual process; I'm working on a script. We also would like S3 access from the WDQS nodes to make this easy to do before the DAG setup is ready: T427319

An index and metadata file with files & timestamp now exists in the S3 bucket, and latest.json points to it.

I generated this by manually performing each step. There should be a script too but it seemed like a good idea to do the steps manually first, to validate it works and to learn about issues that might come up at each step.

The process was as follows for the main graph (scholarly would be the same):

1. hadoop distcp the import_ttl output (1024 gzipped n-triples files) from HDFS to the S3 bucket
2. from a WDQS node, s3cmd get the files so they are available locally
3. Move the files into a input/ dir and gunzip them
4. Create a data/ dir with a settings file for the indexer as described in the README here
5. Fetch the docker image of the qlever indexer (if needed)
6. Run the indexer as described in that README, with an override: ulimit -n 1048576 (necessary to prevent a too-many-files-open crash halfway through the indexing job)
7. Find the oldest timestamp represented in the dump by going back to the n-triples files and grepping for dateModified lines (I wrote a little python script, but nothing fancy is happening here)
8. Create the manifest file wikidata_dump_meta.json with that timestamp and the list of created index files
9. s3cmd sync the contents of the data/ directory, including the manifest, to the S3 bucket under some prefix (I used 20260525 because that was the date of the dump). Use --no-check-md5 to skip expensive/slow md5 fingerprinting of the files, since we're just using sync as a quick way to handle the whole directory. (Might also work to do put --recursive: check on this.)
10. create a new latest.json pointing to that prefix and s3cmd put it in the S3 bucket

The data flow is: HDFS >> S3 >> WDQS node >> S3. Note that this is temporary: in the future the data movement will be orchestrated by an Airflow task, which is the "right" way to do this kind of thing. We got temporary permissions to interact with S3 from the WDQS nodes to facilitate this method of index-building for now.

Many of the steps are quite long-running. Even the ones that are copying to/from S3 or gunzipping take a while (~an hour or more) and the index takes a nontrivial amount of time too. @trueg and I discussed an idea to speed up the indexer by giving it all the files at once vs concatenating them, so I'll experiment about that (in another task).

I'm going to mark this as closed since the index exists there and the steps are clear, even if they were manually performed.