In WDQSv1, data preparation ETL and database indexing are disjoint processes.
The ETL runs on Hadoop (orchestrated via Airflow), and indexing is executed (independently) on each single WDQS/Blazegraph node. During indexing, the Blazegraph nodes needs to be depooled for a few days.
Long (3+ days) and unstable Blazegraph indexing run-times, and gating steps for move data from HDFS to WDQS v1 hosts (which require root privileges for tansfer) make it impraticaly to programmatically re-index the dabase on a fixed scedule.
One consequence of this is that we can't rely on weekly RDF exports for reconciliation. Blazegraph can get out of sync if MediaWiki updates never make it to the streaming updater pipeline (see T407702).
WDQS v2 architecture
Our V2 architecture should support a way of augmenting the streaming realtime updater with a periodic bulk sync so that any missed events are eventually reflected in WDQS.
The reindexing pipeline largely builds on the existing Blazegraph/WDQSv1 ETL process for normalization and graph splitting, but has been re-architected around new API boundaries for generation and ingestion of dumps, orchestration of the indexing process, and distribution of indexed data (binary qlever index files) to WDQSv2 service hosts.
Dumps, temp munged data, and index files are persisted on S3. The flow is roughly the following:
- We schedule an ephemeral RDF content export ("dump") POD in the dumps-legacy k8s namespace. This gives us control over the TTL export schedule and allows for gating data quality checks.
- Spark jobs kick in to "munge" the TTL export and generate N3 files for both wikidata_main and scholarly_articles graphs.
- A qlever-index process is scheduled in k8s (airflow-wikidata namespace) that builds indexes for the subgraphs and uploads them to an S3 bucket that WDQS backend hosts can reload from.
- Data retention (for temp data) is enforced using the drop_data_daily dag airflow pattern.
Currently this approach mixes YARN/Hadoop and k8s for compute, and S3 and HDFS for storage. Data is stored on HDFS for offline analytics, but S3 is the main API boundary for the WDQS v2 components across compute infrastructures.