Assuming it will take some time to develop the proper plumbing between the Wikibase Cloud frontend and dumpRdf.php, I wonder if an acceptable workaround would be for me to go to Special:EntityData for every single property and item on my Wikibase, in TTL format, concatenate that into a single file, and then import it like a Wikidata dump.

I have my use case now. After seeing Hannah Bast and Johannes Kalmbach present on Qlever today, I went and spun up an instance of the back-end indexer/SPARQL service and the UI tool on a cloud VM. It's so cool when software actually works like it's supposed to! The approach to indexing and query federation there is fantastic, and I think I'll be able to jettison some of what I reorganized from sources that already conform to RDF fairly well and keep our wb.c instance focused on messier data. The Qlever SPARQL performance is truly outstanding compared to anything else I've tried.

I'd actually love to see us replace Blazegraph in WBStack with Qlever, but that's another discussion. In the meantime, we do need a way to a) get a Wikibase dump in RDF and b) some reasonable means of tapping an update stream. Once baselined, a combination of an API call for changes and serial/parallel retrieval of changed items in TTL encoding would likely work, but it will be highly inefficient. I'm a huge fan of cutting out or at least minimizing HTTP transfer of data wherever possible. That's a massive bottleneck in any process. I can't tell exactly where wikibase.cloud sits with a little trace routing, but the ideal situation would be some dump process running locally to the wb.c infrastructure that drops files on somebody's cloud for access. We might then replicate files elsewhere for practical use, but if we can shove them onto a cloud storage infrastructure somewhere for direct access, then we can deploy capabilities like Qlever onto that same infrastructure.

A full dump of geokb.wikibase.cloud to Wikibase's dump format, including full history, took almost a week to run with two hiccups. That's about a half million edits. That's fine for a process that maintains full history and something we will employ as a future-proofing strategy. But we need current state RDF dump capability for wikibase.cloud instances that gives us close to real time state to be spun up in any other infrastructure as needed.

I've been able to dump a subset of items from geokb.wikibase.cloud as NT files, index them using Qlever's indexing engine (Docker on a cloud-based Ubuntu machine), spin up the Qlever SPARQL back-end on that index, and then query it using the Qlever UI (on the same machine). Both the back-end and UI are proxied via Nginx, and it seems to be working quite well. I don't yet know what my resource load is going to look like and what memory I'll need to allocate, but I noticed the Qlever instance of U. Frieburg going down a fair bit after they publicized it at the Wikidata modeling conference.

I wrote a Python script that runs a query for what I want and then hits /Special:EntityData/<qid>.nt to retrieve and cache individual files for indexing. Blazegraph's query service is supposed to respond with n-triples or turtle, but I've not been able to get that to work. I can only get the standard XML response back no matter whether I use the format parameter or an accept header. I found the NT format to work with Qlever so far, and I've had some trouble with TTL I've not worked out yet.

Even running individual item pulls in parallel takes a bunch of time, and this seems a super inefficient and duplicative method of pulling data. I'd imagine there's got to be a better way to go about it.

The good news is, I think I can offload some of what I had been working to reorganize into Wikibase's specific knowledge representation to other collections of RDF data federated through Qlever indexing to give me a ton of reference sources I need. For instance, I also spun up a process to build an index from select DOI metadata from both Crossref and DataCite, using their native support for RDF from their APIs. That gives me the basis for linking on DOIs where I have other content associated with those. I should also be able to leverage the sameAs relationships I've built in claims to Wikidata entities, either federating with the University of Freiburg instance of the full Wikidata store or building a selective index on my instance.

I realize there would be a ton of engineering involved in retuning WBStack to use Qlever in place of Blazegraph, but I really like how it is operating thus far. It sounds like the current infrastructure for wikibase.cloud uses a message queue process on changes with multiple subscribers, including whatever is writing to Blazegraph. If that could be tapped to write triples to another store, that could be very cool!

Desires related to T312131.

Just adding my specific use case for my instance here; the reason I think this would be useful for me specifically is so that my instance can be made into an ontology posted quarterly or yearly to GitHub as a TTL file. This is because my instance is, in many ways, a successor to an ontology I created for my thesis that is continually updated. Being able to pipeline this out in a semi-automated manner 1-4 times a year would be ideal, since BioPortal, Ontobee, etc. automatically take and curate from GitHub.

The only alternative solution right now is to use dumpgenerator (https://github.com/mediawiki-client-tools/mediawiki-dump-generator) to download the XML (which takes many hours), then I have to reupload the dump into a local Wikibase instance, then use the local maintenance scripts to download the RDF, which is then uploaded to GitHub.

If the dumps were made available at request, with some capping to prevent downloads of too much frequency, then I could simply download and automate the GitHub portion of the pipeline, which would be significantly less overhead.