Ignore my last note on this. That was a problem in my query. I hadn't realized I didn't have all the alt labels in the data yet.

Thanks to the team for all the work on this issue. I ended up changing some things on the items I had posted about here (different property for the URL associated with USGS staff). The problems I noted no longer seem to be an issue as those entities/claims/triples were all updated. I am, however, seeing a problem in a different area this morning. I was doing some work to resolve names of organizational units to identifiers instantiated in the Wikibase instance. I'm not turning up entities in that query that should be there. Here's a query:

Thank you for providing all the detail on this process. It's helpful to see that you're working through the issue and that there's a process for syncing everything across the WB environment. For me as a user, it's not so much that there are hiccups along the way. I expect that. Working on a shared platform like WB.c, though, it would be really useful to gain some visibility into logging to see where I might have issues if I've got unexpected query results. Seeing stuff in tickets is fine, or you might think about what aspects of the logs could be opened up for access by WB instance operators.

I've been able to dump a subset of items from geokb.wikibase.cloud as NT files, index them using Qlever's indexing engine (Docker on a cloud-based Ubuntu machine), spin up the Qlever SPARQL back-end on that index, and then query it using the Qlever UI (on the same machine). Both the back-end and UI are proxied via Nginx, and it seems to be working quite well. I don't yet know what my resource load is going to look like and what memory I'll need to allocate, but I noticed the Qlever instance of U. Frieburg going down a fair bit after they publicized it at the Wikidata modeling conference.

I have my use case now. After seeing Hannah Bast and Johannes Kalmbach present on Qlever today, I went and spun up an instance of the back-end indexer/SPARQL service and the UI tool on a cloud VM. It's so cool when software actually works like it's supposed to! The approach to indexing and query federation there is fantastic, and I think I'll be able to jettison some of what I reorganized from sources that already conform to RDF fairly well and keep our wb.c instance focused on messier data. The Qlever SPARQL performance is truly outstanding compared to anything else I've tried.

Here's my vote for moving this issue up the queue if at all possible. The problem with the Blazegraph store not being synced or whatever it is that's going on is really hurting productivity on wikibase.cloud. We rely on SPARQL for just about everything. Every time I push a bunch of edits like adding thousands of claims with new identifier references to another source (something I just did), I have a situation where only some of the data winds up being returned via SPARQL while I can go to the UI or API and see that the information made it into the core data store. It's quite frustrating.

I'd be interested to learn more about the use case be for the full TTL encoded dump. Maybe there's a cool usage pattern I could use. It seems like the SPARQL query service can be used to generate a full graph, and it supports offset/pagination to handle scale. Trying to work against a massive TTL encoding means a lot of data being loaded into memory somewhere just to operate on. And the SPARQL/Blazegraph capability does also have its own issues with scalability, especially on queries requiring a filter approach.

In a lot of ways, I'd be completely fine sticking with the Wikidata standard prefixes pointed at a specific wb.c instance's domain/paths. Or perhaps a slight tweak from "wd" to "wb". The big conceptual thing is distinguishing between terms/concepts that are a part of the Wikibase and larger frameworks (owl, rdfs, prov, etc.) and then the specific knowledge graph being queried. All Wikibase SPARQL queries share common characteristics because of the statement/reference/qualifier dynamic, and many of the examples online (and generative via LLMs) are going to refer to Wikidata, so it's reasonable to keep it s short step between those examples and what the query would look like for a specific Wikibase instance.

I still have information that is not being retrieved via SPARQL queries when I can see it visibly on items in the geokb.wikibase.cloud instance. Here's a query:

Interestingly enough, I've been posting some one-off things to Wikidata as part of some research into Cherokee Nation Tribal leaders today. SPARQL queries there are not immediately showing results of adding things like start time/end time qualifiers on claims. Here's the query showing missing information I'm attempting to contribute. Maybe it's an inherent delay into Blazegraph?

One other thing I tried doing is replacing and then recreating a P114 claim. If I simply use wikibaseintegrator with action_if_exists=REPLACE_ALL, this doesn't have any effect. However, if I first remove the P114 claim entirely and then run another operation to add it back in using wikibaseintegrator, a query on the ID turns up the expected result. This one, P114:70235876, was missing before I did that.

Thanks for the input. I probably provided too much detail. That's a useful query, but it wasn't the issue. Bottom line is that I have items that are not showing up in SPARQL queries that absolutely should be showing up based on how they look like they are structured visibly in the UI and how they are returned via the wikimedia API. Here is a single concrete example:

It does appear that certain triples may have been lost in the process of ingesting to Blazegraph. For a query like the following where I'm trying to get all of the items that have a particular instance of classification, I'm missing a bunch of items that are actually classified that way.

I thought this was something different at first, but perhaps it is related. I noticed I had at least one item where the "what links here" page is showing things linked but without labels.

I do have other things missing in SPARQL queries now. The following is supposed to pull about 83K items (a whole tranche of publication items I just brought in):

Hi @Evelien_WMDE. Thank you for connecting on this. I really appreciate what WMDE is working to build out with this infrastructure.

@Evelien_WMDE Yes. The GeoKB instance does seem to be working better now. Elasticsearch responses seem to be what I would expect, and SPARQL queries are returning what they should. Thank you! I was definitely having some performance problems there for a few days. I've continued to do work on aspects of the model since @Fring indicated things should be "safe," but I'm happy to have it humming along again!

Hi @Fring. Any progress on the SQL replica problems or other related issues? I'm definitely seeing some performance problems in the instance in work I'm trying to do today with a new class if entities I'm working through. Perhaps I need to just cease working with this until the problem is resolved?

Here's some visible behavior in the system arising from the problem. If you check out the WhatLinksHere for one of our science centers, you'll see a listing of people affiliated with that organization. Some of the items in the list show up without labels like this one. Those same entities are not returned in a SPARQL search such as trying to retrieve everyone employed by our organization.

Glad to see this working its way toward some kind of resolution. In the meantime, would someone mind kicking off a job to complete jobs for the geokb.wikibase.cloud instance? I've got some things backed up that are causing problems in SPARQL responses.

@Fring - Fantastic! Thank you! I see they cleared out pretty quick, and search is all working as it should.

Well, I apologize, but I've once again started to clog up the system with edits. If someone can kick off some job processing for geokb.wikibase.cloud again, that would be great.

Well, I managed to get a whole backlog of jobs going again in another wikibase.cloud instance. This one is showing over 50K jobs.

Following up on my previous comment, I tried this type of write/read/write approach in the latest work for the https://eew-edgi.wikibase.cloud Wikibase instance. I needed to work through all the versions of the North American Industry Classification System (NAICS), creating items for the different industries to be linked to and support analysis of regulated facilities. The bot for this using WikibaseIntegrator works through the 6 different years/editions of the NAICS to develop logical items and their ExternalIDs. I first ran through and created the individual 1981 items and then worked over the items again to add in hierarchical relationships through "has part" and "part of" claims. This involved reading each item serially to add in additional claims, with the idea this might trigger something to complete the jobs.

To try and answer this bit you can get a count (but you won't know exactly what they are) of jobs backed up from the api. Have a look at https://eew-edgi.wikibase.cloud/w/api.php?action=query&meta=siteinfo&siprop=statistics for example. Under the jobs key you should see the count of jobs. Hope that helps as a very short term signal!

I'm curious about this statement from the original post:

Thank you very much for filling in a few blanks for me. Search in the eew-edgi instance is turning up expected results now. I did also subscribe to that issue.

I'm continuing to see some issues that appear to be related to incomplete jobs somewhere in the WBStack pipeline. It would be good to understand where these are and if there is work going on somewhere to resolve them. It would at least be good to know if there is any way to get visibility on when something is stuck re. the comments from @Tarrow on jobs in a queue somewhere.

I appreciate all the work digging into this issue.

My guess is this is a case similar to what I'd seen earlier where an update job somewhere is clogged or backed up. I just added a new property and a couple of new items in the GUI to set up some further work pulling in additional items. Here's the query I use to pull my classification scheme:

I'm seeing similar issues in work I'm doing at eew-edgi.wikibase.cloud. In my case, it's frequent 504 errors thrown. I've been trying a number of things to better handle large build-out scenarios where I need to populate thousands of reference items such as U.S. states, counties, and cities. Parallel processing using something like Dask just seems to make matters worse. I continue to see problems I reported in a previous issue (https://phabricator.wikimedia.org/T330796) where full availability of an item's functionality is sometimes severely delayed. I see an item in recent changes and can get to it by QID, but I can't turn the item up in UI search or SPARQL queries.

This is continuing to be an issue for me on another use case I am working through. I'm building out information associated with the Facility Registration System operated by the US Environmental Protection Agency. This information is incompletely and inadequately organized into Wikidata, so I am working on methods to process source data and organize into items. I've started on the two classification systems in use which are also incompletely referenced in Wikidata.

Thank you for taking a look at this issue. A bunch of items that I pushed yesterday evening seem to be coming up in search just fine as well. I assume that regular processing for the search index should happen automatically and be close to real time. I built infrastructure years ago between MongoDB and Elasticsearch where we had to deal with scaling issues of keeping an index up to date in real time with the data store. I haven't dug enough into your architecture here yet to understand it, but I can imagine it's a challenge.