My guess is this is a case similar to what I'd seen earlier where an update job somewhere is clogged or backed up. I just added a new property and a couple of new items in the GUI to set up some further work pulling in additional items. Here's the query I use to pull my classification scheme:

https://eew-edgi.wikibase.cloud/query/#PREFIX%20wdt%3A%20%3Chttps%3A%2F%2Feew-edgi.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fclass%20%3FclassLabel%20%0AWHERE%20%7B%0A%20%20%3Fclass%20wdt%3AP2%20%3Fsubclass.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20.%20%7D%0A%7D%0A

It does not return this new item as a subclass like it should.

This is kind of a problem in workflow as I'm not really able to complete work without hardcoding a bunch of values, and if items aren't being fully "built" within the Wikibase structure, I don't really have a functional system.

There are currently +70,000 jobs in the pipeline for eew-edgi.wikibase.cloud - I triggered a job to process them in the next few hours.

We did a little research because we expect that this may actually be due to some queryservice updater batch not having succeeded.

For a future paper trail:

// wiki id for this wiki is 461

// check count of not-done and done queryservice batches
>>> QsBatch::where('wiki_id', 461)->where('done', 0)->count()
=> 0
>>> QsBatch::where('wiki_id', 461)->where('done', 1)->count()
=> 3721

Given that we are unsure which batch failed I had set them all going again:

>>> QsBatch::where('wiki_id', 461)->update(['done' => 0]);
[!] Aliasing 'QsBatch' to 'App\QsBatch' for this Tinker session.
=> 3721

This will take quite some time to work through (these older job will be done will lower priority than newer edits). Let's check back tomorrow to see how it went.

It went well; could you take a look and see if it feels like everything that should be present is actually there?

Hi @Skybristol , this issue should be fixed now, can you confirm? Thanks!

I appreciate all the work digging into this issue.

SPARQL queries do seem to be returning everything they should at this point. However, searches through the UI now seem to be a problem.

For instance, I can return all U.S. county items with this query:

https://eew-edgi.wikibase.cloud/query/#PREFIX%20wd%3A%20%3Chttps%3A%2F%2Feew-edgi.wikibase.cloud%2Fentity%2F%3E%0APREFIX%20wdt%3A%20%3Chttps%3A%2F%2Feew-edgi.wikibase.cloud%2Fprop%2Fdirect%2F%3E%0A%0ASELECT%20%3Fitem%20%3FitemLabel%20%0AWHERE%20%7B%0A%20%20%3Fitem%20wdt%3AP1%20wd%3AQ655%20.%0A%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20.%20%7D%0A%7D%0A

But this API query should have turned up one of those items but does not:

https://eew-edgi.wikibase.cloud/w/index.php?search=adams&title=Special%3ASearch&fulltext=1&ns0=1&ns120=1

Type ahead search also does not seem to be returning any results for items that were created some time ago and used to come up. This applies everywhere type-ahead is invoked like creating a new statement and trying to find a property.

I also notice that mouseover on the item list in recent changes shows the "There was an issue displaying this preview" message.

Is that now an Elasticsearch issue rather than WQDS?

I'm continuing to see some issues that appear to be related to incomplete jobs somewhere in the WBStack pipeline. It would be good to understand where these are and if there is work going on somewhere to resolve them. It would at least be good to know if there is any way to get visibility on when something is stuck re. the comments from @Tarrow on jobs in a queue somewhere.

Here's a query for items I just added yesterday describing several programs administered by the US Environmental Protection Agency. The SPARQL query is returning them as a result, but a UI/API query for a term like "toxics" is not returning an item like this one that it should be. This is not as debilitating as not having records returned in a SPARQL query, but it indicates a lag or problem in the overall platform that is at least a pesky slow down.

I'm getting ready to start organizing data on all of the facilities regulated by the USEPA under these programs in order to integrate other information about the facilities from other sources (e.g., parent companies, historic changes in ownership, connections to impacted resources). Wherever possible, I'm establishing connections to Wikidata entities and leveraging property definitions aligned with Wikidata properties. This is important in a two-way relationship as there are cases where someone has developed really important connections in related Wikidata entities like Legal Entity Identifiers for companies. My hope is to develop adjacent knowledge graphs that serve specific analytical and reporting purposes but also inform the global knowledge commons. I see wikibase.cloud as the perfect opportunity to do that, but I do need to understand how the system is working and where we may run into limitations or issues to be dealt with.

Hi @Skybristol , thanks for bringing this to our attention.
The issue most recently reported in your latest comment is a result of our ongoing challenges with ElasticSearch, where outages cause a backlog of jobs to pile up and not properly cleared. We manually cleared it now and it seems to be working on our end, please let us know if you notice anything off.

For the latter part of your comment: We're looking into finding a sustainable fix for this, and are not sure if we can 100% guarantee it being fixed without a very significant overhaul of the system. Instead, our current plan is to introduce a workaround that allows for these backlogs to clear faster, so that when an issue occurs, it's quickly resolved and not noticed on your end. If you'd like to stay updated on this, please subscribe to https://phabricator.wikimedia.org/T330389
As long as we're in Beta, we expect to guarantee that the platform is up and running and all features are working properly, albeit slow and with hiccups now and then.

Hi @Evelien_WMDE,

Thank you very much for filling in a few blanks for me. Search in the eew-edgi instance is turning up expected results now. I did also subscribe to that issue.

Having built my own infrastructure around Elasticsearch, I totally get some of the scaling challenges you all are facing. I'm very aware that this is a beta system and don't mind the occasional interruptions at all. Your WBStack is still working better and more reliably than a separate Wikibase instance I've been struggling to get fully operational on AWS in another project. I am also very excited about the potential of this platform and what you all are working to achieve, and I'm anxious to help in whatever ways I can.

Cheers!

Hi Skybristol!

It would be good to understand where these are and if there is work going on somewhere to resolve them. It would at least be good to know if there is any way to get visibility on when something is stuck re. the comments from @Tarrow on jobs in a queue somewhere.

To try and answer this bit you can get a count (but you won't know exactly what they are) of jobs backed up from the api. Have a look at https://eew-edgi.wikibase.cloud/w/api.php?action=query&meta=siteinfo&siprop=statistics for example. Under the jobs key you should see the count of jobs. Hope that helps as a very short term signal!

Cheers :)

To try and answer this bit you can get a count (but you won't know exactly what they are) of jobs backed up from the api. Have a look at https://eew-edgi.wikibase.cloud/w/api.php?action=query&meta=siteinfo&siprop=statistics for example. Under the jobs key you should see the count of jobs. Hope that helps as a very short term signal!

Sweet! Thank you!