Page MenuHomePhabricator

QueryService missing data for snarc-llgc: 9-12 August '23
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

Here is an example of an item created which does not show up in queries - https://snarc-llgc.wikibase.cloud/wiki/Item:Q106004

This seems to have effected several large batches of id's added to items. In the following example the edits made on 11 and 12 August are not returned in queries. With one batch of around 20k Ids added, only 2k show up in query results. - https://snarc-llgc.wikibase.cloud/w/index.php?title=Item:Q63206&action=history

This problem seems to have effected 10,000's edits between abt 9th and 12th of August. The items show up in search but not query.

What happens?:

Data added between 9-12 August is not showing up in the QueryService but exists in the Wikibase.

What should have happened instead?:

Data should be query able through the QueryService as usual.
It's working for the dates before and after those reported, we expect the same behaviour.

Event Timeline

Evelien_WMDE renamed this task from QueryService missing data 9-12 August '23 to QueryService missing data for a Wiki: 9-12 August '23.Aug 14 2023, 10:16 AM

Another batch of 12k statements uploaded on the 20/21 of September also do not appear in the query service, bar about 150. Ive noticed that once items have been edited manually, they soon appear in the query service. This query should return the 12k statements uploaded on the 20/21 - https://tinyurl.com/2dhyn6hd

For more help with the investigation, Telegram reports:

September 18:

"Morning. We've noticed some indexing (we think) issues that I wonder if others are experiencing. Specifically, date statements added using quickstatements (some in July, some this month) that aren't appearing in sparql queries: e.g. P15 here https://beyond-notability.wikibase.cloud/w/index.php?title=Item:Q1030&action=history but appearing here https://tinyurl.com/ymwcfjda I know there was lots of issues with search during the summer, so perhaps this is related? And is there a way to trigger a fresh reindexing of a wikibase hosted by cloud?

I'm having the same problem. My issues is not with dates, and the qualifiers in question were uploaded via OpenRefine.
Here's an example of the problem. These two queries that differ only on line 16. The first uses P9 as a qualifier, and the second uses P34. Almost all the records in question have both P9 and P34 qualifiers, so the number of results should be the same. But the first returns 283 results, while the second returns only 81
Query 1: https://tinyurl.com/27okahnl
Query 2: https://tinyurl.com/27mm73uv
An additional clue: almost all of the results in query 2 start with A. This makes me think that the indexing stopped partway through a bulk upload I was doing from OpenRefine, which was proceeding alphabetically.

September 21:
I had this issue of data not showing up in the query service back in August (still not resolved) and thought it was isolated but I added a batch of 12k statements last night with QS and only 138 show up in query this morning. I hope a fix is possible soon as my grant funding is dependant on a working database!

Tarrow renamed this task from QueryService missing data for a Wiki: 9-12 August '23 to QueryService missing data for snarc-llgc: 9-12 August '23.Feb 23 2024, 5:16 PM
Tarrow claimed this task.
Tarrow subscribed.

Hi @Jason.nlw!

Could you take a little look and see how the queryservice is looking for you now? Over the last months we've made a number of improvements to the reliability of the queryservice and also rebuild all wikibases queryservices.

I'm gonna now close this ticket as resolved but if you have any issues please don't hesitate to reopen it. From my rough testing I now see 12k-ish responses for the query you were worried about.

Cheers!

Hi @Tarrow

All my missing items for the date range stated seem to show up in the QS now and I recently did a fresh upload of about 10k items which all worked as expected so it seems the problem is fixed! Thanks so much!

Jason