Page MenuHomePhabricator

Elasticsearch not updating
Closed, ResolvedPublicBUG REPORT

Description

Our recently migrated wikibase https://beyond-notability.wikibase.cloud/wiki/Main_Page is one of those with Elasticsearch turned on. However, we've noticed that recent additions (e.g. https://beyond-notability.wikibase.cloud/wiki/Item:Q1153) are not being indexed - that is, whilst we can search for pre-migration items without using exact Q numbers, we can't do the same for new additions.

Looking at that email "Important information about your Wikibase.cloud migration" sent June 10, this error is not explicitly described as part of the 'degraded experience' that was expected for some users.

I appreciate wikibase cloud are expereince Elasticsearch issues, but it seemed worth logging this issue.

Thanks!

Event Timeline

Hello @Drjwbaker there is currently no dedicated job-runner for wikibase.cloud and there could therefore be a delay in how fast the items are updated/added to elasticsearch

Looking at https://beyond-notability.wikibase.cloud/w/api.php?action=wbsearchentities&search=miss%20florence&format=json&errorformat=plaintext&language=en&uselang=en&type=item I can see this is now indexed.

Jobs are currently executed on requests, and you can keep an eye on the number of jobs in the queue by checking https://beyond-notability.wikibase.cloud/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=json

Hi @toan: that is super useful to know. Thank you. Happy to close this ticket now that I have a sense of how ES is updating.

I think I might have hit the same (or at least similar) bug: since pm 4 days any changes in Ps and Qs (including labels, descriptions, aliases) I make via the GUI on https://kbtestwikibase.wikibase.cloud are not when running SPARQL queries.

For instance, when I run https://tinyurl.com/23hvllyx, multiple results I removed 4 days ago via the GUI (such as https://kbtestwikibase.wikibase.cloud/wiki/Property:P58 , or P51) are still shown in that results set, while not showing up anymore in the list of Properties in my Wikibase GUI: https://kbtestwikibase.wikibase.cloud/wiki/Special:AllPages?from=&to=&namespace=122

Before the weekend (24 June) the SPARQL seemed to work normally, ie any changes I made in Q-en P-items showed up within a minuute in the SPARQL results.

This Wikibase was part of the "B batch" from the wbstack migration

Yeah, I'm seeing the same @OlafJanssen: statements added on 23 June are not appearing in SPARQL queries: e.g. this revision https://beyond-notability.wikibase.cloud/w/index.php?title=Item:Q709&oldid=8117 should show on this query https://tinyurl.com/23bu8awx but doesn't.

I think I might have hit the same (or at least similar) bug: since pm 4 days any changes in Ps and Qs (including labels, descriptions, aliases) I make via the GUI on https://kbtestwikibase.wikibase.cloud are not when running SPARQL queries.

For instance, when I run https://tinyurl.com/23hvllyx, multiple results I removed 4 days ago via the GUI (such as https://kbtestwikibase.wikibase.cloud/wiki/Property:P58 , or P51) are still shown in that results set, while not showing up anymore in the list of Properties in my Wikibase GUI: https://kbtestwikibase.wikibase.cloud/wiki/Special:AllPages?from=&to=&namespace=122

Before the weekend (24 June) the SPARQL seemed to work normally, ie any changes I made in Q-en P-items showed up within a minuute in the SPARQL results.

This Wikibase was part of the "B batch" from the wbstack migration

I'm having the exact same issue as that this query should be returning around 20 Q items but returns nothing: https://tinyurl.com/22j36cg4
despite many items having matching triplets: https://mcje-data.wikibase.cloud/wiki/Item:Q382

also searching for "Copper Block" in the wiki search bar doesn't work despite this item having that label: https://mcje-data.wikibase.cloud/wiki/Item:Q1182

any data that's after that June 24th/June 23rd mark hasn't properly updated at all

I am wondering why this is moved to "Review". Isn't it already applied?

I am wondering why this is moved to "Review". Isn't it already applied?

Correct it was some confusion since this was sort of re-used for the queryservice issues. Will close this ticket.

I think I might have hit the same (or at least similar) bug: since pm 4 days any changes in Ps and Qs (including labels, descriptions, aliases) I make via the GUI on https://kbtestwikibase.wikibase.cloud are not when running SPARQL queries.

For instance, when I run https://tinyurl.com/23hvllyx, multiple results I removed 4 days ago via the GUI (such as https://kbtestwikibase.wikibase.cloud/wiki/Property:P58 , or P51) are still shown in that results set, while not showing up anymore in the list of Properties in my Wikibase GUI: https://kbtestwikibase.wikibase.cloud/wiki/Special:AllPages?from=&to=&namespace=122

Before the weekend (24 June) the SPARQL seemed to work normally, ie any changes I made in Q-en P-items showed up within a minuute in the SPARQL results.

This Wikibase was part of the "B batch" from the wbstack migration

I'm having the exact same issue as that this query should be returning around 20 Q items but returns nothing: https://tinyurl.com/22j36cg4
despite many items having matching triplets: https://mcje-data.wikibase.cloud/wiki/Item:Q382

also searching for "Copper Block" in the wiki search bar doesn't work despite this item having that label: https://mcje-data.wikibase.cloud/wiki/Item:Q1182

any data that's after that June 24th/June 23rd mark hasn't properly updated at all

@OlafJanssen

Thank you for reporting the issues. We had a deployment last week that caused the queryservice-updater to stop updating T311495: Wikibase.cloud instance does not update triple store when after edits in the wiki. This has been resolved and the failed batches have been re-run.

As for ElasticSearch not updating your changes on time we currently have no jobrunner for wbstack to support all the jobs CirrusSearch creates, this will probably be a requirement for the future but the whole ElasticSearch topic is still up for debate.

I tried searching for the terms you mentioned and they seem to have been indexed now, there are some explanations higher up in this ticket that could help you understand this current limitation.

Yes, I noticed yesterday evening that it seems to work now in my wb.cloud instance, eg https://tinyurl.com/23hvllyx,

@Evelien_WMDE: The project tag got archived and this open task has no other active project tags. Could you please either add an active project tag so this task can be found, or update the task status? Thanks a lot!