Page MenuHomePhabricator

Increased delay in indexing of new Items on Wikidata
Open, HighPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Create a new Item
  • Wait until it shows up in search results

What happens?:

  • The Item shows up after 5 to 6 minutes according to some reports, which is slower than it has been in the past.

What should have happened instead?:

  • The new Item should be indexed more quickly. This is especially important for common workflows of people creating Items and then searching for it when using them in a new statement

Other information (browser name/version, screenshots, etc.):

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel triaged this task as High priority.Mon, May 27, 12:39 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.

The additional delay is coming from a new implementation of the search update pipeline, which has a deduplication window. In principles, search is an asynchronous system and should not be used in a synchronous workflow. Our informal and undocumented (yet) SLO is that updates should be visible in 10 minutes.

It might be time to review how we use search in the edit workflow and find alternatives.

This is hugely disrupting my workflow. What could the alternatives be, do an API call on (X) new items created by the user?

I'll let @Lydia_Pintscher give more context on the future plans and workarounds. We will be working on a prioritization of Wikidata updates in the search pipeline, but it will take some time. In the meantime, it should be possible to refer to items by their QID.

We've discussed how to go about this problem some more. Here is the option we have come up with so far to address the problem of using the asynchronous service for synchronous work: We could have some form of local storage for the user that contains the entities they recently created. Those would be used for suggestions when making new statements.
Additional thoughts:

  • One interesting step in this direction is allowing to create a new Item directly from the statement editing UI. @Celenduin has created a user script to explore how this part could look like and the details are in T107693.
  • People have said that it would not just be interesting to use this local storage for the recently created entity of the user but any they have recently used. This would make some edits easier where you repeatedly use the same entity while editing many entities.