Page MenuHomePhabricator

Search results not updating on the English Wikipedia
Closed, ResolvedPublic

Event Timeline

MZMcBride raised the priority of this task from to Needs Triage.
MZMcBride updated the task description. (Show Details)
MZMcBride subscribed.

The updating of the index has resumed, but articles that were updated during the outage will have to be null-edited or something in order to be shown properly in the search results. Is there a procedure for finding and fixing them?

Indexing was delayed? Hmmm, that's not good.

Do we have a period of time in which the updates were missed? We can force a reindex on that period of time from RecentChanges.

demon set Security to None.
demon moved this task from Out of scope work to Backlog on the Wikimedia-Search board.
demon added a subscriber: Manybubbles.

@Joe says we had a Redis outage that would have caused jobs to get dropped onto the floor. I'm going to kick off Cirrus's reindexing script for the duration of the outage.

@Joe, do you have a link to any incident documentation? You said the outage started around 9:30 PM UTC yesterday. When did it end?

Kicked off the rebuilds for all of yesterday.

Kicked off rebuilds for the past three days and that seems to have got everything. I wish I knew what time periods were effected but gratuitously reindexing everything got the job done. enwiki seems all better now. I've got the rest of the wikis reindex scripts running in the screen session on terbium. They'll likely be done sometime tomorrow. Instructions for how to recover are here: https://wikitech.wikimedia.org/wiki/Search#Recovering_from_an_Elasticsearch_outage.2Finterruption_in_updates

Manybubbles claimed this task.

The script finished running last night some time. Its probably a low priority, but we could make that faster by running more than on wiki at a time. Also, wikidata get lots and lots of updates.