Search index not updating on en.wikipedia.org
Closed, ResolvedPublic

Description

Author: SpontaneousGrumbler

Description:
Symptoms similar to bugzilla 46530. Search index build last completed about 5–6 January 2014. To demonstrate, do a Special:search on ~2014, notice that the search indicates the page was updated 5 January 2014. Pull up the page's revision history and see that many updates have occurred on 6-7-8-9-10-11-12 January 2014.


Version: wmf-deployment
Severity: major

bzimport set Reference to bz59979.
bzimport created this task.Via LegacyJan 12 2014, 7:38 PM
demon added a comment.Via ConduitJan 13 2014, 5:27 AM

Confirmed. Probably unrelated to bug 46530 though.

Thgoiter added a comment.Via ConduitJan 13 2014, 8:20 AM

Same for German Wikipedia.

Manybubbles added a comment.Via ConduitJan 13 2014, 3:17 PM

Hint:
2014-01-11 18:45:05,115 [main] WARN org.wikimedia.lsearch.util.Localization - Error processing message file at file:///a/search/conf/messages/MessagesNg.php

Manybubbles added a comment.Via ConduitJan 13 2014, 3:37 PM

Looks like the PHPParser in lsearchd doesn't use the java open, try, do, finally, close idiom so errors in reading the localization will leak file handles. Yick. That might be the cause of the occasional too many open files errors we see. Probably not this but I saw it while investigating so I'm recording it just in case.

Manybubbles added a comment.Via ConduitJan 13 2014, 7:01 PM

I'm really not sure what else to do here. I don't know what "working" should look like unfortunately. I see that most of the "update" lucene files have timestamps around 2014-1-7 6am UTC. I don't know if that is expected for having data from the 5th or not. I don't have logs from before those localization errors started but a quick perusal of the code tells me they _shouldn't_ be causing this error.

Oh, disks aren't full.

Manybubbles added a comment.Via ConduitJan 13 2014, 7:24 PM

it looks like things aren't moving from the updated state to the ready for use state. The "updated" indexes are all new enough.

Manybubbles added a comment.Via ConduitJan 13 2014, 7:53 PM

I _think_ the incremental updater stopped when the machine was rebooted seven days ago. I don't have sudo so I can't fix it but I've made my case to folks that can.

Manybubbles added a comment.Via ConduitJan 13 2014, 11:42 PM

We've find the errant job and restarted it so I'm marking this as resolved. If everything doesn't start showing up to date then we'll take another look at it.

Thgoiter removed a subscriber: Thgoiter.Via WebNov 30 2014, 3:14 PM

Add Comment

Column Prototype
This is a very early prototype of a persistent column. It is not expected to work yet, and leaving it open will activate other new features which will break things. Press "\" (backslash) on your keyboard to close it now.