Author: SpontaneousGrumbler
Description:
Symptoms similar to bugzilla 46530. Search index build last completed about 5–6 January 2014. To demonstrate, do a Special:search on ~2014, notice that the search indicates the page was updated 5 January 2014. Pull up the page's revision history and see that many updates have occurred on 6-7-8-9-10-11-12 January 2014.
Version: wmf-deployment
Severity: major
Hint:
2014-01-11 18:45:05,115 [main] WARN org.wikimedia.lsearch.util.Localization - Error processing message file at file:///a/search/conf/messages/MessagesNg.php
Looks like the PHPParser in lsearchd doesn't use the java open, try, do, finally, close idiom so errors in reading the localization will leak file handles. Yick. That might be the cause of the occasional too many open files errors we see. Probably not this but I saw it while investigating so I'm recording it just in case.
I'm really not sure what else to do here. I don't know what "working" should look like unfortunately. I see that most of the "update" lucene files have timestamps around 2014-1-7 6am UTC. I don't know if that is expected for having data from the 5th or not. I don't have logs from before those localization errors started but a quick perusal of the code tells me they _shouldn't_ be causing this error.
Oh, disks aren't full.
it looks like things aren't moving from the updated state to the ready for use state. The "updated" indexes are all new enough.
I _think_ the incremental updater stopped when the machine was rebooted seven days ago. I don't have sudo so I can't fix it but I've made my case to folks that can.
We've find the errant job and restarted it so I'm marking this as resolved. If everything doesn't start showing up to date then we'll take another look at it.