Author: bugs
Description:
CentralNotice is no longer working (displays "<centralnotice-template-test_notice>" instead of the notice). It would be helpful if this could be fixed as soon as possible (we need it for Wikimania). Adding a bug here to document the issue and then hopefully we can have the solution listed here when the issue comes up again.
Version: unspecified
Severity: enhancement
URL: http://test.wikipedia.org/w/index.php?title=Main_Page&action=purge
Fixed the initial problem of the cron not running due to permission issues. The secondary issue is why the cron retrieves a failed message from the mediawiki store, while running the script by hand produces the correct output.
Interestingly enough this bug is only triggered when a large amount of messages (10+) are being retrieved for a template.
For instance,
test/aa
test/ab
test/ace
test/af
test/ak
test/aln
test/als
test/am
test/an
test/ang
all generate correctly while anything after will fail. But run those templates that fail by hand one by one and everything works out just fine.
Have we imposed any new restrictions on the throughput of message pulls? Time for me to find out.
Quite possibly.
I'm also seeing this bug in the all language view for Special:CentralNotice making me suspect less that the rebuild is at fault and that something internal could be malfunctioning or being used improperly.
Brion suggests looking at MessageCaching.php or Language.php to further diag.
"Out of memory" as such should just cause a failure; but I do recall seeing some weird stuff before with cached language objects getting corrupted (in that case when having the destructor run during parser test runs, but the emptied-out object retained in a cache array and returned by a later fetch). Could be something similar.
There is defiantly something funny going on at the caching layer. $this->mMemc->add() within MessageCache is failing when it pulls any message past test/ang. This failure causes mDisable to be set and each subsequent call for messages will fail outright.
The memcache add is currently failing with "SERVER_ERROR out of memory" errors" .
Sounds like it's trying to set a value that is larger than memcache's compiled maximum, which is most likely 1MB after gzip compression.
Actually it seems to have been a problem with the memcached instance in question, on srv63. I restarted it.
Cached message are now getting properly set. I've re-enabled the cron for all messages and have it set to generated at 30min intervals. I'll be keeping an eye out on the memcached node in question.