Exception caught inside exception handler
Closed, ResolvedPublic

Subscribers
Tokens
"The World Burns" token, awarded by Sylvain_WMFr."The World Burns" token, awarded by Steinsplitter."The World Burns" token, awarded by Thibaut120094."The World Burns" token, awarded by Vituzzu.
Assigned To
Authored By
Vituzzu, Oct 26 2015

Description

Any page request turns into:
MediaWiki internal error.

Exception caught inside exception handler.

Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.

Vituzzu created this task.Oct 26 2015, 3:15 PM
Vituzzu updated the task description. (Show Details)
Vituzzu raised the priority of this task from to Unbreak Now!.
Vituzzu added a subscriber: Vituzzu.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 26 2015, 3:15 PM

Probably caused by a SWAT change.

Caused by https://gerrit.wikimedia.org/r/248877 rollback in progress

MWException from line 331 of /srv/mediawiki/php-1.27.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader duplicate registration error. Another module has already been registered as schema.Search
Taketa added a subscriber: Taketa.Oct 26 2015, 3:21 PM
Sylvain_WMFr added a subscriber: Sylvain_WMFr.
Stryn added a subscriber: Stryn.Oct 26 2015, 3:22 PM
Taketa removed a subscriber: Taketa.Oct 26 2015, 3:23 PM
Elitre added a subscriber: Elitre.Oct 26 2015, 3:31 PM
Risker added a subscriber: Risker.Oct 26 2015, 4:07 PM

I'm not that familiar with the deploy process, but maybe the deployment script should have some sort of smoke test built in (e.g. Deploy to test.wikipedia.org, make sure its not fataling, before deploying to rest of cluster)

I think the incident report should have some sort of actionables on time to revert. 9 minutes is a very long time. Do our docs need to be modified about how to revert multiple patchsets quickly? Do we need a new script for returning the wikis to a previous known good state? Or something else?

Scap has some smoke testing and automated rollback functionality. It's not currently tailored to work with MediaWiki deployments but it is on our radar, specifically T114313: [EPIC] Migrate the MW weekly train deploy to scap3.

demon added a subscriber: demon.Oct 26 2015, 8:44 PM

@mmodell beat me to everything I was going to say. I'll just throw these two at you for reference: T113073: Canary deploy for scap3 and T109514: Scap3 should support rollback as well.

TTO added a subscriber: TTO.Oct 27 2015, 12:33 AM

See T112845 for improving that somewhat cryptic error message

Deskana closed this task as Resolved.Oct 27 2015, 8:44 AM

This can be considered resolved, since the offending patches were reverted.