Page MenuHomePhabricator
Create Task
Maniphest T91399

MediaWiki job failing with "Exception thrown with an uncommited database transaction"
Closed, ResolvedPublic
Actions

Description

Starting today, all mediawiki-extension QUnit builds fail with the same error:

11:24:29 >> Message: SyntaxError: Parse error
11:24:29 >> Actual: null
11:24:29 >> Expected: undefined

Examples:
https://integration.wikimedia.org/ci/job/mwext-ContentTranslation-qunit/1443/consoleFull
https://integration.wikimedia.org/ci/job/mwext-UploadWizard-qunit/918/consoleFull
https://integration.wikimedia.org/ci/job/mwext-MobileFrontend-qunit-mobile/10263/consoleFull
https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/14129/consoleFull
https://integration.wikimedia.org/ci/job/mwext-Flow-qunit/4420/consoleFull

CCed some people involved in the changesets I've found which are currently affected, I figured you'd be interested in knowing when this is fixed.

Related Objects

Event Timeline

• Gilles created this task.Mar 3 2015, 3:01 PM
• Gilles raised the priority of this task from to Unbreak Now!.
• Gilles updated the task description. (Show Details)
• Gilles added a project: Quality-Assurance.
• Gilles added subscribers: • Gilles, Amire80, Krinkle and 3 others.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2015, 3:01 PM
• Gilles added a comment.Mar 3 2015, 3:09 PM

I couldn't reproduce the issue locally

• Gilles added a subscriber: hashar.Mar 3 2015, 3:11 PM
• Gilles added a project: Jenkins.Mar 3 2015, 3:24 PM
Krinkle renamed this task from QUnit extension tests failing to MediaWiki job failing with "Exception thrown with an uncommited database transaction".Mar 3 2015, 3:38 PM
Krinkle set Security to None.
Krinkle added a comment.EditedMar 3 2015, 3:41 PM

Two issues.

First. There is a regression with database locking (see T89180). Either a new query got added somewhere in core, or it's gotten stricter in what it tolerates. All mentioned jobs have the same error in their log.

[SQLBagOStuff] SqlBagOStuff: connecting to [unknown]
DatabaseBase::query: Writes done: PRAGMA case_sensitive_like = N
SQL ERROR: database is locked
MessageBlobStore::insertMessageBlob failed to update DB: exception 'DBQueryError' with message 'A database error has occurred. Did you forget to run maintenance/update.php after upgrading?  See: https://www.mediawiki.org/wiki/Manual:Upgrading#Run_the_update_script
Query: INSERT OR IGNORE INTO msg_resource (mr_lang,mr_resource,mr_blob,mr_timestamp) VALUES ('en','ext.eventLogging.subscriber','{}','20150303112411')
Function: MessageBlobStore::insertMessageBlob/single-row
Error: 5 database is locked
' in /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/Database.php:1253
Stack trace:
#0 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/Database.php(1209): DatabaseBase->reportQueryError('database is loc...', 5, 'INSERT OR IGNOR...', 'MessageBlobStor...', false)
#1 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/Database.php(2094): DatabaseBase->query('INSERT OR IGNOR...', 'MessageBlobStor...')
#2 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/DatabaseSqlite.php(584): DatabaseBase->insert('msg_resource', Array, 'MessageBlobStor...', Array)
#3 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/MessageBlobStore.php(105): DatabaseSqlite->insert('msg_resource', Array, 'MessageBlobStor...', Array)
#4 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/MessageBlobStore.php(68): MessageBlobStore->insertMessageBlob('ext.eventLoggin...', Object(ResourceLoaderFileModule), 'en')
#5 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/resourceloader/ResourceLoader.php(875): MessageBlobStore->get(Object(ResourceLoader), Array, 'en')
#6 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/resourceloader/ResourceLoader.php(620): ResourceLoader->makeModuleResponse(Object(ResourceLoaderContext), Array, Array)
#7 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/load.php(44): ResourceLoader->respond(Object(ResourceLoaderContext))
#8 {main}
SQL ERROR (ignored): database is locked
..
[Bug56269] Exception thrown with an uncommited database transaction: [05c2b168] /jenkins-mwext-ContentTranslation-qunit-1443/load.php?debug=false&lang=en&modules=ext.eventLogging.subscriber%7Cext.uls.pt%7Cjquery.suggestions%7Cmediawiki.hidpi%2CsearchSuggest&skin=fallback&version=20150303T112403Z&*   DBUnexpectedError from line 3669 of /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/Database.php: Attempted to commit transaction while atomic sections are still open
#0 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/LoadBalancer.php(979): DatabaseBase->commit('LoadBalancer::c...', 'flush')
#1 [internal function]: LoadBalancer->commitMasterChanges()
#2 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/LBFactory.php(176): call_user_func_array(Array, Array)
#3 [internal function]: {closure}(Object(LoadBalancer), 'commitMasterCha...', Array)
#4 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/LBFactory.php(321): call_user_func_array(Object(Closure), Array)
#5 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/LBFactory.php(177): LBFactorySimple->forEachLB(Object(Closure), Array)
#6 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/LBFactory.php(184): LBFactory->forEachLBCallMethod('commitMasterCha...')
#7 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/db/LBFactory.php(336): LBFactory->commitMasterChanges()
#8 /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/load.php(50): LBFactorySimple->shutdown()
#9 {main}

Second. It fails to produce an error page because it tries to parse a message but has to title to callback to when coming in load.php, which (contrary to api.php) has no $wgTitle.

[GlobalTitleFail] RequestContext::getTitle called by ContextSource::msg/call_user_func_array/RequestContext::msg/Message::setContext/RequestContext::getTitle with no title set.
[GlobalTitleFail] RequestContext::getTitle called by OutputPage::prepareErrorPage/OutputPage::setPageTitle/OutputPage::setHTMLTitle/Message::setContext/RequestContext::getTitle with no title set.
Unstubbing $wgParser on call of $wgParser::firstCallInit from MessageCache::getParser
Parser: using preprocessor: Preprocessor_DOM
[GlobalTitleFail] RequestContext::getTitle called by Hooks::run/call_user_func_array/ContentTranslationHooks::addModules/ContextSource::getTitle/RequestContext::getTitle with no title set.
User::getBlockedStatus: checking...
[GlobalTitleFail] RequestContext::getTitle called by Skin::initPage/Skin::preloadExistence/Skin::getRelevantTitle/ContextSource::getTitle/RequestContext::getTitle with no title set.
..
2015-03-03 11:24:11 integration-slave1003 my_wiki: [2f642a10] /jenkins-mwext-ContentTranslation-qunit-1443/load.php?debug=false&lang=en&modules=ext.eventLogging.subscriber%7Cext.uls.pt%7Cjquery.suggestions%7Cmediawiki.hidpi%2CsearchSuggest&skin=fallback&version=20150303T112403Z&*   ErrorException from line 257 of /mnt/jenkins-workspace/workspace/mwext-ContentTranslation-qunit/src/includes/exception/MWExceptionHandler.php: Fatal Error: Call to a member function isSpecialPage() on a non-object
#0 [internal function]: MWExceptionHandler::handleFatalError()
#1 {main}
Krinkle edited projects, added Continuous-Integration-Infrastructure; removed Jenkins, Quality-Assurance.Mar 3 2015, 3:46 PM

(Not related to the Jenkins software or its configuration, not related to QA/browsertests)

Jdforrester-WMF subscribed.Mar 3 2015, 3:50 PM
Krinkle closed this task as Resolved.Mar 3 2015, 4:03 PM
Krinkle claimed this task.
Krinkle added a subscriber: • demon.

@chad suspected 80a3a30d6a90 as possible cause.

I manually triggered a build of VisualEditor-qunit using https://gerrit.wikimedia.org/r/#/c/194125/ (a revert) instead of master of mediawiki-core and it made the error go away:

https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/14142/console

Krinkle updated the task description. (Show Details)Mar 3 2015, 4:07 PM
Krinkle added a project: MediaWiki-libs-Rdbms.
Krinkle mentioned this in rMWb6ddc2936a89: Revert "Made insertMessageBlob use (start|end)Atomic in case DBO_TRX is off".Mar 3 2015, 4:40 PM
Nikerabbit merged a task: T91383: Jenkins qunit builds fail for many ContentTranslation patches and complain about EventLogging.Mar 3 2015, 4:51 PM
Nikerabbit subscribed.
hashar added a comment.Mar 3 2015, 10:12 PM

From my comment T91383#1081569 :

The MediaWiki logs are capture for each builds and attached to the build result. https://integration.wikimedia.org/ci/job/mwext-ContentTranslation-qunit/1432/ , compared to the previous passing build 1431, the mw-error.log file is bigger and there is a mw-exception.log file:

2015-03-03 04:48:49 integration-slave1001 my_wiki: [115b7dc6] /jenkins-mwext-ContentTranslation-qunit-1432/load.php?debug=false&lang=en&modules=ext.eventLogging.subscriber%7Cext.uls.pt%7Cjquery.suggestions%7Cmediawiki.hidpi%2CsearchSuggest&skin=fallback&version=20150303T044842Z&*   DBUnexpectedError from line 3669 of src/includes/db/Database.php: Attempted to commit transaction while atomic sections are still open
#0 src/includes/db/LoadBalancer.php(979): DatabaseBase->commit('LoadBalancer::c...', 'flush')
#1 [internal function]: LoadBalancer->commitMasterChanges()
#2 src/includes/db/LBFactory.php(176): call_user_func_array(Array, Array)
#3 [internal function]: {closure}(Object(LoadBalancer), 'commitMasterCha...', Array)
#4 src/includes/db/LBFactory.php(321): call_user_func_array(Object(Closure), Array)
#5 src/includes/db/LBFactory.php(177): LBFactorySimple->forEachLB(Object(Closure), Array)
#6 src/includes/db/LBFactory.php(184): LBFactory->forEachLBCallMethod('commitMasterCha...')
#7 src/includes/db/LBFactory.php(336): LBFactory->commitMasterChanges()
#8 src/load.php(50): LBFactorySimple->shutdown()
#9 {main}
2015-03-03 04:48:52 integration-slave1001 my_wiki: [e71ca201] /jenkins-mwext-ContentTranslation-qunit-1432/load.php?debug=false&lang=en&modules=ext.uls.buttons%2Cdisplaysettings%2Cinputsettings%2Clanguagesettings&skin=fallback&version=20150303T044842Z&*   DBUnexpectedError from line 3669 of src/includes/db/Database.php: Attempted to commit transaction while atomic sections are still open
#0 src/includes/db/LoadBalancer.php(979): DatabaseBase->commit('LoadBalancer::c...', 'flush')
#1 [internal function]: LoadBalancer->commitMasterChanges()
#2 src/includes/db/LBFactory.php(176): call_user_func_array(Array, Array)
#3 [internal function]: {closure}(Object(LoadBalancer), 'commitMasterCha...', Array)
#4 src/includes/db/LBFactory.php(321): call_user_func_array(Object(Closure), Array)
#5 src/includes/db/LBFactory.php(177): LBFactorySimple->forEachLB(Object(Closure), Array)
#6 src/includes/db/LBFactory.php(184): LBFactory->forEachLBCallMethod('commitMasterCha...')
#7 src/includes/db/LBFactory.php(336): LBFactory->commitMasterChanges()
#8 src/load.php(50): LBFactorySimple->shutdown()
#9 {main}

Ie Attempted to commit transaction while atomic sections are still open no clue what it is the cause. Maybe it is caused by the mediawiki/core change https://gerrit.wikimedia.org/r/#/c/193261/ ?

In short: please attempt to reproduce locally with the tip of the master branches of your extension / core and vendor. Then trie with the mediawiki/core commit before 80a3a30d6a90b21f68cca4dfe5b3623cb9e5e7fc and see if it works.

hashar moved this task from Untriaged to Done on the Continuous-Integration-Infrastructure board.Mar 4 2015, 12:43 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL