Page MenuHomePhabricator

Importing some pages fails because of deadlockLoop (using postgres)
Closed, ResolvedPublic


Author: brj

Short version: certain pages from an XML dump (I haven't figured out any distinguishing characteristics) fail to import. Removing the call to deadlockLoop in SpecialImport.php fixes the problem.

Background: I'm attempting to migrate a wiki from UseMod wiki to MediaWiki. Luckily, MediaWiki includes a script to dump a UseMod wiki into MediaWiki's import/export format. I used this script, it worked without problems, and I now have a dump.

Now the problem: importing the dump. Running importDump.php with the dump file gives an error, so I split the dump into individual pages in order to try to isolate the problem. About a quarter of the pages imported successfully, the rest failed with the same error:

Warning: pg_query(): Query failed: ERROR: current transaction is aborted, commands ignored until end of transaction block in /mnt/home/brj/public_html/fw/includes/DatabasePostgres.php on line 499
SQL error: ERROR: current transaction is aborted, commands ignored until end of transaction block
#0 /mnt/home/brj/public_html/fw/includes/LinksUpdate.php(542): DatabasePostgres->fetchObject(false)
#1 /mnt/home/brj/public_html/fw/includes/LinksUpdate.php(91): LinksUpdate->getExistingImages()
#2 /mnt/home/brj/public_html/fw/includes/LinksUpdate.php(77): LinksUpdate->doIncrementalUpdate()
#3 /mnt/home/brj/public_html/fw/includes/Article.php(2297): LinksUpdate->doUpdate()
#4 /mnt/home/brj/public_html/fw/includes/Article.php(2366): Article->editUpdates('''The time spen...', 'link fix', true, 1188199043, '2110', true)
#5 /mnt/home/brj/public_html/fw/includes/SpecialImport.php(366): Article->createUpdates(Object(Revision))
#6 [internal function]: WikiRevision->importOldRevision()
#7 /mnt/home/brj/public_html/fw/includes/Database.php(1861): call_user_func_array(Array, Array)
#8 /mnt/home/brj/public_html/fw/includes/SpecialImport.php(511): Database->deadlockLoop(Array)
#9 [internal function]: WikiImporter->importRevision(Object(WikiRevision))
#10 /mnt/home/brj/public_html/fw/maintenance/importDump.php(58): call_user_func(Array, Object(WikiRevision))
#11 [internal function]: BackupReader->handleRevision(Object(WikiRevision), Object(WikiImporter))
#12 /mnt/home/brj/public_html/fw/includes/SpecialImport.php(762): call_user_func_array(Array, Array)
#13 [internal function]: WikiImporter->out_revision(Resource id #31, 'revision')
#14 /mnt/home/brj/public_html/fw/includes/SpecialImport.php(426): xml_parse(Resource id #31, '<?xml version="...', 0)
#15 /mnt/home/brj/public_html/fw/maintenance/importDump.php(109): WikiImporter->doImport()
#16 /mnt/home/brj/public_html/fw/maintenance/importDump.php(96): BackupReader->importFromHandle(Resource id #30)
#17 /mnt/home/brj/public_html/fw/maintenance/importDump.php(131): BackupReader->importFromStdin()

I dug into SpecialImport.php to figure out why this was happening. Eventually I guessed that deadlockLoop might have been written with some MySQL assumptions, and this hadn't been noticed because importing is relatively rare. Since I'm doing a migration, the site I'm importing to isn't live, so I didn't think deadlocks would be an issue. So, I made the following change:

  • SpecialImport.php.orig 2007-04-21 05:13:47.000000000 -0700

+++ SpecialImport.php 2007-08-27 00:55:33.000000000 -0700
@@ -506,8 +506,9 @@

    • @private */ function importRevision( &$revision ) {
  • $dbw = wfGetDB( DB_MASTER );
  • return $dbw->deadlockLoop( array( &$revision, 'importOldRevision' ) );

+ #$dbw = wfGetDB( DB_MASTER );
+ #return $dbw->deadlockLoop( array( &$revision, 'importOldRevision' ) );
+ return $revision->importOldRevision();


And then importing worked! I'm sure this isn't the actual fix for the problem, as there was probably a reason for the deadlock avoidance, but there's a bug in there somewhere. I can provide sample pages that succeed and fail, if needed. I believe, though I haven't confirmed for sure, that it is consistent which pages succeed and which fail. The proportion has definitely been the same (1/4 succeed, 3/4 fail, out of about 2000 pages) the times I've tried. Also, when attempting to import the failed pages again, all of them fail.

Version: 1.10.x
Severity: normal



Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:49 PM
bzimport set Reference to bz11080.
bzimport added a subscriber: Unknown Object (MLST).

You probably don't want to comment that section out, as it is doing some needed work. The error returned is the error *after* the error that failed the transaction. Can you check the database logs and see what the original error was that aborted the transaction?

brj wrote:

Sep 3 19:49:59 kay postgres[32058]: [7-1] ERROR: syntax error at or near "," at character 129
Sep 3 19:49:59 kay postgres[32058]: [7-2] STATEMENT: INSERT /* LinksUpdate::incrTableUpdate */ INTO pagelinks (pl_from,pl_namespace,pl_title) VALUES
Sep 3 19:49:59 kay postgres[32058]: [7-3] ('4','0','BagOfChips'),('4','0','ShowOff')
Sep 3 19:49:59 kay postgres[32058]: [8-1] ERROR: current transaction is aborted, commands ignored until end of transaction block
Sep 3 19:49:59 kay postgres[32058]: [8-2] STATEMENT: SELECT /* LinksUpdate::getExistingImages */ il_to FROM imagelinks WHERE il_from = '4' FOR UPDATE

I think this is a side effect of a recently fixed bug (#24702). Try changing 8.1 to 8.2 around line 700 of DatabasePostgres.php and see if the import works with the deadlock stuff un-uncommented.

brj wrote:

Indeed, that seems to have fixed it.

Jdforrester-WMF subscribed.

Migrating from the old tracking task to a tag for PostgreSQL-related tasks.