Page MenuHomePhabricator
Create Task
Maniphest T407357

Ignore duplicate key errors when creating Flow posts from LQT
Closed, ResolvedPublic
Actions

Description

It rare situations (about 1 in 1,000) it seems we try to insert a Flow revision with the same ID twice. As revision IDs contain a random seed, this means it must be the same object being inserted twice, so I'm fairly confident we will not be losing data.

Example error:

[2025-10-15 12:56:22] Wikimedia\Rdbms\DBQueryError: Error 1062: Duplicate entry '\x04\xA2\x9D\xBF\xB0\x15\x16\xCA\x7F\xFA\x90' for key 'PRIMARY'
Function: Flow\Data\Storage\RevisionStorage::insert
Query: INSERT INTO `flow_revision` (rev_id,rev_user_id,rev_user_ip,rev_user_wiki,rev_parent_id,rev_change_type,rev_type,rev_type_id,rev_content,rev_flags,rev_mod_state,rev_mod_user_id,rev_mod_user_ip,rev_mod_user_wiki,rev_mod_timestamp,rev_mod_reason,rev_last_edit_id,rev_edit_user_id,rev_edit_user_ip,rev_edit_user_wiki,rev_content_length,rev_previous_content_length) VALUES ('�������',8006,NULL,'enwiktionary',NULL,'reply','post','�������','DB://cluster30/4173353','utf-8,gzip,html,external','',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,224,0),('�������',8006,NULL,'enwiktionary',NULL,'reply','post','�������','DB://cluster31/4173620','utf-8,gzip,html,external','',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,224,0)
 in /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/Database.php:1225
Stack trace:
#0 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/Database.php(1209): Wikimedia\Rdbms\Database->getQueryException('Duplicate entry...', 1062, 'INSERT INTO `fl...', 'Flow\\Data\\Stora...')
#1 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/Database.php(1183): Wikimedia\Rdbms\Database->getQueryExceptionAndLog('Duplicate entry...', 1062, 'INSERT INTO `fl...', 'Flow\\Data\\Stora...')
#2 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/Database.php(640): Wikimedia\Rdbms\Database->reportQueryError('Duplicate entry...', 1062, 'INSERT INTO `fl...', 'Flow\\Data\\Stora...', false)
#3 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/Database.php(1505): Wikimedia\Rdbms\Database->query(Object(Wikimedia\Rdbms\Query), 'Flow\\Data\\Stora...')
#4 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(127): Wikimedia\Rdbms\Database->insert('flow_revision', Array, 'Flow\\Data\\Stora...', Array)
#5 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(455): Wikimedia\Rdbms\DBConnRef->__call('insert', Array)
#6 /srv/mediawiki/php-1.45.0-wmf.23/includes/libs/rdbms/querybuilder/InsertQueryBuilder.php(340): Wikimedia\Rdbms\DBConnRef->insert('flow_revision', Array, 'Flow\\Data\\Stora...', Array)
#7 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Data/Storage/RevisionStorage.php(373): Wikimedia\Rdbms\InsertQueryBuilder->execute()
#8 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Data/ObjectManager.php(248): Flow\Data\Storage\RevisionStorage->insert(Array)
#9 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Data/ObjectManager.php(173): Flow\Data\ObjectManager->insert(Array, Array)
#10 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Data/ManagerGroup.php(98): Flow\Data\ObjectManager->multiPut(Array, Array)
#11 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Data/ManagerGroup.php(107): Flow\Data\ManagerGroup->multiMethod('multiPut', Array, Array)
#12 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/PageImportState.php(117): Flow\Data\ManagerGroup->multiPut(Array, Array)
#13 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/TalkpageImportOperation.php(413): Flow\Import\PageImportState->put(Array, Array)
#14 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/TalkpageImportOperation.php(217): Flow\Import\TalkpageImportOperation->importPost(Object(Flow\Import\TopicImportState), Object(Flow\Import\LiquidThreadsApi\ImportPost), Object(Flow\Model\PostRevision))
#15 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/TalkpageImportOperation.php(131): Flow\Import\TalkpageImportOperation->importTopic(Object(Flow\Import\TopicImportState), Object(Flow\Import\LiquidThreadsApi\ImportTopic))
#16 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/Importer.php(114): Flow\Import\TalkpageImportOperation->import(Object(Flow\Import\PageImportState))
#17 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/Converter.php(215): Flow\Import\Importer->import(Object(Flow\Import\LiquidThreadsApi\ImportSource), Object(MediaWiki\Title\Title), Object(MediaWiki\User\User), Object(Flow\Import\SourceStore\FileImportSourceStore))
#18 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/Converter.php(157): Flow\Import\Converter->doConversion(Object(MediaWiki\Title\Title), Object(MediaWiki\Title\Title))
#19 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/includes/Import/Converter.php(113): Flow\Import\Converter->convert(Object(MediaWiki\Title\Title), false, false)
#20 /srv/mediawiki/php-1.45.0-wmf.23/extensions/Flow/maintenance/convertLqtPageOnLocalWiki.php(101): Flow\Import\Converter->convertAll(Array, false, false)
#21 /srv/mediawiki/php-1.45.0-wmf.23/maintenance/includes/MaintenanceRunner.php(696): Flow\Maintenance\ConvertLqtPageOnLocalWiki->execute()
#22 /srv/mediawiki/php-1.45.0-wmf.23/maintenance/run.php(53): MediaWiki\Maintenance\MaintenanceRunner->run()
#23 /srv/mediawiki/multiversion/MWScript.php(221): require_once('/srv/mediawiki/...')
#24 {main}
[2025-10-15 12:56:22] Imported 1 items, failed 1
[2025-10-15 12:56:22] Failed to complete import to Wiktionary:LiquidThreads testing from Wiktionary:LiquidThreads testing/LQT Archive 1
[2025-10-15 12:56:22] Finished LQT conversion of page Wiktionary:LiquidThreads testing/LQT Archive 1

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Follow-up Iedb6361: Set insert-ignore on all insertSelect queriesmediawiki/extensions/Flowwmf/1.45.0-wmf.23+22 -10
Follow-up Iedb6361: Set insert-ignore on all insertSelect queriesmediawiki/extensions/Flowmaster+22 -10
Follow-up I6698875: Set insert-ignore on all insert queriesmediawiki/extensions/Flowwmf/1.45.0-wmf.23+38 -24
Follow-up I6698875: Set insert-ignore on all insert queriesmediawiki/extensions/Flowmaster+38 -24
LQT convert: Ignore duplicate key insert errors when command line flag setmediawiki/extensions/Flowwmf/1.45.0-wmf.23+16 -3
LQT convert: Ignore duplicate key insert errors when command line flag setmediawiki/extensions/Flowmaster+16 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT335670 Disable Flow/Structured Discussions from all namespaces, except for user talk pages, on Catalan Wikipedia
 OpenNoneT106123 Extensions needing to be removed from Wikimedia wikis
 OpenNoneT332022 [Epic] Undeploying StructuredDiscussions (Flow)
 ResolvedTrizek-WMFT332023 Make a comparative review of Flow, LQT and DiscussionTools
 DuplicateNoneT332109 Gather statistics of Flow and Liquidthreads pages on Wikimedia wikis
 OpenNoneT332103 Research Spike: Test Flow -> wikitext conversion
 ResolvedTrizek-WMFT333671 Create a MediaWiki page to initiate a discussion about Flow deprecation
 DuplicateNoneT335772 Remove the activation of Structured Discussions on user talk pages from Beta features
 ResolvedSgsT248309 StructuredDiscussions (Flow): Prevent users from enabling Flow on their user talk page (beta feature)
 ResolvedSgsT352565 Edit flow-talk-page-beta-feature-description to reflect the upcoming deprecation
ResolvedTrizek-WMFT346108 [EPIC] IP Masking: StructuredDiscussions (Flow)/LiquidThreads Community discussion
 OpenNoneT351344 Phase 3: Remove Extension:StructuredDiscussions from wikis where it is not used
 ResolvedMimurawilT368910 Disable interactions with Flow content for users if temp accounts are used
 ResolvedTchandersT370322 Set Flow to read only for testwiki
 In ProgressNoneT370722 Set Flow and LQT sunsetting timeline and sequence
 OpenNoneT350164 Undeploy LiquidThreads
 DeclinedNoneT332104 Research Spike: Test LiquidThreads -> wikitext conversion
 ResolvedEsandersT407357 Ignore duplicate key errors when creating Flow posts from LQT
 OpenNoneT377050 [epic] Phase #1: Archive all active Flow boards
 ResolvedEsandersT365846 Assist communities in archiving Flow/StructuredDiscussions boards
 ResolvedNoneT366260 Identify communities that use Extension:StructuredDiscussions as their default talk pages system
 ResolvedDLynchT371738 Write a script to automatically migrate Flow boards to sub-pages
 OpenDLynchT371769 Moving a Flow discussion page can render it inaccessible
 ResolvedUrbanecm_WMFT371586 Flow internal error on frwiki not in logstash
 ResolvedDLynchT377990 [Config] Set Flow to "read-only" at all Phase 0 wikis
 ResolvedDLynchT376749 Run Flow migration script at *Phase 0* wikis
 OpenEsandersT379982 Manually migrate pages skipped during automatic migration (Phase 0)
 ResolvedRyasmeenT378833 [Config] Set Flow to read-only at all *Phase 1* wikis
 ResolvedUrbanecm_WMFT378827 Run Flow migration script at *Phase 1* wikis
 ResolvedDLynchT379984 Amend archival script to include a fallback option when first move option fails
 ResolvedTrizek-WMFT380316 Inform Phase 1 wikis of Flow removal plan and timing
 ResolvedzoeT378834 [Config] Set Flow to read-only at all *Phase 2a* wikis
 ResolvedRyasmeenT378829 Run Flow migration script at *Phase 2a* wikis
 ResolvedBUG REPORTEtonkovidovaT378781 Error when moving a StructuredDiscussion page
 ResolvedTrizek-WMFT380910 Inform Phase 2a wikis of Flow removal plan and timing
 ResolvedNoneT381290 Phase 2a wikis of Flow removal moves errors
 ResolvedPRODUCTION ERRORhubaishanT381284 Fatal exception: "Structured Discussions workflow is not associated with this page" on arwiki
 ResolvedTrizek-WMFT381292 Remove Flow as the default talk pages system at concerned phase 2a wikis
 ResolvedDLynchT381295 Disabling Flow in cawiki 'Viquiprojecte Discussió' namespace
 ResolvedTrizek-WMFT383775 Run Flow migration script at *Phase 2a* wikis, second run
 ResolvedppelbergT380909 [Config] Set Flow to read-only at all *Phase 2b* wikis
 ResolvedQuiddityT380912 Inform *Phase 2b* wikis of Flow removal plan and timing
 ResolvedppelbergT380911 Run Flow migration script at *Phase 2b* wikis
 ResolvedzoeT389247 Run Flow migration script at *gomwiki*
 ResolvedzoeT389113 [SPIKE] Investigate feasibility of revising Flow migration script
 ResolvedRyasmeenT383569 Set DiscussionTools as default talk pages system at Phase 2b wikis
 ResolvedzoeT401825 Manually archive Village Pump Flow boards on sewikimedia and fiwikimedia
 OpenNoneT412829 Stop doing database dumps of flow boards?
 OpenNoneT377051 Phase #2: Convert archived Flow boards to wikitext
 OpenNoneT379671 Remove Flow from doc.wikimedia.org
 OpenNoneT379985 Improve clarity of Flow warnings when set to read only
 OpenNoneT382021 Fix page handled by Structured Discussions, while the Title class indicates that the content model is 'wikitext'
 ResolvedmatmarexT389673 Make it possible to remove extensions' event data from Echo tables after undeploying them
 OpenNoneT389680 Consider creating static-flow.wikimedia.org

Event Timeline

Esanders created this task.Oct 15 2025, 1:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2025, 1:30 PM
gerritbot added a comment.Oct 15 2025, 1:32 PM

Change #1196444 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@master] LQT convert: Ignore duplicate key insert errors when command line flag set

https://gerrit.wikimedia.org/r/1196444

gerritbot added a project: Patch-For-Review.Oct 15 2025, 1:32 PM
gerritbot added a comment.Oct 16 2025, 10:18 AM

Change #1196645 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@wmf/1.45.0-wmf.23] LQT convert: Ignore duplicate key insert errors when command line flag set

https://gerrit.wikimedia.org/r/1196645

gerritbot added a comment.Oct 16 2025, 11:28 AM

Change #1196444 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] LQT convert: Ignore duplicate key insert errors when command line flag set

https://gerrit.wikimedia.org/r/1196444

ReleaseTaggerBot added a project: MW-1.45-notes (1.45.0-wmf.24; 2025-10-21).Oct 16 2025, 12:00 PM
gerritbot added a comment.Oct 16 2025, 1:02 PM

Change #1196645 merged by jenkins-bot:

[mediawiki/extensions/Flow@wmf/1.45.0-wmf.23] LQT convert: Ignore duplicate key insert errors when command line flag set

https://gerrit.wikimedia.org/r/1196645

Stashbot added a comment.Oct 16 2025, 1:03 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-16T13:03:31Z] <esanders@deploy2002> Started scap sync-world: Backport for [[gerrit:1196645|LQT convert: Ignore duplicate key insert errors when command line flag set (T407357)]]

Stashbot added a comment.Oct 16 2025, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-16T13:06:12Z] <esanders@deploy2002> esanders: Backport for [[gerrit:1196645|LQT convert: Ignore duplicate key insert errors when command line flag set (T407357)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 16 2025, 1:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-16T13:13:45Z] <esanders@deploy2002> Finished scap sync-world: Backport for [[gerrit:1196645|LQT convert: Ignore duplicate key insert errors when command line flag set (T407357)]] (duration: 10m 14s)

Maintenance_bot removed a project: Patch-For-Review.Oct 16 2025, 1:31 PM
ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.23; 2025-10-14); removed MW-1.45-notes (1.45.0-wmf.24; 2025-10-21).Oct 16 2025, 2:00 PM
gerritbot added a comment.Oct 17 2025, 8:49 AM

Change #1196864 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@master] Follow-up I6698875: Set insert-ignore on all insert queries

https://gerrit.wikimedia.org/r/1196864

gerritbot added a project: Patch-For-Review.Oct 17 2025, 8:49 AM
gerritbot added a comment.Oct 17 2025, 9:11 AM

Change #1196864 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] Follow-up I6698875: Set insert-ignore on all insert queries

https://gerrit.wikimedia.org/r/1196864

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2025, 9:31 AM
ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.24; 2025-10-21); removed MW-1.45-notes (1.45.0-wmf.23; 2025-10-14).Oct 17 2025, 10:00 AM
gerritbot added a comment.Oct 17 2025, 10:50 AM

Change #1196884 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@wmf/1.45.0-wmf.23] Follow-up I6698875: Set insert-ignore on all insert queries

https://gerrit.wikimedia.org/r/1196884

gerritbot added a project: Patch-For-Review.Oct 17 2025, 10:50 AM
gerritbot added a comment.Oct 20 2025, 1:04 PM

Change #1196884 merged by jenkins-bot:

[mediawiki/extensions/Flow@wmf/1.45.0-wmf.23] Follow-up I6698875: Set insert-ignore on all insert queries

https://gerrit.wikimedia.org/r/1196884

Stashbot added a comment.Oct 20 2025, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-20T13:04:48Z] <esanders@deploy2002> Started scap sync-world: Backport for [[gerrit:1196884|Follow-up I6698875: Set insert-ignore on all insert queries (T407357)]]

Stashbot added a comment.Oct 20 2025, 1:30 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-20T13:30:05Z] <esanders@deploy2002> esanders: Backport for [[gerrit:1196884|Follow-up I6698875: Set insert-ignore on all insert queries (T407357)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2025, 1:30 PM
Stashbot added a comment.Oct 20 2025, 1:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-20T13:43:24Z] <esanders@deploy2002> Finished scap sync-world: Backport for [[gerrit:1196884|Follow-up I6698875: Set insert-ignore on all insert queries (T407357)]] (duration: 38m 36s)

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.23; 2025-10-14); removed MW-1.45-notes (1.45.0-wmf.24; 2025-10-21).Oct 20 2025, 2:00 PM
gerritbot added a comment.Oct 20 2025, 2:07 PM

Change #1197275 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@master] Follow-up Iedb6361: Set insert-ignore on all insertSelect queries

https://gerrit.wikimedia.org/r/1197275

gerritbot added a project: Patch-For-Review.Oct 20 2025, 2:07 PM
gerritbot added a comment.Oct 20 2025, 2:35 PM

Change #1197275 merged by jenkins-bot:

[mediawiki/extensions/Flow@master] Follow-up Iedb6361: Set insert-ignore on all insertSelect queries

https://gerrit.wikimedia.org/r/1197275

gerritbot added a comment.Oct 20 2025, 2:53 PM

Change #1197284 had a related patch set uploaded (by Esanders; author: Esanders):

[mediawiki/extensions/Flow@wmf/1.45.0-wmf.23] Follow-up Iedb6361: Set insert-ignore on all insertSelect queries

https://gerrit.wikimedia.org/r/1197284

ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.24; 2025-10-21); removed MW-1.45-notes (1.45.0-wmf.23; 2025-10-14).Oct 20 2025, 3:00 PM
gerritbot added a comment.Oct 21 2025, 7:10 AM

Change #1197284 merged by jenkins-bot:

[mediawiki/extensions/Flow@wmf/1.45.0-wmf.23] Follow-up Iedb6361: Set insert-ignore on all insertSelect queries

https://gerrit.wikimedia.org/r/1197284

Stashbot added a comment.Oct 21 2025, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-21T07:10:54Z] <esanders@deploy2002> Started scap sync-world: Backport for [[gerrit:1197284|Follow-up Iedb6361: Set insert-ignore on all insertSelect queries (T407357)]]

Stashbot added a comment.Oct 21 2025, 7:15 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-21T07:15:50Z] <esanders@deploy2002> esanders: Backport for [[gerrit:1197284|Follow-up Iedb6361: Set insert-ignore on all insertSelect queries (T407357)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Oct 21 2025, 7:22 AM

Mentioned in SAL (#wikimedia-operations) [2025-10-21T07:22:39Z] <esanders@deploy2002> Finished scap sync-world: Backport for [[gerrit:1197284|Follow-up Iedb6361: Set insert-ignore on all insertSelect queries (T407357)]] (duration: 11m 45s)

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2025, 7:30 AM
ReleaseTaggerBot edited projects, added MW-1.45-notes (1.45.0-wmf.23; 2025-10-14); removed MW-1.45-notes (1.45.0-wmf.24; 2025-10-21).Oct 21 2025, 8:00 AM
Esanders closed this task as Resolved.Oct 21 2025, 9:45 AM
Esanders claimed this task.

This has been worked around now.

The cause appears to be comments that are edited and then reverted back to their original state, creating two revisions with the same content. I tried to reproduce this locally but without success, so I will continue with the INSERT IGNORE strategy, as no real data is lost, only a tiny amount of edit history.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits