Page MenuHomePhabricator
Create Task
Maniphest T323080

"Duplicate entry 'XXX-YYY' for key 'itr_itemid_id_revision_id'" in "INSERT INTO `discussiontools_item_revisions`"
Closed, ResolvedPublic
Actions

Description

(The latest deployment of DiscussionTools permalinks backend to group1 wikis T315353#8392662 has resulted in some new errors, although a lot less numerous this time. It really is a gift that keeps on giving.)

There are a few occurrences of "Duplicate entry 'XXX-YYY' for key 'itr_itemid_id_revision_id'" (where "XXX" and "YYY" are different numbers).

https://logstash.wikimedia.org/goto/9f8248c640bb01e49ba706363ee5c5a9

image.png (529×2 px, 104 KB)

Error
Error 1062: Duplicate entry 'XXX-YYY' for key 'itr_itemid_id_revision_id'
Function: MediaWiki\Extension\DiscussionTools\ThreadItemStore::insertThreadItems
Query: INSERT INTO `discussiontools_item_revisions` (itr_itemid_id,itr_revision_id,itr_items_id,itr_parent_id,itr_transcludedfrom,itr_level) VALUES ('XXX',YYY,'ZZZ','AAA',NULL,2)
exception.trace
…
#4 /srv/mediawiki/php-1.40.0-wmf.17/extensions/DiscussionTools/includes/ThreadItemStore.php(555): Wikimedia\Rdbms\Database->insert(string, array, string)
#5 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/Database.php(2620): MediaWiki\Extension\DiscussionTools\ThreadItemStore->MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#6 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/DBConnRef.php(119): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure, string)
#7 /srv/mediawiki/php-1.40.0-wmf.17/includes/libs/rdbms/database/DBConnRef.php(680): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#8 /srv/mediawiki/php-1.40.0-wmf.17/extensions/DiscussionTools/includes/ThreadItemStore.php(566): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure, string)
#9 /srv/mediawiki/php-1.40.0-wmf.17/extensions/DiscussionTools/includes/Hooks/DataUpdatesHooks.php(47): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)
#10 /srv/mediawiki/php-1.40.0-wmf.17/includes/deferred/MWCallableUpdate.php(38): MediaWiki\Extension\DiscussionTools\Hooks\DataUpdatesHooks->MediaWiki\Extension\DiscussionTools\Hooks\{closure}()
#11 /srv/mediawiki/php-1.40.0-wmf.17/includes/deferred/DeferredUpdates.php(474): MWCallableUpdate->doUpdate()
#12 /srv/mediawiki/php-1.40.0-wmf.17/includes/deferred/RefreshSecondaryDataUpdate.php(103): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#13 /srv/mediawiki/php-1.40.0-wmf.17/includes/deferred/DeferredUpdates.php(474): RefreshSecondaryDataUpdate->doUpdate()
#14 /srv/mediawiki/php-1.40.0-wmf.17/includes/Storage/DerivedPageDataUpdater.php(1834): DeferredUpdates::attemptUpdate(RefreshSecondaryDataUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#15 /srv/mediawiki/php-1.40.0-wmf.17/includes/page/WikiPage.php(2150): MediaWiki\Storage\DerivedPageDataUpdater->doSecondaryDataUpdates(array)
#16 /srv/mediawiki/php-1.40.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php(243): WikiPage->doSecondaryDataUpdates(array)
#17 /srv/mediawiki/php-1.40.0-wmf.17/includes/jobqueue/jobs/RefreshLinksJob.php(152): RefreshLinksJob->runForTitle(Title)
#18 /srv/mediawiki/php-1.40.0-wmf.17/extensions/EventBus/includes/JobExecutor.php(79): RefreshLinksJob->run()
#19 /srv/mediawiki/rpc/RunSingleJob.php(77): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#20 {main}

It looks like the revisions are being processed simultaneously by multiple jobs? (as Ladsgroup was guessing in T322701#8382329)

I don't know if we can dig into that, but if that's the case, this should be harmless from our perspective – we should avoid logging errors about it when it happens, though…

(This is distinct from T323079: "Duplicate entry '0-XXX' for key 'itr_itemid_id_revision_id'" in "INSERT INTO `discussiontools_item_revisions`")

Details

SubjectRepoBranchLines +/-
ThreadItemStore: Ignore duplicates caused by duplicate executionsmediawiki/extensions/DiscussionToolswmf/1.41.0-wmf.20+28 -12
ThreadItemStore: Ignore duplicates caused by duplicate executionsmediawiki/extensions/DiscussionToolsmaster+28 -12
Customize query in gerrit

Related Objects
Search...

Event Timeline

matmarex created this task.Nov 14 2022, 10:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 14 2022, 10:09 PM
matmarex mentioned this in T323079: "Duplicate entry '0-XXX' for key 'itr_itemid_id_revision_id'" in "INSERT INTO `discussiontools_item_revisions`".Nov 14 2022, 10:11 PM
matmarex updated the task description. (Show Details)
matmarex added a project: Wikimedia-production-error.
matmarex added a parent task: T315353: Create database tables for permalinks in production wikis, and enable the feature.
brennen added a project: User-brennen.Nov 15 2022, 5:39 PM
brennen moved this task from Backlog to Logs/Train on the User-brennen board.
Ladsgroup subscribed.Nov 15 2022, 6:13 PM

Looking at an example of it for example https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2022.11.15?id=R15SfIQBGZfvtWAWG4lG
The revision is quite old, there is no newer edit on the page so it is very likely caused by refreshlinks job being somehow trying to run the DT update twice (or the update itself is trying it twice).

Ladsgroup added a comment.Nov 15 2022, 6:33 PM

I might be wrong but it feels like the last update in the transaction (starting line 526 on discussiontools_item_revisions) is suffering from REPEATABLE_READ issue as well?

• ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.Nov 17 2022, 4:07 PM
• ppelberg added a project: Editing-team.
• ppelberg moved this task from Untriaged to Upcoming on the Editing-team board.
Krinkle moved this task from Untriaged to Nov 2022 on the Wikimedia-production-error board.Nov 17 2022, 4:23 PM
brennen subscribed.Nov 18 2022, 5:27 PM

Noting that this and T323079: "Duplicate entry '0-XXX' for key 'itr_itemid_id_revision_id'" in "INSERT INTO `discussiontools_item_revisions`" are happening at what feels like a somewhat higher rate than the last few days. Checking logstash, that may not actually be the case, but they're persistent enough to stand out to deployers. (We've filtered them out of the mediawiki-new-errors dashboard in logstash, but they're something of an annoyance in logspam-watch.)

Krinkle updated the task description. (Show Details)Jan 5 2023, 9:43 PM
matmarex merged a task: T333245: Wikimedia\Rdbms\DBQueryError: Error 1062: Duplicate entry for key 'itr_itemid_id_revision_id'.Jun 19 2023, 11:38 AM
matmarex added a subscriber: TheresNoTime.
matmarex mentioned this in T315510: Start maintenance script to backfill talk page comment database.Jun 19 2023, 11:49 AM
matmarex mentioned this in T339882: Bad rows in discussiontools_item_pages, discussiontools_item_revisions with itp_items_id, itr_itemid_id, itr_items_id equal to 0.Jun 19 2023, 9:04 PM
Urbanecm subscribed.Jun 20 2023, 2:25 PM
In T323080#8406279, @brennen wrote:

Noting that this and T323079: "Duplicate entry '0-XXX' for key 'itr_itemid_id_revision_id'" in "INSERT INTO `discussiontools_item_revisions`" are happening at what feels like a somewhat higher rate than the last few days. Checking logstash, that may not actually be the case, but they're persistent enough to stand out to deployers. (We've filtered them out of the mediawiki-new-errors dashboard in logstash, but they're something of an annoyance in logspam-watch.)

This is probably because it is happening from within a maintenance script which is usually not running (see T315510).

Krinkle mentioned this in T341811: DBQueryError: "Duplicate entry … for key 'itp_items_id_page_id'" in "INSERT INTO `discussiontools_item_pages`".Jul 13 2023, 3:23 PM
matmarex updated the task description. (Show Details)Jul 31 2023, 12:12 PM
gerritbot added a comment.Aug 2 2023, 12:21 PM

Change 944892 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] ThreadItemStore: Ignore duplicates caused by duplicate executions

https://gerrit.wikimedia.org/r/944892

gerritbot added a project: Patch-For-Review.Aug 2 2023, 12:21 PM
gerritbot added a comment.Aug 4 2023, 7:11 PM

Change 944892 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] ThreadItemStore: Ignore duplicates caused by duplicate executions

https://gerrit.wikimedia.org/r/944892

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2023, 7:31 PM
gerritbot added a comment.Aug 7 2023, 7:42 PM

Change 945803 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.41.0-wmf.20] ThreadItemStore: Ignore duplicates caused by duplicate executions

https://gerrit.wikimedia.org/r/945803

gerritbot added a project: Patch-For-Review.Aug 7 2023, 7:42 PM
gerritbot added a comment.Aug 7 2023, 8:06 PM

Change 945803 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.41.0-wmf.20] ThreadItemStore: Ignore duplicates caused by duplicate executions

https://gerrit.wikimedia.org/r/945803

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2023, 8:11 PM
Stashbot mentioned this in T341255: Design: Provide wordmarks/taglines for Wikisource projects.Aug 7 2023, 8:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-07T20:14:21Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:945803|ThreadItemStore: Ignore duplicates caused by duplicate executions (T323080 T341811)]], [[gerrit:946616|Update wikisource wordmarks and taglines (T341255)]], [[gerrit:946555|update idwiktionary legacy vector logo (T341175)]]

Stashbot mentioned this in T341175: Change the Indonesian Wiktionary's name and project namespace from Wiktionary to Wikikamus.Aug 7 2023, 8:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-07T20:15:45Z] <urbanecm@deploy1002> urbanecm and jdlrobson and anzx and matmarex: Backport for [[gerrit:945803|ThreadItemStore: Ignore duplicates caused by duplicate executions (T323080 T341811)]], [[gerrit:946616|Update wikisource wordmarks and taglines (T341255)]], [[gerrit:946555|update idwiktionary legacy vector logo (T341175)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet,

Stashbot added a comment.Aug 7 2023, 8:24 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-07T20:24:43Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:945803|ThreadItemStore: Ignore duplicates caused by duplicate executions (T323080 T341811)]], [[gerrit:946616|Update wikisource wordmarks and taglines (T341255)]], [[gerrit:946555|update idwiktionary legacy vector logo (T341175)]] (duration: 10m 22s)

matmarex closed this task as Resolved.Aug 8 2023, 2:31 PM
matmarex claimed this task.

No more errors since the change was deployed, and everything else seems to be working normally:

image.png (523×2 px, 132 KB)

matmarex mentioned this in T343859: DiscussionTools: LogicException: Database can't find our row and won't let us insert it.Aug 8 2023, 9:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL