DBQueryError when using rollback on incubatorwiki
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

None

Authored By

	DannyS712
	Aug 19 2020, 2:03 PM

Description

Error message

[379e779f-8fa6-495c-b1da-c447aed4f159] 2020-08-19 13:50:57: Fatal exception of type "Wikimedia\Rdbms\DBQueryError"

Impact

Rollback failed

Notes

Was part of a series of rollbacks (with bot=1) cleaning up https://incubator.wikimedia.org/wiki/Special:Contributions/こんやくらべてみました
I got a number of DBTransactionSizeError failures, but also this DBQueryError (for the record, DBTransactionSizeError codes were ff982676-e1f6-4127-8563-95064bc5145c, f4a3ef2c-a480-41d8-9277-277edeb0c2d7, 15d1ca5a-2628-47a0-b52a-e52fc816e5ec, 63cd20b1-d172-4e80-802d-27a9889288e4, and one other I missed)
The rollbacks were reverting huge edits (2 million bytes added in the underlying edit, and attempting to remove in the rollback)

Details

Request URL: https://incubator.wikimedia.org/w/index.php?title=…&action=rollback&from=…&token=[snip]&bot=1

	Subject	Repo	Branch	Lines +/-
	RollbackPage: Include patrol status of revisions with same timestamp	mediawiki/core	master	+15 -16
	RollbackPage: Make rollback not overwrite manual RC patrol status	mediawiki/core	master	+159 -38

Customize query in gerrit

Related Objects

Mentioned Here: T304766: Evaluate the secret "bot marks reverted edits as bot" feature

Event Timeline

DannyS712 created this task.Aug 19 2020, 2:03 PM

Restricted Application added a project: User-DannyS712. · View Herald TranscriptAug 19 2020, 2:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

DannyS712 moved this task from Unsorted to Reports on the User-DannyS712 board.Aug 19 2020, 2:03 PM

Lock wait timeout exceeded; try restarting transaction (10.64.48.35)
/srv/mediawiki/php-1.36.0-wmf.4/includes/libs/rdbms/database/Database.php:1680

#0 /srv/mediawiki/php-1.36.0-wmf.4/includes/libs/rdbms/database/Database.php(1658): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.4/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /srv/mediawiki/php-1.36.0-wmf.4/includes/libs/rdbms/database/Database.php(2457): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#4 /srv/mediawiki/php-1.36.0-wmf.4/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.36.0-wmf.4/includes/page/WikiPage.php(3464): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#6 /srv/mediawiki/php-1.36.0-wmf.4/includes/page/WikiPage.php(3239): WikiPage->commitRollback(string, string, boolean, NULL, User, NULL)
#7 /srv/mediawiki/php-1.36.0-wmf.4/includes/actions/RollbackAction.php(124): WikiPage->doRollback(string, string, string, boolean, NULL, User)
#8 /srv/mediawiki/php-1.36.0-wmf.4/includes/actions/RollbackAction.php(87): RollbackAction->handleRollbackRequest()
#9 /srv/mediawiki/php-1.36.0-wmf.4/includes/MediaWiki.php(527): RollbackAction->show()
#10 /srv/mediawiki/php-1.36.0-wmf.4/includes/MediaWiki.php(313): MediaWiki->performAction(Article, Title)
#11 /srv/mediawiki/php-1.36.0-wmf.4/includes/MediaWiki.php(940): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.36.0-wmf.4/includes/MediaWiki.php(543): MediaWiki->main()
#13 /srv/mediawiki/php-1.36.0-wmf.4/index.php(53): MediaWiki->run()
#14 /srv/mediawiki/php-1.36.0-wmf.4/index.php(46): wfIndexMain()
#15 /srv/mediawiki/w/index.php(3): require(string)
#16 {main}

thcipriani added projects: Release-Engineering-Team (Logspam), MediaWiki-Patrolling.Aug 19 2020, 6:50 PM

thcipriani moved this task from Untriaged to Aug 2020 on the Wikimedia-production-error board.

daniel added a project: Platform Team Workboards (Clinic Duty Team).Aug 25 2020, 8:58 PM

daniel moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.

daniel triaged this task as Medium priority.Aug 25 2020, 9:00 PM

thcipriani removed a project: Release-Engineering-Team (Logspam).Sep 9 2020, 10:10 PM

The failing update is:

			$dbw->update( 'recentchanges', $set,
				[ /* WHERE */
					'rc_cur_id' => $current->getPageId(),
					'rc_timestamp > ' . $dbw->addQuotes( $s->rev_timestamp ),
					$actorWhere['conds'], // No tables/joins are needed for rc_user
				],
				__METHOD__
			);

How to find the reason for the lock wait?

Umherirrender unsubscribed.Feb 27 2021, 9:52 PM

Krinkle changed Request URL from https://incubator.wikimedia.org/w/index.php?title=Wy/ja/メインページ&action=rollback&from=こんやくらべてみました&token=[snip]&bot=1 to https://incubator.wikimedia.org/w/index.php?title=…&action=rollback&from=…&token=[snip]&bot=1.Apr 1 2021, 9:55 PM

Krinkle edited Stack Trace. (Show Details)

In T260798#6576372, @Umherirrender wrote:

The failing update is:

	$dbw->update( 'recentchanges', $set,
		[ /* WHERE */
			'rc_cur_id' => $current->getPageId(),
			'rc_timestamp > ' . $dbw->addQuotes( $s->rev_timestamp ),
			$actorWhere['conds'], // No tables/joins are needed for rc_user
		],
		__METHOD__
	);

Source code: https://gerrit.wikimedia.org/g/mediawiki/core/+/63c77060a0e446d3a5c7ace7d15647d7517444a6/includes/page/RollbackPage.php#438

Change 773927 had a related patch set uploaded (by Krinkle; author: Matěj Suchánek):

[mediawiki/core@master] RollbackPage: Make rollback not overwrite manual RC patrol status

https://gerrit.wikimedia.org/r/773927

gerritbot added a project: Patch-For-Review.May 31 2022, 9:51 PM

Krinkle removed a project: WMF-General-or-Unknown.May 31 2022, 9:57 PM

Krinkle edited projects, added Performance-Team (Radar); removed Platform Team Workboards (Clinic Duty Team).

Change 773927 merged by jenkins-bot:

[mediawiki/core@master] RollbackPage: Make rollback not overwrite manual RC patrol status

https://gerrit.wikimedia.org/r/773927

Maintenance_bot removed a project: Patch-For-Review.Jun 1 2022, 11:30 PM

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.15; 2022-06-06).Jun 2 2022, 12:00 AM

Change 802765 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] rollback: Also adjust patrol status for revision with same timestamp

https://gerrit.wikimedia.org/r/802765

gerritbot added a project: Patch-For-Review.Jun 3 2022, 12:29 PM

Change 802765 merged by jenkins-bot:

[mediawiki/core@master] RollbackPage: Include patrol status of revisions with same timestamp

https://gerrit.wikimedia.org/r/802765

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2022, 9:30 PM

Umherirrender unsubscribed.Jun 20 2022, 4:17 PM

Krinkle moved this task from Aug 2020 to Older on the Wikimedia-production-error board.Sep 8 2022, 4:20 PM

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.Sep 26 2022, 8:09 PM

Since https://gerrit.wikimedia.org/r/c/mediawiki/core/+/773927 all updates on action=rollback are on primary keys - deadlocks should be fixed, if there are still deadlocks the stack trace would looks very different now and a new task should be created

DBQueryError when using rollback on incubatorwikiClosed, ResolvedPublicPRODUCTION ERRORActions