Page MenuHomePhabricator

ReadingListRepository::deleteListEntryQuery: BIGINT UNSIGNED value is out of range in '`wikishared`.`reading_list`.`rl_size` - 1'
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error

MediaWiki version: 1.36.0-wmf.2

message
Error 1690: BIGINT UNSIGNED value is out of range in '`wikishared`.`reading_list`.`rl_size` - 1' (10.64.0.164)
Function: MediaWiki\Extensions\ReadingLists\ReadingListRepository::deleteListEntry
Query: UPDATE  `reading_list` SET rl_size = rl_size - 1 WHERE rl_id = '2439265'

Impact / Notes

One of these just now on enwiki.

Details

Request ID
128d187c-adb5-457e-b5ce-935ff0472214
Request URL
https://en.wikipedia.org/w/api.php
Stack Trace
exception.trace
#0 /srv/mediawiki/php-1.36.0-wmf.2/includes/libs/rdbms/database/Database.php(1679): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.36.0-wmf.2/includes/libs/rdbms/database/Database.php(1654): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.36.0-wmf.2/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.36.0-wmf.2/includes/libs/rdbms/database/Database.php(2453): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.2/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->update(string, array, string, string)
#5 /srv/mediawiki/php-1.36.0-wmf.2/includes/libs/rdbms/database/DBConnRef.php(375): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.36.0-wmf.2/extensions/ReadingLists/src/ReadingListRepository.php(752): Wikimedia\Rdbms\DBConnRef->update(string, array, array, string)
#7 /srv/mediawiki/php-1.36.0-wmf.2/extensions/ReadingLists/src/Api/ApiReadingListsDeleteEntry.php(28): MediaWiki\Extensions\ReadingLists\ReadingListRepository->deleteListEntry(integer)
#8 /srv/mediawiki/php-1.36.0-wmf.2/extensions/ReadingLists/src/Api/ApiReadingLists.php(46): MediaWiki\Extensions\ReadingLists\Api\ApiReadingListsDeleteEntry->execute()
#9 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiMain.php(1593): MediaWiki\Extensions\ReadingLists\Api\ApiReadingLists->execute()
#10 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiMain.php(529): ApiMain->executeAction()
#11 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiMain.php(500): ApiMain->executeActionWithErrorHandling()
#12 /srv/mediawiki/php-1.36.0-wmf.2/api.php(90): ApiMain->execute()
#13 /srv/mediawiki/php-1.36.0-wmf.2/api.php(45): wfApiMain()
#14 /srv/mediawiki/w/api.php(3): require(string)
#15 {main}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I guess rl_size - 1 is somehow coming out negative here. I notice that the rl_size column is described as temporary (pending expansion of the x1 MariaDB cluster) on the SQL patch adding the table, though the discussion on the associated ticket (T187226) doesn't really reflect that.

wikiadmin@10.64.32.11(wikishared)> select rl_size from reading_list where rl_id = 2439265;
+---------+
| rl_size |
+---------+
|       1 |
+---------+
1 row in set (0.00 sec)

wikiadmin@10.64.32.11(wikishared)> select count(*) from reading_list_entry where rle_rl_id = 2439265 and not rle_deleted;
+----------+
| count(*) |
+----------+
|        8 |
+----------+
1 row in set (0.00 sec)

Oops.

Maybe a good time to run fixListSize.php. If it only fixes a few entries, eh, weird stuff always happens in a large database. If many rows need fixing, there must be some kind of mistake in the transaction handling.

LGoto triaged this task as Medium priority.Aug 12 2020, 3:43 PM
LGoto moved this task from Needs triage to Upcoming on the Product-Infrastructure-Team-Backlog board.

Good news: The number of reading_list entries that needed fixing was fairly low (787 out of 2504051).

Bad news: It appears that fixListSize.php put a lot of pressure on the DB and probably shouldn't be run in production again, at least in its current form.

Created T262575 about the DB issues caused by fixListSize.php.

A possible follow-up would be to replace rl_size = rl_size - 1 with max(rl_size = rl_size - 1, 0) in that update query to make this error somewhat self-healing.

Change 626500 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/extensions/ReadingLists@master] Prevent attempting to set rl_size to a negative value

https://gerrit.wikimedia.org/r/626500

Change 626500 merged by jenkins-bot:
[mediawiki/extensions/ReadingLists@master] Decrement rl_size on entry deletion only if it is currently positive

https://gerrit.wikimedia.org/r/626500

This can be resolved after we verify that the error no longer occurs after the patch rolls out to the wikis next week.