Page MenuHomePhabricator

wikidata RecentChangesUpdateJob::purgeExpiredRows failing
Closed, ResolvedPublic

Description

We're seeing a few errors due to large transactions on S5. Unsure yet if this is Wikidata in particular or a general MW core issue, however all the recent examples are from wikidatawiki.

Wed Feb 18 6:05:39 UTC 2015 mw1011 wikidatawiki RecentChangesUpdateJob::purgeExpiredRows 10.64.32.28 1197 \
   Multi-statement transaction required more than 'max_binlog_cache_size' bytes of storage

max_binlog_cache_size = 4GB. It's technically possible to increase this limit, but it would be unwise to do so since large transactions are going to cause replication lag. Whatever job generates this query really needs improved batching with more frequent commits.

https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/#max_binlog_cache_size

Event Timeline

Springle raised the priority of this task from to Needs Triage.
Springle updated the task description. (Show Details)
Springle added a project: Wikimedia-Rdbms.
Springle added a subscriber: Springle.
Springle added a subscriber: hoo.
gerritbot added a subscriber: gerritbot.

Change 191269 had a related patch set uploaded (by Hoo man):
Flush any implicit transactions in RecentChangesUpdateJob

https://gerrit.wikimedia.org/r/191269

Patch-For-Review

According to several comments in the code this being run after every 100th edit and it should never delete more than 100 rows at once (it deletes 100 rows per iteration in a loop).

I guess this could have happened yesterday while the job runners were paused: A lot of rows to prune gathered up during that. I've just uploaded a change which flushes all implicit transactions after purging 100 rows.

Change 191269 merged by jenkins-bot:
Flush any implicit transactions in RecentChangesUpdateJob

https://gerrit.wikimedia.org/r/191269

hoo claimed this task.

Change 192716 had a related patch set uploaded (by Aaron Schulz):
Flush any implicit transactions in RecentChangesUpdateJob

https://gerrit.wikimedia.org/r/192716

Change 192716 merged by Aaron Schulz:
Flush any implicit transactions in RecentChangesUpdateJob

https://gerrit.wikimedia.org/r/192716