Maniphest T193246

JobQueueDB::doBatchPush fails with data too long for job_params when purging a template used on a few million pages
Open, HighPublic
Actions

Assigned To

None

Authored By

	Nikerabbit
	Apr 27 2018, 1:30 PM

Description

2018-04-27 13:25:05 refreshLinks Malline:Keruumerkintä table=templatelinks recursive=1 rootJobIsSelf=1 rootJobSignature=7aa83f4a5c79f8fa3ac035f3a34fdb228caa2e2d rootJobTimestamp=20180427132505 causeAction=api-purge causeAgent=WikiSysop requestId=WuMYQlVtm71TO8srrAoVRwAAAAU (id=19514342,timestamp=20180427132505) t=66 error=JobQueueError: Wikimedia\Rdbms\DBQueryError: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: INSERT  INTO `job` (job_cmd,job_namespace,job_title,job_params,job_timestamp,job_sha1,job_random) VALUES ('refreshLinks','10','Keruumerkintä','a:10:{s:9:\"recursive\";b:1;s:5:\"table\";s:13:\"templatelinks\";s:5:\"range\";a:4:{s:5:\"start\";i:18101;s:3:\"end\";b:0;s:9:\"batchSize\";i:300;s:9:\"subranges\";
[snip]
','20180427132505','r22bcs0doqwg97pwkh2evou3ekyphph','359693319')
Function: JobQueueDB::doBatchPush
Error: 1406 Data too long for column 'job_params' at row 1 ([redacted])

To reproduce set $wgSQLMode = 'TRADITIONAL'; – otherwise the insert might succeed but get truncated, in which case it will fail later on unserialize.

MediaWiki 1.31.0-alpha

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Stalled		None	T108255 Enable MariaDB/MySQL's Strict Mode
		Open		None	T193246 JobQueueDB::doBatchPush fails with data too long for job_params when purging a template used on a few million pages

Event Timeline

Nikerabbit created this task.Apr 27 2018, 1:30 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2018, 1:30 PM

Nikerabbit triaged this task as High priority.Apr 27 2018, 1:31 PM

Nikerabbit added a project: User-Nikerabbit.

Nikerabbit updated the task description. (Show Details)

I found a work-around that seems to work:

$wgUpdateRowsPerJob = 3000;
$wgUpdateRowsPerQuery = 1000;

This is 10x increase from defaults. I am not sure whether updating just one of them would be enough. I am not sure how this affects the database load either.

Nemo_bis subscribed.Apr 27 2018, 2:01 PM

The work-around doesn't seem to work very well, now I got about 20k pages updated but no more.

Nikerabbit added a parent task: T108255: Enable MariaDB/MySQL's Strict Mode.Apr 27 2018, 3:24 PM

Forgive me if I say something stupid, but could an increase of the field maximum size be possible instead (and easier?). After all, we would be losing data now, right?

To clarify, I had this happening on my own wiki, not in WMF production. I am not sure if WMF production uses the job table at all, doesn't it use Redis or EventBus or something? In logstash I only see indications of this issue happening in labweb100[12].

we == mediawiki (even if I am not a real contributor to it)

It's just a BLOB (64k) right now, so there is a good argument for increasing it. It's likely easier than fixing BacklinkJobUtils and would help Translate too (it can cause big job_params when marking long pages for translation).

Nikerabbit moved this task from Backlog to Needing on the User-Nikerabbit board.Jun 20 2018, 2:25 PM

• Vvjjkkii renamed this task from JobQueueDB::doBatchPush fails with data too long for job_params when purging a template used on a few million pages to d3daaaaaaa.Jul 1 2018, 1:13 AM

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

Yann renamed this task from d3daaaaaaa to JobQueueDB::doBatchPush fails with data too long for job_params when purging a template used on a few million pages.Jul 1 2018, 1:19 PM

Yann removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

Yann updated the task description. (Show Details)

Yann added a subscriber: Aklapper.

JobQueueDB::doBatchPush fails with data too long for job_params when purging a template used on a few million pagesOpen, HighPublicActions

Description

Related ObjectsSearch...

Event Timeline

JobQueueDB::doBatchPush fails with data too long for job_params when purging a template used on a few million pages
Open, HighPublic
Actions

Related Objects
Search...