the job is batching 10 updates (e.g. 10 cirrus documents for wikibase items) and the size of these together is too much for a mysql blob :/

maybe the batch could be smaller / configurable and/or job_params could be mediumblob.

When will this bug get fix? Using Redis instead of mysql to store jobs now.
However, We can't afford a large memory, which made jobs stored in Redis been truncated when reaching memory cap (2,000 to 3,000 jobs).

It is not immediately obvious how to install Redis for this scenario. For my setup, on CentOS 7, I had to do the following:

sudo yum install redis php70u-pecl-redis
sudo systemctl start redis.service
sudo systemctl enable redis.service

Then, per the Redis setup docs, I added the following to my LocalSettings.php.
https://www.mediawiki.org/wiki/Redis#Job_queue

$wgJobTypeConf['default'] = [
    'class' => 'JobQueueRedis'
    , 'redisServer' => '127.0.0.1:6379'
    , 'redisConfig' => []
    , 'claimTTL' => 3600
    , 'daemonized' => true
];

NOTE: The daemonized parameter is required, see: https://www.mediawiki.org/w/index.php?title=Topic:Ss9ues5n7gtctppm&topic_showPostId=tbv2820lgm06ipbo#flow-post-tbv2820lgm06ipbo

At this point I expected my setup to be functional again, however it was not. I now received errors when searching. Maybe I didn't give it long enough, but I waited a few minutes and restarted apache with no luck. Finally I just re-indexed per the CirrusSearch README: https://phabricator.wikimedia.org/diffusion/ECIR/browse/master/README

I have tested that pre-existing content as well as new content now searches as expected without any errors.

Using reddis worked for me too @GFXDude2010

Is this still an issue?

Removing WMF-JobQueue as we don't use JobQueueDB in production.

Oops, used the wrong tag. Thanks!

Why does the job itself have all of the transformed text rather than just a revision/page ID and use them to derive the transformed text? I get that some metadata is not stored elsewhere and would have to go in the job.

In T218943#5067284, @EBernhardson wrote:

The two available fixes are a complete rewrite of the cirrussearch indexing retry pipeline, or changing the job queue to use a non-size limited field type [..]

Those are two ways to actually build the behaviour that the code currently pretends exists. That's cool, but the immediate fix is to not produce a fatal error.

A fatal error signifies that the code is broken, raises our error levels, and may abort a deployment, or trigger Ops pages.

In this case, however, it is known that this ability doesn't exist. Until this ability exists, the code either needs to be disabled (e.g. not deployed on Wikitech), or the code needs to handle this error and respond in some way. E.g. avoid queuing updates of this type or this size (possibly configurable), or run them differently, or to try it as today and then catch/suppress the failure - maybe logging a warning in its stead.

Until this ability exists, the code either needs to be disabled (e.g. not deployed on Wikitech), or the code needs to handle this error and respond in some way. E.g. avoid queuing updates of this type or this size (possibly configurable), or run them differently,

What you just described is option 1, rewrite the indexing retry pipeline. If you think turning off CirrusSearch on wikitech is the best alternative I can float that to wikitech list, but perhaps unsurprisingly I don't prefer that option.

E.g. avoid queuing updates of this type or this size (possibly configurable), or run them differently, or to try it as today and then catch/suppress the failure - maybe logging a warning in its stead.

Imo the JobQueue should raise an error if it's not able to save the message correctly. Since the Queue owns the way the message is serialized it's hard for an extension to determine what will be the actual size of the stored message.

Change 500481 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[mediawiki/core@master] Change job table params from blob to mediumblob

https://gerrit.wikimedia.org/r/500481

Change 500481 merged by jenkins-bot:
[mediawiki/core@master] Change job table params from blob to mediumblob

https://gerrit.wikimedia.org/r/500481