I've been checking the benchmarks on compression of text. Given that we store each revision separately in external store, the biggest contributor to the growth of ES is wikidata (given the rate of edits on the wiki). Any small improvements could have a decent impact on the storage capacity and lifetime.

I picked 500 pages in a couple of wikis, compressed with different algorithms and then averaged them. The results for normal wikis are a bit all over the place (the ration being higher, the better):

zhwiki:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 2.0011580657564485
gzip            : 1.9857647546756871
bz2 (Bzip2)     : 1.926178296495638
lzma (XZ)       : 2.0368913021126174

ruwiki:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 2.863096812245039
gzip            : 2.848671946252398
bz2 (Bzip2)     : 3.1432688715395587
lzma (XZ)       : 2.974737944477271

enwiki:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 2.427702836086557
gzip            : 2.4112390670333816
bz2 (Bzip2)     : 2.3381988304527472
lzma (XZ)       : 2.392846695043902

arwiki:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 2.705738147892982
gzip            : 2.6791698087892235
bz2 (Bzip2)     : 2.8576198567047317
lzma (XZ)       : 2.6826789430837596

But for Wikidata, no matter how many times I repeat it, lzma always wins:

Try 1:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 5.069681227948362
gzip            : 5.072687651518624
bz2 (Bzip2)     : 4.609058556764682
lzma (XZ)       : 5.318371805033433

Try II:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 5.123151927192675
gzip            : 5.122254116011882
bz2 (Bzip2)     : 4.6121314372912305
lzma (XZ)       : 5.3488774386100895

Try III:
Algorithm        | Ratio     
--------------------------------------------------------------
zlib (Deflate)  : 4.934902621719569
gzip            : 4.932698731722822
bz2 (Bzip2)     : 4.4532215294494355
lzma (XZ)       : 5.127213408027472

And it has a decent gap with gzip. I wonder if we can make the flag configurable and set it to wikidata to be lzma instead of gzip. There shouldn't be any issues with existing rows, they will just stay there with flag of gzip in the db.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Use maximum compression level in SqlBlobStore and SqlBagOStuff	Ladsgroup	mediawiki/core	wmf/1.47.0-wmf.14	+2 -2
	Use maximum compression level in SqlBlobStore and SqlBagOStuff	Ladsgroup	mediawiki/core	master	+2 -2

Customize query in gerrit

Related Objects

Mentioned In: T434392: Package and deploy php-zstd
Mentioned Here: T434392: Package and deploy php-zstd

Event Timeline

Ladsgroup created this task.Jun 8 2026, 1:05 AM

Restricted Application added subscribers: hubaishan, Stang, Aklapper. · View Herald TranscriptJun 8 2026, 1:05 AM

There is a php extension for lzma compression but doesn't seem to be widely used: https://github.com/codemasher/php-ext-xz I can't find it in debian packages either: https://packages.debian.org/search?suite=stable&section=all&arch=any&searchon=names&keywords=php8-xz this could complicate things :(

Restricted Application added a subscriber: alaa. · View Herald TranscriptJun 8 2026, 1:17 AM

• karapayneWMDE subscribed.Jul 10 2026, 12:43 PM

Another random idea. We could look into zstd, the new cool kid in town (developed by Meta) and it's already packaged. My tests show a decent improvements over gzip and deflate but at max compression level it's rather slow (note that compression level usually translates to slower compression but not necessarily slower decompression). Worth looking into.

Okay, we have also the compression level knob too. So I made another set of benchmarks:

Wikidata 500 random pages, first run:

Algorithm        | Ratio      | Time      
--------------------------------------------------------------
zlib (Deflate) 1: 4.474367674584579 0.03425884246826172
zlib (Deflate) 3: 4.626636094032429 0.034479379653930664
zlib (Deflate) 5: 4.958753533763209 0.05431675910949707
zlib (Deflate) 7: 5.015500664036064 0.06850552558898926
zlib (Deflate) 9: 5.026310099502973 0.08515048027038574
gzip 1          : 4.450586225135544 0.032232046127319336
gzip 3          : 4.601547442964843 0.03478407859802246
gzip 5          : 4.930936686623036 0.05526161193847656
gzip 7          : 4.987242926798398 0.06894850730895996
gzip 9          : 4.997988337529218 0.08405089378356934
zstd 5          : 5.125873399736087 0.034752607345581055
zstd 10         : 5.263509709478594 0.07099437713623047
zstd 13         : 5.265768010737612 0.27332639694213867
zstd 17         : 5.379248656074199 0.7080748081207275
zstd 19         : 5.394163386928446 1.1054208278656006
zstd 22         : 5.401113391758018 5.469714879989624

Wikidata 500 random pages, second run:
Algorithm        | Ratio      | Time      
--------------------------------------------------------------
zlib (Deflate) 1: 4.5097563006179096 0.03454732894897461
zlib (Deflate) 3: 4.658110819603024 0.03333902359008789
zlib (Deflate) 5: 4.990863353387424 0.05137228965759277
zlib (Deflate) 7: 5.050937968320975 0.0667421817779541
zlib (Deflate) 9: 5.06401848300169 0.08234238624572754
gzip 1          : 4.485468210547386 0.03169989585876465
gzip 3          : 4.632482103841804 0.03407740592956543
gzip 5          : 4.962456742705894 0.05180478096008301
gzip 7          : 5.022071947883201 0.06734085083007812
gzip 9          : 5.035078211504532 0.08267593383789062
zstd 5          : 5.184067520626032 0.031891822814941406
zstd 10         : 5.30968583818919 0.07234907150268555
zstd 13         : 5.309237342150649 0.2779653072357178
zstd 17         : 5.418811563151901 0.7129464149475098
zstd 19         : 5.434171427731077 1.0659914016723633
zstd 22         : 5.442171440732256 5.246222496032715

And enwiki for comparison:

First run:
Algorithm        | Ratio      | Time      
--------------------------------------------------------------
zlib (Deflate) 1: 2.261899614623446 0.044068098068237305
zlib (Deflate) 3: 2.319258913031901 0.04637718200683594
zlib (Deflate) 5: 2.4197677465019463 0.0645904541015625
zlib (Deflate) 7: 2.429518944756461 0.08003067970275879
zlib (Deflate) 9: 2.431699325353053 0.0955054759979248
gzip 1          : 2.2435962376748186 0.04004359245300293
gzip 3          : 2.3004524640167276 0.04915618896484375
gzip 5          : 2.3999625147090384 0.0644216537475586
gzip 7          : 2.4096467558442387 0.0803072452545166
gzip 9          : 2.411815829072911 0.09485101699829102
zstd 5          : 2.385864178338882 0.040975332260131836
zstd 10         : 2.40617156084392 0.08073782920837402
zstd 13         : 2.4334816013677876 0.23806285858154297
zstd 17         : 2.4560830263292854 0.5688936710357666
zstd 19         : 2.4570686829834005 0.8288838863372803
zstd 22         : 2.45709696429133 0.8443603515625
Second run:
Algorithm        | Ratio      | Time      
--------------------------------------------------------------
zlib (Deflate) 1: 2.2412729125127324 0.04558157920837402
zlib (Deflate) 3: 2.2961713609210475 0.05092620849609375
zlib (Deflate) 5: 2.3944626055447586 0.07166457176208496
zlib (Deflate) 7: 2.4024346294722374 0.09027266502380371
zlib (Deflate) 9: 2.403743791528663 0.09609055519104004
gzip 1          : 2.2229864531760377 0.04330182075500488
gzip 3          : 2.2774229997818773 0.051061391830444336
gzip 5          : 2.374736513642953 0.07199716567993164
gzip 7          : 2.382648043167001 0.0904383659362793
gzip 9          : 2.3839478442905326 0.09593939781188965
zstd 5          : 2.354684411408755 0.04309344291687012
zstd 10         : 2.374501571673827 0.08589482307434082
zstd 13         : 2.4036537768355126 0.24552345275878906
zstd 17         : 2.422877624519167 0.5500578880310059
zstd 19         : 2.4238513253607374 0.7869982719421387
zstd 22         : 2.4238578592195372 0.795494794845581

Some random notes:

We currently use deflate in production, it uses gzip library which confused me but it's deflate.
Compression level impacts compression time but the impact in decompression time is much smaller (and non-existent in some algorithms).
zstd compression levels 20 and above are considered "ultra" and are extremely slow by design.

So I suggest using zstd level 19 which in wikidata is showing the improvement of 10% in compression ratio over the strongest compression of deflate which would give us months more runway in each new ES cluster and while it's slower, the slowdown is 1s over 500 pages (averaging out to 2ms) which is slowdown we definitely can absorb.

Change #1323809 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@master] Use maximum compression level in SqlBlobStore and SqlBagOStuff

https://gerrit.wikimedia.org/r/1323809

gerritbot added a project: Patch-For-Review.Aug 9 2026, 2:37 PM

T434392: Package and deploy php-zstd

When you go from zlib level 6 to level 9, the maximum chain length increases by a factor of 32. I think it would be possible to attack the hash function it uses to get quadratic performance in the chain length, for a worst case slowdown of 1024x. Still it would be hard to turn that into a DoS if that only takes it from microseconds to milliseconds. There are easier hashtables to attack. But it's something to keep in mind if you want to switch to a library which scales all the parameters far beyond that without seeding the hash functions. Even for non-malicious input, this zlib benchmark shows a strong dependence on the input data: a 5x slowdown for "html" and 33x for "pngpixels" when going from gzip -5 to gzip -9.

Change #1323809 merged by jenkins-bot:

[mediawiki/core@master] Use maximum compression level in SqlBlobStore and SqlBagOStuff

https://gerrit.wikimedia.org/r/1323809

Maintenance_bot removed a project: Patch-For-Review.Aug 10 2026, 1:31 AM

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.15; 2026-08-11).Aug 10 2026, 2:00 AM

Krinkle added projects: MediaWiki-libs-BagOStuff, MediaWiki-Core-Platform-Team (Radar).Aug 10 2026, 2:57 AM

Thank you for the detailed look at it. I keep an eye on things to make sure nothing breaks.

Change #1324283 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@wmf/1.47.0-wmf.14] Use maximum compression level in SqlBlobStore and SqlBagOStuff

https://gerrit.wikimedia.org/r/1324283

gerritbot added a project: Patch-For-Review.Aug 11 2026, 10:48 AM

Change #1324283 merged by jenkins-bot:

[mediawiki/core@wmf/1.47.0-wmf.14] Use maximum compression level in SqlBlobStore and SqlBagOStuff

https://gerrit.wikimedia.org/r/1324283

Maintenance_bot removed a project: Patch-For-Review.Tue, Aug 11, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-11T11:48:42Z] <ladsgroup@deploy1003> Started scap sync-world: Backport for [[gerrit:1324283|Use maximum compression level in SqlBlobStore and SqlBagOStuff (T428377)]]

Mentioned in SAL (#wikimedia-operations) [2026-08-11T11:52:28Z] <ladsgroup@deploy1003> ladsgroup: Backport for [[gerrit:1324283|Use maximum compression level in SqlBlobStore and SqlBagOStuff (T428377)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-08-11T12:00:19Z] <ladsgroup@deploy1003> Finished scap sync-world: Backport for [[gerrit:1324283|Use maximum compression level in SqlBlobStore and SqlBagOStuff (T428377)]] (duration: 11m 37s)

ReleaseTaggerBot edited projects, added: MW-1.47-notes (1.47.0-wmf.14; 2026-08-04); removed: MW-1.47-notes (1.47.0-wmf.15; 2026-08-11).Tue, Aug 11, 12:00 PM

Consider using lzma compression for storing wikidata entries in external storageOpen, Needs TriagePublicActions

Description

Details

Related Objects

Event Timeline

Consider using lzma compression for storing wikidata entries in external storage
Open, Needs TriagePublic
Actions