WikiWho Pickle Compression Investigation - COMPLETE

TL;DR: ✓ Hypothesis CONFIRMED - Compression ratios well over 5x are easily achievable.

Results Summary

Tested 6 compression methods on sample WikiWho data (757KB uncompressed):

All methods exceeded 5x target:
- LZMA: **13.77x** compression (55KB) - Best ratio
- bz2-9: **9.90x** compression (76KB) - Best balance
- bz2-1: **9.64x** compression (78KB)
- gzip-9: **6.38x** compression (119KB)
- gzip-6: **6.18x** compression (123KB) - Recommended
- gzip-1: **5.78x** compression (131KB) - Fastest

Performance Metrics

Storage Impact for enwiki (7M articles)

• Current (uncompressed): ~5.3 TB
• With gzip-6: ~858 GB (saves 4.4 TB, 84% reduction)
• With LZMA: ~385 GB (saves 4.9 TB, 93% reduction)

My recommendation: gzip-6

Recommended for production use:
- Excellent 6.18x compression (23% above 5x target)
- Fast performance (76ms compress, 23ms decompress)
- Python standard library (no dependencies)
- Battle-tested and widely supported

Conclusion

Compression is highly feasible for supporting all wikis. Even the fastest compression method (gzip-1) achieves 5.78x, exceeding the target. Recommended approach is gzip-6 for optimal balance of compression ratio and performance.

Next Steps we can follow:

1. Integrate compression into WikiWho Pywikibot implementation (T414071)
2. Test with real WikiWho API data from wikiwho-api.wmcloud.org
3. Combine with subdirectory structure (T414087)
4. Consider testing zstandard library for potential further optimization (Once Available)

In T414075#11506347, @Xinacod wrote:

WikiWho Pickle Compression Investigation - COMPLETE

TL;DR: ✓ Hypothesis CONFIRMED - Compression ratios well over 5x are easily achievable.

This is fantastic! Thank you so much for doing this investigation :)

I think we'd gladly accept any storage optimization, provided the uncompression isn't too slow, which it doesn't seem to be per your tests.

Currently from normal direct usage, it feels like we are lacking with even rudimentary caching/lookups. The pickle files, even when small, can take multiple seconds to load. Related: T414087#11747611

Thanks for the context @MusikAnimal. To answer your question directly: yes, this cache is per-Pywikibot instance — each deployment writes its own local copies of the WikiWho data. There's no shared storage between instances.

Your point about upstreaming is well taken. The subdirectory fix addresses the immediate filesystem performance problem (7M files in a flat directory is slow regardless of what's in them), but you're right that if the WikiWho API itself can be made fast enough, local caching at this scale becomes unnecessary overhead rather than a genuine optimisation.

I'd be glad to help investigate what's causing the upstream slowness if that's useful. Happy to follow this work T414075 or wherever makes sense.

I'd say let's do T414087 and this task – both are useful! :) T414087 seems more urgent, though. I had long wondered about lookup times, it seems obvious now in hindsight that the sheer size of the directories would be a culprit. Even doing ls | wc -l (a raw count) can take minutes. I wanted to re-open T414087 and tag it with WikiWho but I also don't want to step on any toes.

The source code was externally written and inherited by CommTech and WikiEdu. I admittedly have little faith in my Python skills, hehe, but it seems we should be able to at least get r1224790 upstreamed with moderate effort? I can try to take a stab at it and/or help with reviewing. This is a priority for my team given it is a highly-voted Community wish.

To whom it may help, these changes I don't think would be part of the core WIkiWho algorithm, and instead live only in the /api directory, specifically utils_pickles.py.

For local setup of WikiWho, you can follow WIKIMEDIA_VPS_SETUP.md, but skip the import step of the "Adding languages" section (unless you really want all of those files locally!).

Update: PR is up — https://github.com/wikimedia/wikiwho_api/pull/22
Covers both the subdirectory structure (T414087) and gzip-6 compression (T414075) as a single drop-in replacement for utils_pickles.py. Backward compatible — existing uncompressed files in the old flat layout are still readable without any migration step. Tests included.

PR merged and deployed! Thank you!

@Xinacod it looks like we forgot to also update the WhoColor handler, which is what powers the visualizations seen in Who-Wrote-That. Things work fine for older pages that still use the legacy paths, but for all new pages, Who Wrote That? thinks the pickle doesn't exist. I will try to take a stab at fixing it by going off of your code, but I figure you will be much quicker, so if you see this in time please feel free to step in :)

Hi @MusikAnimal , I will take a look at the Whocolor handler and update the code, thanks.

Thank you!! I will be online for a few more hours, and can revert the deployed commit if necessary (and we'll re-deploy later).

In addition, I think we might be reading from the legacy path, but always writing to the new path:

$ ls -d /pickles/en/*/ | wc -l
16404

That's 16K directories, then looking at newly created pickles that are for old articles, we have:

$ ls -l /pickles/en/534000/
total 1268
-rw-r--r-- 1 wikiwho wikiwho  212722 Apr  1 20:53 534441.p
-rw-r--r-- 1 wikiwho wikiwho 1079788 Apr  1 21:55 534622.p
$ ls -l /pickles/en/534441.p
-rw-r--r-- 1 wikiwho wikiwho 1435057 Dec 25 07:43 /pickles/en/534441.p

Automatically converting to the new format actually seems ideal, so this is great! Only I think we should delete the old pickle when this happens (there's a pickle_delete() method for this).

For the deletion part, an alternative solution is to add a cronjob to automatically delete legacy pickle files older than N days (or whatever), and slowly over time it will eventually get down to 0. Any pickle that gets deleted that doesn't exist in the new directories will simply get re-created automatically.

Which leads me to another discovery I made (unrelated to this task):

The on-the-fly pickle generation (as proposed at T310386) isn't really feasible for very high revision-count and content-heavy articles like [[Barack Obama]]. I manually deleted the pickle file for Obama to make it regenerate a new pickle, but it seems it timed out after a few hours. This is why we used the XML dumps for the initial import, as that avoids the slower process of making requests to Wikipedia's action API. For already-existing pickles (including the new compressed pickles), we only append to the existing file, so there's no issue there. It's only when we delete pickle files solely to regenerate them that it could be a problem.

The ongoing issues with API limitations at Wikimedia might also be at play here (though as I understand it, Wikimedia Cloud Services should be exempt from said limitations).

Anyway, I am just mentioning this as we were using [[Barack Obama]] as a benchmark, and currently it is not available to WikiWho.

Good call on the cronjob approach — I'll revert the immediate deletion and add a management command instead. Also noted on the Barack Obama timeout; makes sense given the XML dump bypass was specifically for high-revision articles. Will keep that in mind going forward.

Good suggestion — reverted the immediate deletion and added a cleanup_legacy_pickles management command instead. It only deletes legacy files older than N days that already have a new-format counterpart, so nothing gets lost.
[PR #23] is up : https://github.com/wikimedia/wikiwho_api/pull/23
Covers:

1. WhoColor handler fix — restores Who-Wrote-That for new pages
2. cleanup_legacy_pickles management command — cronjob-friendly, only deletes legacy files older than N days that already have a new-format counterpart, --dry-run supported

Thanks for the swift response! I'm afraid I don't have time to review things tonight, so I've reverted the deployment. I will review #23 tomorrow and get back to you. Thank you for contributing :)

Alright! After a lot of tedious work, I managed to get our benchmark article [[Barack Obama]] back in production (gzip'd and in the new directory structure). I've re-deployed PR #22 and #23 to production, and things seem to be running swimmingly!

From my rudimentary tests, [[Barack Obama]] now loads about ~1-2 seconds faster than before. So long as it isn't slower, I'm happy :) The new directory structure by itself makes any manual maintenance of the pickle files a lot easier.

I've also manually ran the new cleanup_legacy_pickles for pickles older than 3 days. Change in the older pickle files:

A more meaningful metric would be to check the change in disk size, but I unfortunately didn't take not of that before deleting the pickles :/

Going off of T407660#11596620 however (from February), we have this:

(all values are approximate)

en grows the fastest, and we see only it and nl grew since February. This is likely expected. If I had captured the sizes correctly just before deleting pickles, I'm sure it would have been a net decrease over all.

Overall I'm quite pleased with this. Thank you so much for your contributions, @Xinacod !! :)

Thank you so much, @MusikAnimal! Really glad it's working well in production and that Barack Obama is back 🎉 The 1-2s improvement and the disk cleanup numbers are very satisfying to see at scale. Happy to have contributed!