Change Details

Currently WikiWho uses too much disk space. Investigate how compressing pickles would affect disk usage and speed (compression pickle_dump() /decompression pickle_load()). Try at least LZ4/Snappy, Gzip, Brotli, and zstd. Try to do a drop-in replacement implementation that would be easy to migrate to production. - https://github.com/wikimedia/wikiwho_api/blob/main/api/utils_pickles.py Notes - AFAIK, compression ratios of 5 or more are enough for supporting all wikis using current disk space infra. - The expected result is that the fastest compression methods should increase saving speed, as it is faster to compress data than to save it to HDD. On network drives, the effect should be even more visible.

Currently WikiWho uses too much disk space. Investigate how compressing pickles would affect disk usage and speed (compression pickle_dump() /decompression pickle_load()). Try at least LZ4/Snappy, Gzip, Brotli, and zstd. Try to do a drop-in replacement implementation that would be easy to migrate to production. - https://github.com/wikimedia/wikiwho_api/blob/main/api/utils_pickles.py Notes - AFAIK, compression ratios of 5 or more are enough for supporting all wikis using current disk space infra. - The expected result is that the fastest compression methods should increase saving speed, as it is faster to compress data than to save it to HDD. On network drives, the effect should be even more visible. - Current diskspace usage in production: T372340#11332801

Currently WikiWho uses too much disk space. Investigate how compressing pickles would affect disk usage and speed (compression pickle_dump() /decompression pickle_load()). Try at least LZ4/Snappy, Gzip, Brotli, and zstd. Try to do a drop-in replacement implementation that would be easy to migrate to production. - https://github.com/wikimedia/wikiwho_api/blob/main/api/utils_pickles.py Notes - AFAIK, compression ratios of 5 or more are enough for supporting all wikis using current disk space infra. - The expected result is that the fastest compression methods should increase saving speed, as it is faster to compress data than to save it to HDD. On network drives, the effect should be even more visible. - Current diskspace usage in production: T372340#11332801