The Snappy compressor we use for the external sorter has limitations when compressing certain index key data. Notably, it favors speed over compression. We may benefit from using a different compression algorithm to achieve higher compression ratios in exchange for potentially worse performance.

Because index builds transiently store all data on disk, they can consume much more space than the prefix-compressed data that is stored in the storage engine.

Compression is still heavily data-dependent, but having the option to use a different compression would help customers with indexes of very large data sets. This would help with initial sync, in particular, where we build all indexes at once.