Summary

Using pigz parallel compressor to create binary tarball would reduce archive_dist_test_debug task runtime from ~14 min to ~5 min

Long description

The majority of evergreen tasks will run only after mongo binaries have been compiled, compressed and uploaded to S3.

For amazon linux 2 variant these steps will take roughly:

• 16 min -> compile_dist_test
• 3.20 min -> archive_dist_test
• 14 min -> archive_dist_test_debug

In archive_dist_test_debug is mostly composed by two parts:

• 12 min Compressing all the binaries into a tar file
• 2 min Uploading the tarball to S3

The compression is performed using the following tar command:

/bin/tar -C build/install -T /data/mci/5098d994527fa548b1195cf0b5831e45/src/mongo-debugsymbols.tgz.filelist -czf mongo-debugsymbols.tgz

tar by default use single thread compression algorithm, this means that we are using only 1 out of the 16 cores available (we currently use amazon2-arm64-large for this task).

It is possible to simply tell tar command to use the parallel compessor pigz to make use of all the available core.

A quick experiment showed how using pigz will reduce the tar command execution time from 9.22 min to 35 seconds.