The original problem: when we upgraded our MongoDB servers to 4.0.12, we started experiencing issues with building indexes. When building an index, the queries and update operations got hung during the time the index was built. To solve this problem, we would restart one of the secondary replicas and make that replica the primary one. That would completely resolve this issue.

We were hoping that after upgrading our MongoDB servers to 4.2.17, the new optimized build process would completely eliminate the problem.

However, when we tried to build an index with seven keys (not unique, with partial filter expression with two equality expressions) on a non-sharded collection with about 750k document (size - ~430 MB, storage size - ~170 MB), we observed the same behavior. The MongoDB server got very slow for the index build time. The queries got slowed down during the time the index was being built (that was about 66 seconds). The queries took several dozen seconds when usually they take less than 100 milliseconds.

We tried to restart some of the secondaries and make it primary, and the index got built within several seconds and no slowdowns were observed.

Our cluster configuration:

• shard cluster with 10 shards
• four replicas in each shard
• about 400 GB of data in storage size per shard

Replica server configuration:

• Ubuntu 16.04
• XSF
• Kernel - 4.4.0-1128-aws #142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Disable Transparent Huge disabled
• AWS m5.2xlarge (8cpu\32gb)
• SSD GP3 450 Gb
• monogo-org-server - 4.2.17