Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63095

Operations slow down during index build

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.2.17
    • Component/s: None
    • Environment:
      Ubuntu 16.04
      XSF
      Kernel - 4.4.0-1128-aws #142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
      Disable Transparent Huge disabled
      AWS m5.2xlarge (8cpu\32gb)
      SSD GP3 450 Gb
      monogo-org-server - 4.2.17
    • ALL
    • Hide

      1. Use the configuration mentioned above.
      2. Apply the following approximate amount of operations to the cluster:

      • query - 7k op/sec
      • command - 2.5k op/sec
      • getmore - 1k op/sec
      • insert - 500 op/sec
      • update - 300 op/sec
      • deletes - 100 op/sec

      3. Create an index with seven key on a collection with about 750k document (size - ~430 MB, storage size - ~170 MB).

      Show
      1. Use the configuration mentioned above. 2. Apply the following approximate amount of operations to the cluster: query - 7k op/sec command - 2.5k op/sec getmore - 1k op/sec insert - 500 op/sec update - 300 op/sec deletes - 100 op/sec 3. Create an index with seven key on a collection with about 750k document (size - ~430 MB, storage size - ~170 MB).

      The original problem: when we upgraded our MongoDB servers to 4.0.12, we started experiencing issues with building indexes. When building an index, the queries and update operations got hung during the time the index was built. To solve this problem, we would restart one of the secondary replicas and make that replica the primary one. That would completely resolve this issue.

      We were hoping that after upgrading our MongoDB servers to 4.2.17, the new optimized build process would completely eliminate the problem.

      However, when we tried to build an index with seven keys (not unique, with partial filter expression with two equality expressions) on a non-sharded collection with about 750k document (size - ~430 MB, storage size - ~170 MB), we observed the same behavior. The MongoDB server got very slow for the index build time. The queries got slowed down during the time the index was being built (that was about 66 seconds). The queries took several dozen seconds when usually they take less than 100 milliseconds.

      We tried to restart some of the secondaries and make it primary, and the index got built within several seconds and no slowdowns were observed.

      Our cluster configuration:

      • shard cluster with 10 shards
      • four replicas in each shard
      • about 400 GB of data in storage size per shard

      Replica server configuration:

      • Ubuntu 16.04
      • XSF
      • Kernel - 4.4.0-1128-aws #142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
      • Disable Transparent Huge disabled
      • AWS m5.2xlarge (8cpu\32gb)
      • SSD GP3 450 Gb
      • monogo-org-server - 4.2.17

        1. Screen Shot 2022-03-04 at 6.00.57 PM.png
          Screen Shot 2022-03-04 at 6.00.57 PM.png
          679 kB
        2. diagnostic.data.zip
          59.65 MB

            Assignee:
            edwin.zhou@mongodb.com Edwin Zhou
            Reporter:
            vladimirred456@gmail.com Vladimir Beliakov
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: