Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-61284

$project to exclude an array is expensive and scales poorly as the number of threads increases

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 6.2.0-rc0
    • Affects Version/s: 5.0.2, 4.2.17, 4.4.10
    • Component/s: None
    • None
    • Fully Compatible
    • ALL
    • QE 2021-11-15, QE 2021-11-29, QE 2021-12-13, QE 2021-12-27, QE 2022-01-10, QE 2022-04-04, QE 2022-02-07, QE 2022-02-21, QE 2022-03-07, QE 2022-03-21, QE 2022-01-24, QE 2022-04-18, QE 2022-05-02, QO 2022-05-16, QO 2022-05-30, QO 2022-06-13, QO 2022-06-27, QE 2022-10-17, QE 2022-10-31
    • 35

      Repro script attached. It creates a document of this form:

          doc = {
              other: "xxxxxxx",
              array: new Array(100000).fill("xxxxxxxxxxxxxx")
          }
      

      Then runs an aggregation that uses $project in one of two ways:

      • exclude the array field using {$project: {array: 0}}, or
      • include only the other field using {$project: {other: 1}}.

      The exclude version is much slower and also scales very poorly as the number of threads increases, even though both are computing the same result. Following table shows time per $project operation for the two cases and for single threaded vs multi-threaded, showing both

      • the poor single-threaded performance for the exclusive version, adn
      • the poor scaling; good scaling would have the multi-threaded time per operation approximately the same as the single-threaded case, up to the number of CPUs.
                                            nthreads=1    nthreads=40
          exclusive: {$project: {array: 0}}      18 ms        631 ms
          inclusive: {$project: {other: 1}}       1 ms          2 ms
      

      FTDC data and PMP (stack trace) profiling taken from a customer incident show that the scaling bottleneck is the allocator, so it appears that the exclusive projection is doing a very large number of memory allocations, which could explain both the lower single-threaded performance and the poor scaling. In the multi-threaded case a typical stack for the exclusion project shows it waiting for access to the allocator central cache in this stack:

      #0  0x0000561f79a845e6 in base::internal::SpinLockDelay(int volatile*, int, int) ()
      #1  0x0000561f79a84413 in SpinLock::SlowLock() ()
      #2  0x0000561f79a86495 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
      #3  0x0000561f79a902c2 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long)) ()
      #4  0x0000561f798ff9bd in mongo::mongoMalloc(unsigned long) ()
      #5  0x0000561f796db81d in mongo::RCString::create(mongo::StringData) ()
      #6  0x0000561f796d47f9 in mongo::ValueStorage::putString(mongo::StringData) ()
      #7  0x0000561f796d9430 in mongo::Value::Value(mongo::BSONElement const&) ()
      #8  0x0000561f796d91a7 in mongo::Value::Value(mongo::BSONElement const&) ()
      #9  0x0000561f796c4fbf in mongo::DocumentStorage::constructInCache(mongo::BSONElement const&) ()
      #10 0x0000561f796c50c3 in mongo::DocumentStorageIterator::constructInCache() ()
      #11 0x0000561f789bf160 in mongo::projection_executor::ProjectionNode::applyProjections(mongo::Document const&, mongo::MutableDocument*) const ()
      #12 0x0000561f789c1492 in mongo::projection_executor::ProjectionNode::applyToDocument(mongo::Document const&) const ()
      #13 0x0000561f789b9cc5 in mongo::projection_executor::ExclusionProjectionExecutor::applyProjection(mongo::Document const&) const ()
      #14 0x0000561f7832d3b9 in mongo::projection_executor::ProjectionExecutor::applyTransformation(mongo::Document const&) ()
      #15 0x0000561f787caa88 in mongo::DocumentSourceSingleDocumentTransformation::doGetNext() ()
      #16 0x0000561f782db0a1 in mongo::DocumentSource::getNext() ()
      #17 0x0000561f787d7edc in mongo::Pipeline::getNext() ()
      

            Assignee:
            ivan.fefer@mongodb.com Ivan Fefer
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            28 Start watching this issue

              Created:
              Updated:
              Resolved: