Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-82410

DocumentSourceListSearchIndexes should hold owned copy of command object

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 7.2.0-rc0, 6.0.12, 7.0.4
    • Affects Version/s: 7.0.2, 6.0.11, 7.1.0
    • Component/s: None
    • Fully Compatible
    • ALL
    • v7.1, v7.0, v6.0
    • QI 2023-10-30

      Issue summary

      Symptoms & User-visible impact:

      Sharded clusters on versions 6.0.7-6.0.11 and 7.0.0-7.0.3 can see mongod process crashes as a result of a use-after-free memory issue in the $listSearchIndexes aggregation stage.

      Key Diagnostics:

      • Server mongod process may crash with an Invalid Access at address error and Segmentation fault.
      • Log file may contain an Invariant failure referring to the file src/mongo/util/future_impl.h, such as:
      msg":"Invariant failure","attr":{"expr":"!callback","file":"src/mongo/util/future_impl.h","line":443}}
      
      • Segmentation faults with backtraces including DocumentSourceListSearchIndexes, such as:
      _ZN5mongo18stack_trace_detail12_GLOBAL__N_117getStackTraceImplERKNS1_7OptionsE.constprop.205
      _ZN5mongo15printStackTraceEv
      abruptQuitWithAddrSignal
      _L_unlock_13
      __memcpy_ssse3_back
      _ZN5mongo31DocumentSourceListSearchIndexes9doGetNextEv
      _ZN5mongo14DocumentSource7getNextEv
      _ZN5mongo8Pipeline7getNextEv
      _ZN5mongo20PlanExecutorPipeline11_tryGetNextEv
      _ZN5mongo20PlanExecutorPipeline8_getNextEv
      _ZN5mongo20PlanExecutorPipeline15getNextDocumentEPNS_8DocumentEPNS_8RecordIdE
      _ZN5mongo20PlanExecutorPipeline7getNextEPNS_7BSONObjEPNS_8RecordIdE
      _ZN5mongo12_GLOBAL__N_110GetMoreCmd10Invocation28acquireLocksAndIterateCursorEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceERNS_15ClientCursorPinEPNS_5CurOpE
      _ZN5mongo12_GLOBAL__N_110GetMoreCmd10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE
      _ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE
      _ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE
      _ZN5mongo12_GLOBAL__N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE
      _ZN5mongo12_GLOBAL__N_114RunCommandImpl11_runCommandEv
      _ZN5mongo12_GLOBAL__N_132RunCommandAndWaitForWriteConcern24_runCommandWithFailPointEv
      _ZN5mongo12_GLOBAL__N_132RunCommandAndWaitForWriteConcern8_runImplEv
      _ZN5mongo12_GLOBAL__N_114RunCommandImpl3runEv
      _ZN5mongo12_GLOBAL__N_119ExecCommandDatabase12_commandExecEv
      _ZZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv
      _ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_
      _ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv
      _ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1_
      _ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_
      _ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv
      _ZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE
      _ZN5mongo12_GLOBAL__N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE
      _ZN5mongo12_GLOBAL__N_115CommandOpRunner3runEv
      _ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE
      _ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE
      _ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv
      _ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE
      _ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_
      _ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_
      _ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv
      _ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv
      _ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv
      _ZN5mongo12_GLOBAL__N_17runFuncEPv
      start_thread
      clone
      
      • Users who use Compass may have observed an increase in failures as a result of this issue given the 1.40.0 release of Compass which started offering support for Search indexes.
      • In some situations, the server mongod process may become deadlocked after failing to process the abort signal due to SERVER-82459.

      Root cause:

      A use-after-free error within the $listSearchIndexes aggregation pipeline stage resulted in a data structure pointing to a freed location within Heap memory. This allows for a wide variety of failure scenarios where threads unrelated to the operation which issued the $listSearchIndexes operation could terminate the server process with Invalid Access segmentation faults.

      Fix or Remediations:

      • Upgrade to 6.0.12 or 7.0.4 when available.
      • Reduce or stop the use of the $listSearchIndexes aggregation stage.
        • This may require using a pre-1.40.0 version of Compass until updating to a version of MongoDB containing a fix for this issue.

      Previous Description

      Current copy is unowned and at risk of being destroyed while held

            Assignee:
            james.wahlin@mongodb.com James Wahlin
            Reporter:
            james.wahlin@mongodb.com James Wahlin
            Votes:
            0 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: