Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-33284

Garbage collection of DBCommandCursor created via replica set connection may fail to run killCursors command

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.0
    • Component/s: Shell
    • None
    • ALL
    • v3.2

      The changes from bdf345a as part of SERVER-22347 only addressed the issue of routing a "getMore" or "killCursors" command when the operation was performed explicitly by the user. For cursors where the user manually constructs a DBCommandCursor instance by writing new DBCommandCursor(db.getMongo(), cmdResult) rather than new DBCommandCursor(cmdResult._mongo, cmdResult), the mongo shell will attempt to later run the "killCursors" command on the replica set connection string and hit this verify failure.

      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.361-0500 2018-02-12T19:35:59.357-0500 I -        [thread1]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.361-0500
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.361-0500 ***aborting after verify() failure as this is a debug/test build
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.361-0500
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.361-0500
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.368-0500 2018-02-12T19:35:59.367-0500 F -        [thread1] Got signal: 6 (Aborted).
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.368-0500
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.368-0500  0x7fe6cda2f0b1 0x7fe6cda2e2c9 0x7fe6cda2e7ad 0x7fe6cc48e330 0x7fe6cc0ebc37 0x7fe6cc0ef028 0x7fe6cd6e58ac 0x7fe6cd78fc7c 0x7fe6cd92e7ef 0x7fe6cdf76932 0x7fe6cdf59819 0x7fe6cdf76c2a 0x7fe6cdf5ca11 0x7fe6cdf658e9 0x7fe6cdf68bb1 0x7fe6cdf6c32f 0x7fe6cdf6cf99 0x7fe6cdf6d382 0x7fe6cdf6d68d 0x7fe6cdf24e9c 0x7fe6cd93ce01 0x7fe6cd93cf21 0x7fe6cd96d834 0x7fe6cd9231cc 0x7fe6ce2c1760 0x7fe6cc486184 0x7fe6cc1b303d
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.368-0500 ----- BEGIN BACKTRACE -----
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.369-0500 {"backtrace":[{"b":"7FE6CD1E9000","o":"8460B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7FE6CD1E9000","o":"8452C9"},{"b":"7FE6CD1E9000","o":"8457AD"},{"b":"7FE6CC47E000","o":"10330"},{"b":"7FE6CC0B5000","o":"36C37","s":"gsignal"},{"b":"7FE6CC0B5000","o":"3A028","s":"abort"},{"b":"7FE6CD1E9000","o":"4FC8AC","s":"_ZN5mongo12verifyFailedEPKcS1_j"},{"b":"7FE6CD1E9000","o":"5A6C7C"},{"b":"7FE6CD1E9000","o":"7457EF","s":"_ZN5mongo5mozjs16CursorHandleInfo8finalizeEP8JSFreeOpP8JSObject"},{"b":"7FE6CD1E9000","o":"D8D932","s":"_ZN2js2gc5Arena8finalizeI8JSObjectEEmPNS_6FreeOpENS0_9AllocKindEm"},{"b":"7FE6CD1E9000","o":"D70819"},{"b":"7FE6CD1E9000","o":"D8DC2A","s":"_ZN2js2gc10ArenaLists16forceFinalizeNowEPNS_6FreeOpENS0_9AllocKindENS1_14KeepArenasEnumEPPNS0_11ArenaHeaderE"},{"b":"7FE6CD1E9000","o":"D73A11","s":"_ZN2js2gc10ArenaLists30queueForegroundObjectsForSweepEPNS_6FreeOpE"},{"b":"7FE6CD1E9000","o":"D7C8E9","s":"_ZN2js2gc9GCRuntime22beginSweepingZoneGroupEv"},{"b":"7FE6CD1E9000","o":"D7FBB1","s":"_ZN2js2gc9GCRuntime15beginSweepPhaseEb"},{"b":"7FE6CD1E9000","o":"D8332F","s":"_ZN2js2gc9GCRuntime23incrementalCollectSliceERNS_11SliceBudgetEN2JS8gcreason6ReasonE"},{"b":"7FE6CD1E9000","o":"D83F99","s":"_ZN2js2gc9GCRuntime7gcCycleEbRNS_11SliceBudgetEN2JS8gcreason6ReasonE"},{"b":"7FE6CD1E9000","o":"D84382","s":"_ZN2js2gc9GCRuntime7collectEbNS_11SliceBudgetEN2JS8gcreason6ReasonE"},{"b":"7FE6CD1E9000","o":"D8468D","s":"_ZN2js2gc9GCRuntime2gcE18JSGCInvocationKindN2JS8gcreason6ReasonE"},{"b":"7FE6CD1E9000","o":"D3BE9C","s":"_ZN2js14DestroyContextEP9JSContextNS_18DestroyContextModeE"},{"b":"7FE6CD1E9000","o":"753E01","s":"_ZN5mongo5mozjs14MozJSImplScopeD2Ev"},{"b":"7FE6CD1E9000","o":"753F21","s":"_ZN5mongo5mozjs14MozJSImplScopeD0Ev"},{"b":"7FE6CD1E9000","o":"784834","s":"_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv"},{"b":"7FE6CD1E9000","o":"73A1CC","s":"_ZN4nspr6Thread13ThreadRoutineEPv"},{"b":"7FE6CD1E9000","o":"10D8760"},{"b":"7FE6CC47E000","o":"8184"},{"b":"7FE6CC0B5000","o":"FE03D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.13-7-g9aea96e", "gitVersion" : "9aea96e4bfa630b36ba39f947de8786b18a1b0a8", "compiledModules" : [ "ninja" ], "uname" : { "sysname" : "Linux", "release" : "3.13.0-37-generic", "version" : "#64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "b" : "7FE6CD1E9000", "elfType" : 3, "buildId" : "FFFE629CDA281873A8FB6345205D3D36B7CD42AA" }, { "b" : "7FFF4A0FE000", "elfType" : 3, "buildId" : "0074678E5FFFF79F46C476077E67057161772F37" }, { "b" : "7FE6CCDBC000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "4F930712D3609C93E380E5BE5DF73E7AD273531C" }, { "b" : "7FE6CCBB8000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "034D6A4EE9DCAB4A34ABD644345CBBB42DC63088" }, { "b" : "7FE6CC8B2000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "300C7884CDEB5667BEA2357D2B8E7A76397562D6" }, { "b" : "7FE6CC69C000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7FE6CC47E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "F64B8AD471FBA1B7A3A64EFB01551E694975E1F7" }, { "b" : "7FE6CC0B5000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "D9A10B8EF90300628DD0A3A535106967714D7328" }, { "b" : "7FE6CCFC4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2CA513EDC89C7BC06EC183D1A3A03CC0F606319C" } ] }}
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.369-0500  mongo(_ZN5mongo15printStackTraceERSo+0x41) [0x7fe6cda2f0b1]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.369-0500  mongo(+0x8452C9) [0x7fe6cda2e2c9]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.369-0500  mongo(+0x8457AD) [0x7fe6cda2e7ad]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.369-0500  libpthread.so.0(+0x10330) [0x7fe6cc48e330]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.369-0500  libc.so.6(gsignal+0x37) [0x7fe6cc0ebc37]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  libc.so.6(abort+0x148) [0x7fe6cc0ef028]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN5mongo12verifyFailedEPKcS1_j+0x23E) [0x7fe6cd6e58ac]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(+0x5A6C7C) [0x7fe6cd78fc7c]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN5mongo5mozjs16CursorHandleInfo8finalizeEP8JSFreeOpP8JSObject+0xAF) [0x7fe6cd92e7ef]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN2js2gc5Arena8finalizeI8JSObjectEEmPNS_6FreeOpENS0_9AllocKindEm+0x232) [0x7fe6cdf76932]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(+0xD70819) [0x7fe6cdf59819]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN2js2gc10ArenaLists16forceFinalizeNowEPNS_6FreeOpENS0_9AllocKindENS1_14KeepArenasEnumEPPNS0_11ArenaHeaderE+0xFA) [0x7fe6cdf76c2a]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN2js2gc10ArenaLists30queueForegroundObjectsForSweepEPNS_6FreeOpE+0x81) [0x7fe6cdf5ca11]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN2js2gc9GCRuntime22beginSweepingZoneGroupEv+0xDD9) [0x7fe6cdf658e9]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.370-0500  mongo(_ZN2js2gc9GCRuntime15beginSweepPhaseEb+0x211) [0x7fe6cdf68bb1]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN2js2gc9GCRuntime23incrementalCollectSliceERNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x2DF) [0x7fe6cdf6c32f]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN2js2gc9GCRuntime7gcCycleEbRNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x179) [0x7fe6cdf6cf99]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN2js2gc9GCRuntime7collectEbNS_11SliceBudgetEN2JS8gcreason6ReasonE+0x1B2) [0x7fe6cdf6d382]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN2js2gc9GCRuntime2gcE18JSGCInvocationKindN2JS8gcreason6ReasonE+0x2D) [0x7fe6cdf6d68d]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN2js14DestroyContextEP9JSContextNS_18DestroyContextModeE+0x13C) [0x7fe6cdf24e9c]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN5mongo5mozjs14MozJSImplScopeD2Ev+0x13D1) [0x7fe6cd93ce01]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN5mongo5mozjs14MozJSImplScopeD0Ev+0x11) [0x7fe6cd93cf21]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN5mongo5mozjs15MozJSProxyScope10implThreadEPv+0x1A4) [0x7fe6cd96d834]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(_ZN4nspr6Thread13ThreadRoutineEPv+0x1C) [0x7fe6cd9231cc]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  mongo(+0x10D8760) [0x7fe6ce2c1760]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.371-0500  libpthread.so.0(+0x8184) [0x7fe6cc486184]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.372-0500  libc.so.6(clone+0x6D) [0x7fe6cc1b303d]
      [js_test:replica_set_connection_killcursors_gc] 2018-02-12T19:35:59.372-0500 -----  END BACKTRACE  -----
      

      Note: It is worth mentioning that the changes from f23002e as part of SERVER-30848 has already addressed this issue on the 3.6 and master branches.

            Assignee:
            mira.carey@mongodb.com Mira Carey
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: