Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-51386

Mongo 4.4.1 Crashes often

    • Type: Icon: Question Question
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.4.1
    • Component/s: WiredTiger
    • None
    • Storage - Ra 2020-10-19, Storage - Ra 2020-11-02
    • 0

      I have a replica set where one or more members crash almost daily.  I finally caught a stack trace of the crash.  I'm not sure how to fix this, or interpret the stack trace.

      This is 4.4.1 on Ubuntu 20, with proper ulimits set.  I also have the following sysctls set:

      {{vm.swappiness=1
      }}{{net.ipv4.tcp_keepalive_probes = 6
      }}{{net.core.somaxconn=4096
      }}{{net.ipv4.tcp_fin_timeout=30
      }}{{net.ipv4.tcp_keepalive_intvl=30
      }}{{net.ipv4.tcp_keepalive_time=120
      }}{{net.ipv4.tcp_max_syn_backlog=4096
      }}{{vm.max_map_count=9999999
      }}{{net.ipv4.ip_local_port_range = 1024 65530
      }}{{vm.zone_reclaim_mode=0
      }}{{vm.dirty_ratio = 15
      }}{{vm.dirty_background_ratio = 5}}
      

       
      Backtrace:

      {"t":\{"$date":"2020-10-05T22:25:46.854+00:00"},"s":"F",  "c":"-",        "id":23083,   "ctx":"conn503635","msg":"Invariant failure","attr":\{"expr":"ret","error":"UnknownError: -31803: WT_NOTFOUND: item not found","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":1600}}
      
      {"t":\{"$date":"2020-10-05T22:25:46.856+00:00"},"s":"F",  "c":"-",        "id":23084,   "ctx":"conn503635","msg":"\n\n***aborting after invariant() failure\n\n"}
      
      {"t":\{"$date":"2020-10-05T22:25:46.857+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"conn503635","msg":"Writing fatal message","attr":\{"message":"Got signal: 6 (Aborted).\n"}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"conn503635","msg":"BACKTRACE: \{bt}","attr":\{"bt":{"backtrace":[{"a":"55FF176683B1","b":"55FF14869000","o":"2DFF3B1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"},\{"a":"55FF176699E9","b":"55FF14869000","o":"2E009E9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},\{"a":"55FF17667246","b":"55FF14869000","o":"2DFE246","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},\{"a":"7F938DF733C0","b":"7F938DF5E000","o":"153C0","s":"funlockfile","s+":"60"},\{"a":"7F938DDB218B","b":"7F938DD6C000","o":"4618B","s":"gsignal","s+":"CB"},\{"a":"7F938DD91859","b":"7F938DD6C000","o":"25859","s":"abort","s+":"12B"},\{"a":"55FF157CCEE1","b":"55FF14869000","o":"F63EE1","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j","s+":"179"},\{"a":"55FF154EEB5D","b":"55FF14869000","o":"C85B5D","s":"_ZN5mongo21WiredTigerRecordStore12updateRecordEPNS_16OperationContextERKNS_8RecordIdEPKci.cold.1749","s+":"99"},\{"a":"55FF161F4E5F","b":"55FF14869000","o":"198BE5F","s":"_ZN5mongo14CollectionImpl14updateDocumentEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS5_bPNS_7OpDebugEPNS_20CollectionUpdateArgsE","s+":"1DF"},\{"a":"55FF1639A4BF","b":"55FF14869000","o":"1B314BF","s":"_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE","s+":"A3F"},\{"a":"55FF1639AEC1","b":"55FF14869000","o":"1B31EC1","s":"_ZN5mongo11UpdateStage6doWorkEPm","s+":"361"},\{"a":"55FF1639D06B","b":"55FF14869000","o":"1B3406B","s":"_ZN5mongo11UpsertStage6doWorkEPm","s+":"7B"},\{"a":"55FF1637FBA8","b":"55FF14869000","o":"1B16BA8","s":"_ZN5mongo9PlanStage4workEPm","s+":"68"},\{"a":"55FF163C41C2","b":"55FF14869000","o":"1B5B1C2","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"222"},\{"a":"55FF163C4C0B","b":"55FF14869000","o":"1B5BC0B","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_8DocumentEPNS_8RecordIdE","s+":"4B"},\{"a":"55FF163C4D4D","b":"55FF14869000","o":"1B5BD4D","s":"_ZN5mongo16PlanExecutorImpl11executePlanEv","s+":"4D"},\{"a":"55FF160E8E80","b":"55FF14869000","o":"187FE80","s":"_ZN5mongo14performUpdatesEPNS_16OperationContextERKNS_9write_ops6UpdateE","s+":"E60"},\{"a":"55FF1604E3A6","b":"55FF14869000","o":"17E53A6","s":"_ZNK5mongo12_GLOBAL__N_19CmdUpdate10Invocation7runImplEPNS_16OperationContextERNS_14BSONObjBuilderE","s+":"46"},\{"a":"55FF1604BEED","b":"55FF14869000","o":"17E2EED","s":"_ZN5mongo12_GLOBAL__N_112WriteCommand14InvocationBase3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"15D"},\{"a":"55FF165AA59F","b":"55FF14869000","o":"1D4159F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"},\{"a":"55FF15D5FA81","b":"55FF14869000","o":"14F6A81","s":"_ZN5mongo12_GLOBAL__N_127invokeWithSessionCheckedOutEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationERKNS_30OperationSessionInfoFromClientEPNS_3rpc21ReplyBuilderInterfaceE","s+":"221"},\{"a":"55FF15D609F8","b":"55FF14869000","o":"14F79F8","s":"_ZN5mongo12_GLOBAL__N_114runCommandImplEPNS_16OperationContextEPNS_17CommandInvocationERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceENS_11LogicalTimeERKNS_23ServiceEntryPointCommon5HooksEPNS_14BSONObjBuilderERKNS_30OperationSessionInfoFromClientE","s+":"7C8"},\{"a":"55FF15D631C9","b":"55FF14869000","o":"14FA1C9","s":"_ZN5mongo12_GLOBAL__N_119execCommandDatabaseEPNS_16OperationContextEPNS_7CommandERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceERKNS_23ServiceEntryPointCommon5HooksE","s+":"11B9"},\{"a":"55FF15D6448B","b":"55FF14869000","o":"14FB48B","s":"_ZN5mongo12_GLOBAL__N_116receivedCommandsEPNS_16OperationContextERKNS_7MessageERKNS_23ServiceEntryPointCommon5HooksE","s+":"62B"},\{"a":"55FF15D650CD","b":"55FF14869000","o":"14FC0CD","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE","s+":"50D"},\{"a":"55FF15D5314C","b":"55FF14869000","o":"14EA14C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"3C"},\{"a":"55FF15D5D65C","b":"55FF14869000","o":"14F465C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE","s+":"FC"},\{"a":"55FF15D5B505","b":"55FF14869000","o":"14F2505","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"125"},\{"a":"55FF15D5C4B6","b":"55FF14869000","o":"14F34B6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"55FF1702E042","b":"55FF14869000","o":"27C5042","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE","s+":"182"},\{"a":"55FF15D5A91B","b":"55FF14869000","o":"14F191B","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE","s+":"DB"},\{"a":"55FF15D5BC7D","b":"55FF14869000","o":"14F2C7D","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE","s+":"6AD"},\{"a":"55FF15D5BD60","b":"55FF14869000","o":"14F2D60","s":"_ZN5mongo14future_details4callIRZNS_19ServiceStateMachine14_sourceMessageENS2_11ThreadGuardEEUlNS_10StatusWithINS_7MessageEEEE0_S6_EEDaOT_OT0_","s+":"60"},\{"a":"55FF15D5BFE5","b":"55FF14869000","o":"14F2FE5","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE","s+":"145"},\{"a":"55FF15D5B57A","b":"55FF14869000","o":"14F257A","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"19A"},\{"a":"55FF15D5C4B6","b":"55FF14869000","o":"14F34B6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"55FF1702E6A8","b":"55FF14869000","o":"27C56A8","s":"_ZNSt17_Function_handlerIFvvEZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIS0_ENS2_15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameEEUlvE0_E9_M_invokeERKSt9_Any_data","s+":"B8"},\{"a":"55FF173828A6","b":"55FF14869000","o":"2B198A6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo25launchServiceWorkerThreadESt8functionIS0_EEUlvE1_E9_M_invokeERKSt9_Any_data","s+":"56"},\{"a":"55FF17382914","b":"55FF14869000","o":"2B19914","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"14"},\{"a":"7F938DF67609","b":"7F938DF5E000","o":"9609","s":"start_thread","s+":"D9"},\{"a":"7F938DE8E293","b":"7F938DD6C000","o":"122293","s":"clone","s+":"43"}],"processInfo":\{"mongodbVersion":"4.4.1","gitVersion":"ad91a93a5a31e175f5cbf8c69561e788bbc55ce1","compiledModules":["enterprise"],"uname":{"sysname":"Linux","release":"5.4.0-48-generic","version":"#52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020","machine":"x86_64"},"somap":[\{"b":"55FF14869000","elfType":3,"buildId":"475B9C841EE61B5065AA30B170D19A8F7C3680D8"},\{"b":"7F938DF5E000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"4FC5FC33F4429136A494C640B113D76F610E4ABC"},\{"b":"7F938DD6C000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"F3FF3FDA80B817C464A56EED59FF09DC864EAEB0"}]}}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF176683B1","b":"55FF14869000","o":"2DFF3B1","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1E1"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF176699E9","b":"55FF14869000","o":"2E009E9","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF17667246","b":"55FF14869000","o":"2DFE246","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"7F938DF733C0","b":"7F938DF5E000","o":"153C0","s":"funlockfile","s+":"60"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"7F938DDB218B","b":"7F938DD6C000","o":"4618B","s":"gsignal","s+":"CB"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"7F938DD91859","b":"7F938DD6C000","o":"25859","s":"abort","s+":"12B"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF157CCEE1","b":"55FF14869000","o":"F63EE1","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j","s+":"179"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF154EEB5D","b":"55FF14869000","o":"C85B5D","s":"_ZN5mongo21WiredTigerRecordStore12updateRecordEPNS_16OperationContextERKNS_8RecordIdEPKci.cold.1749","s+":"99"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF161F4E5F","b":"55FF14869000","o":"198BE5F","s":"_ZN5mongo14CollectionImpl14updateDocumentEPNS_16OperationContextENS_8RecordIdERKNS_11SnapshottedINS_7BSONObjEEERKS5_bPNS_7OpDebugEPNS_20CollectionUpdateArgsE","s+":"1DF"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1639A4BF","b":"55FF14869000","o":"1B314BF","s":"_ZN5mongo11UpdateStage18transformAndUpdateERKNS_11SnapshottedINS_7BSONObjEEERNS_8RecordIdE","s+":"A3F"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1639AEC1","b":"55FF14869000","o":"1B31EC1","s":"_ZN5mongo11UpdateStage6doWorkEPm","s+":"361"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1639D06B","b":"55FF14869000","o":"1B3406B","s":"_ZN5mongo11UpsertStage6doWorkEPm","s+":"7B"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1637FBA8","b":"55FF14869000","o":"1B16BA8","s":"_ZN5mongo9PlanStage4workEPm","s+":"68"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF163C41C2","b":"55FF14869000","o":"1B5B1C2","s":"_ZN5mongo16PlanExecutorImpl12_getNextImplEPNS_11SnapshottedINS_8DocumentEEEPNS_8RecordIdE","s+":"222"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF163C4C0B","b":"55FF14869000","o":"1B5BC0B","s":"_ZN5mongo16PlanExecutorImpl7getNextEPNS_8DocumentEPNS_8RecordIdE","s+":"4B"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF163C4D4D","b":"55FF14869000","o":"1B5BD4D","s":"_ZN5mongo16PlanExecutorImpl11executePlanEv","s+":"4D"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF160E8E80","b":"55FF14869000","o":"187FE80","s":"_ZN5mongo14performUpdatesEPNS_16OperationContextERKNS_9write_ops6UpdateE","s+":"E60"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1604E3A6","b":"55FF14869000","o":"17E53A6","s":"_ZNK5mongo12_GLOBAL__N_19CmdUpdate10Invocation7runImplEPNS_16OperationContextERNS_14BSONObjBuilderE","s+":"46"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1604BEED","b":"55FF14869000","o":"17E2EED","s":"_ZN5mongo12_GLOBAL__N_112WriteCommand14InvocationBase3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"15D"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF165AA59F","b":"55FF14869000","o":"1D4159F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5FA81","b":"55FF14869000","o":"14F6A81","s":"_ZN5mongo12_GLOBAL__N_127invokeWithSessionCheckedOutEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationERKNS_30OperationSessionInfoFromClientEPNS_3rpc21ReplyBuilderInterfaceE","s+":"221"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D609F8","b":"55FF14869000","o":"14F79F8","s":"_ZN5mongo12_GLOBAL__N_114runCommandImplEPNS_16OperationContextEPNS_17CommandInvocationERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceENS_11LogicalTimeERKNS_23ServiceEntryPointCommon5HooksEPNS_14BSONObjBuilderERKNS_30OperationSessionInfoFromClientE","s+":"7C8"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D631C9","b":"55FF14869000","o":"14FA1C9","s":"_ZN5mongo12_GLOBAL__N_119execCommandDatabaseEPNS_16OperationContextEPNS_7CommandERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceERKNS_23ServiceEntryPointCommon5HooksE","s+":"11B9"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D6448B","b":"55FF14869000","o":"14FB48B","s":"_ZN5mongo12_GLOBAL__N_116receivedCommandsEPNS_16OperationContextERKNS_7MessageERKNS_23ServiceEntryPointCommon5HooksE","s+":"62B"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D650CD","b":"55FF14869000","o":"14FC0CD","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE","s+":"50D"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5314C","b":"55FF14869000","o":"14EA14C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"3C"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5D65C","b":"55FF14869000","o":"14F465C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE","s+":"FC"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5B505","b":"55FF14869000","o":"14F2505","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"125"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5C4B6","b":"55FF14869000","o":"14F34B6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1702E042","b":"55FF14869000","o":"27C5042","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE","s+":"182"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5A91B","b":"55FF14869000","o":"14F191B","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE","s+":"DB"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5BC7D","b":"55FF14869000","o":"14F2C7D","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE","s+":"6AD"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5BD60","b":"55FF14869000","o":"14F2D60","s":"_ZN5mongo14future_details4callIRZNS_19ServiceStateMachine14_sourceMessageENS2_11ThreadGuardEEUlNS_10StatusWithINS_7MessageEEEE0_S6_EEDaOT_OT0_","s+":"60"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5BFE5","b":"55FF14869000","o":"14F2FE5","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE","s+":"145"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5B57A","b":"55FF14869000","o":"14F257A","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"19A"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF15D5C4B6","b":"55FF14869000","o":"14F34B6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF1702E6A8","b":"55FF14869000","o":"27C56A8","s":"_ZNSt17_Function_handlerIFvvEZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIS0_ENS2_15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameEEUlvE0_E9_M_invokeERKSt9_Any_data","s+":"B8"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF173828A6","b":"55FF14869000","o":"2B198A6","s":"_ZNSt17_Function_handlerIFvvEZN5mongo25launchServiceWorkerThreadESt8functionIS0_EEUlvE1_E9_M_invokeERKSt9_Any_data","s+":"56"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"55FF17382914","b":"55FF14869000","o":"2B19914","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"14"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"7F938DF67609","b":"7F938DF5E000","o":"9609","s":"start_thread","s+":"D9"}}}
      
      {"t":\{"$date":"2020-10-05T22:25:47.177+00:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"conn503635","msg":"  Frame: \{frame}","attr":\{"frame":{"a":"7F938DE8E293","b":"7F938DD6C000","o":"122293","s":"clone","s+":"43"}}}
      

        1. SERVER-51386.png
          SERVER-51386.png
          193 kB
        2. screenshot-1.png
          screenshot-1.png
          45 kB
        3. image-2020-10-08-13-22-04-886.png
          image-2020-10-08-13-22-04-886.png
          45 kB

            Assignee:
            jonathan.streets@mongodb.com Jonathan Streets (Inactive)
            Reporter:
            michael.moore@jhuapl.edu Michael Moore
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: