-
Type: Bug
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.6.7
-
Component/s: Stability
-
None
-
ALL
-
mongodb 3.6.7 will use nearly all the memory in the system when running as primary, and eventually crash with an out of memory exception. This machine has 32GB of RAM and is running Ubuntu 16.04.5 LTS.
2018-10-03T13:38:15.588+0000 I NETWORK [conn1553489] end connection 172.20.0.1:59480 (113 connections now open) 2018-10-03T13:38:15.689+0000 I NETWORK [conn1553490] end connection 10.150.1.1:55372 (113 connections now open) 2018-10-03T13:38:17.184+0000 I NETWORK [conn1553491] received client metadata from 172.20.0.2:35972 conn1553491: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.6" }, os: { type: "Linux", name: "Ubu ntu", architecture: "x86_64", version: "14.04" } } 2018-10-03T13:38:17.566+0000 I NETWORK [conn1553492] received client metadata from 172.20.0.22:54262 conn1553492: { driver: { name: "PyMongo", version: "3.5.1" }, os: { type: "Linux", name: "Ubuntu 16.04 xenial", archite cture: "x86_64", version: "4.17.17-x86_64-linode116" }, platform: "CPython 2.7.15.final.0" } 2018-10-03T13:38:17.832+0000 F - [conn290616] out of memory. 0x55bf275693b1 0x55bf275689e4 0x55bf2755e341 0x55bf25d2de7e 0x55bf2701d730 0x55bf25f53e38 0x55bf25f5549b 0x55bf25f562a1 0x55bf25f627ea 0x55bf25f5e1a7 0x55bf25f615e1 0x55bf26e64022 0x55bf25f5d00f 0x55bf25f5f555 0x55bf25f5 fe4b 0x55bf25f5e22d 0x55bf25f615e1 0x55bf26e64585 0x55bf27422584 0x7f93edae76ba 0x7f93ed81d41d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55BF25330000","o":"22393B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55BF25330000","o":"22389E4","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"55BF25330000","o":"222E341","s":"_ZN5mongo12m ongoReallocEPvm"},{"b":"55BF25330000","o":"9FDE7E","s":"_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi"},{"b":"55BF25330000","o":"1CED730","s":"_ZN5mongo3rpc17OpMsgReplyBuilder22getInPlaceReplyBuil derEm"},{"b":"55BF25330000","o":"C23E38"},{"b":"55BF25330000","o":"C2549B"},{"b":"55BF25330000","o":"C262A1","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55BF2533000 0","o":"C327EA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C2E1A7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55BF25330000","o": "C315E1"},{"b":"55BF25330000","o":"1B34022","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55BF25330000","o":"C2D 00F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55BF25330000","o":"C2F555","s":"_ZN5mo ngo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55BF25330000","o":"C2FE4B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C2E22D","s":"_ZN5mongo19ServiceSta teMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C315E1"},{"b":"55BF25330000","o":"1B34585"},{"b":"55BF25330000","o":"20F2584"},{"b":"7F93EDAE0000","o":"76BA"},{"b":"7F93ED716000","o":"10741D","s": "clone"}],"processInfo":{ "mongodbVersion" : "3.6.7", "gitVersion" : "2628472127e9f1826e02c665c1d93880a204075e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.17.17-x86_64-linode116", "version" : "#1 SMP PREEMPT Mon Aug 20 16:07:40 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55BF25330000", "elfType" : 3, "buildId" : "5D78F445F57AA961C35C97316819BF42C1939FFF" }, { "b" : "7FFFA2BB7000", "elfType" : 3, " buildId" : "20B50035DDDF6D83A1FA4A5C699462D404FFB672" }, { "b" : "7F93EECD5000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7F93EE891 000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "250E875F74377DFC74DE48BF80CCB237BB4EFF1D" }, { "b" : "7F93EE628000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "513282AC7EB386E2C0133FD9E1B6B8A0F38B047D" }, { "b" : "7F93EE424000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7F93EE21C000 ", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7F93EDF13000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB8 5DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7F93EDCFD000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F93EDAE0000", "path" : "/ lib/x86_64-linux-gnu/libpthread.so.0", "elfTy pe" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7F93ED716000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7F93EEEF0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bf275693b1] mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x55bf275689e4] mongod(_ZN5mongo12mongoReallocEPvm+0x21) [0x55bf2755e341] mongod(_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi+0x5E) [0x55bf25d2de7e] mongod(_ZN5mongo3rpc17OpMsgReplyBuilder22getInPlaceReplyBuilderEm+0x80) [0x55bf2701d730] mongod(+0xC23E38) [0x55bf25f53e38] mongod(+0xC2549B) [0x55bf25f5549b] mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2A1) [0x55bf25f562a1] mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55bf25f627ea] mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55bf25f5e1a7] mongod(+0xC315E1) [0x55bf25f615e1] mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55bf26e64022] mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x55bf25f5d00f] mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x55bf25f5f555] mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x55bf25f5fe4b] mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55bf25f5e22d] mongod(+0xC315E1) [0x55bf25f615e1] mongod(+0x1B34585) [0x55bf26e64585] mongod(+0x20F2584) [0x55bf27422584] libpthread.so.0(+0x76BA) [0x7f93edae76ba] libc.so.6(clone+0x6D) [0x7f93ed81d41d] ----- END BACKTRACE -----
The attached screenshot is from a process monitor showing mongodb using almost all the system memory.