-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 2.8.0-rc0
-
Component/s: Storage
-
ALL
Got a crash due to WT error. Master as of this morning plus patch for SERVER-16078.
Startup info:
2014-11-13T08:36:10.404-0500 I CONTROL [conn55] pid=1813 port=27077 64-bit host=erh-twinb 2014-11-13T08:36:10.404-0500 I CONTROL [conn55] db version v2.8.0-rc1-pre- 2014-11-13T08:36:10.404-0500 I CONTROL [conn55] git version: 7bdca162807d6436d469a352a129226252cb451d 2014-11-13T08:36:10.404-0500 I CONTROL [conn55] build info: Linux erh-twinb 3.5.0-54-generic #81~precise1-Ubuntu SMP Tue Jul 15 04:02:22 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49 2014-11-13T08:36:10.404-0500 I CONTROL [conn55] allocator: tcmalloc 2014-11-13T08:36:10.404-0500 I CONTROL [conn55] options: { net: { port: 27077 }, processManagement: { fork: true }, storage: { dbPath: "/mnt/data0/asya/wtStandalone/", engine: "wiredtiger", wiredtiger: { engineConfig: "cache_size=12G,checkpoint=(wait=10,log_size=500MB)" } }, systemLog: { destination: "file", path: "/mnt/data1/asya/wt7.log" } }
After running YCSB mixed load on LSM collection, and running db.serverStatus() at the same time/monitoring, got:
2014-11-13T08:38:14.506-0500 I QUERY [conn42] command ycsb.$cmd command: update { update: "reallsm", ordered: true, updates: [ { q: { _id: "user5342744738353846382" }, u: { $set: { field11: BinData(0, 3D343A29202F3526313A26262A26262A2037363F272B3428353B2823303526243A20293F28373D352B2C3E392B25322B3B21223A2C2C2435262D3D3A3D3E233D372A2230212F...) } } } ] } keyUpdates:0 numYields:0 reslen:55 132ms 2014-11-13T08:38:14.606-0500 E STORAGE WiredTiger (12) [1415885894:606741][1813:0x7f10ef96a700], lsm-worker: Failed doing LSM switch: Cannot allocate memory 2014-11-13T08:38:14.606-0500 E STORAGE WiredTiger (-31803) [1415885894:606816][1813:0x7f10ef96a700], lsm-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic 2014-11-13T08:38:14.606-0500 I - Fatal Assertion 28558 2014-11-13T08:38:14.611-0500 I CONTROL 0x10274b6 0xfcaa5d 0xfb143f 0xe270db 0x147540f 0x1475695 0x1475a74 0x144e835 0x1450569 0x1451329 0x7f10f4c20e9a 0x7f10f3d3031d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"C274B6"},{"b":"400000","o":"BCAA5D"},{"b":"400000","o":"BB143F"},{"b":"400000","o":"A270DB"},{"b":"400000","o":"107540F"},{"b":"400000","o":"1075695"},{"b":"400000","o":"1075A74"},{"b":"400000","o":"104E835"},{"b":"400000","o":"1050569"},{"b":"400000","o":"1051329"},{"b":"7F10F4C19000","o":"7E9A"},{"b":"7F10F3C3C000","o":"F431D"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1-pre-", "gitVersion" : "7bdca162807d6436d469a352a129226252cb451d", "uname" : { "sysname" : "Linux", "release" : "3.5.0-54-generic", "version" : "#81~precise1-Ubuntu SMP Tue Jul 15 04:02:22 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E90FB46BF71A8E0E1A6D807EF9AC021ED6AF1DB6" }, { "b" : "7FFF084FF000", "elfType" : 3, "buildId" : "CAB4773A6DC38342CE3FAEC3A776EF5FC8A277D4" }, { "b" : "7F10F4C19000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7F10F4A11000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7F10F480D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7F10F450D000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7F10F4211000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7F10F3FFB000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7F10F3C3C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "E7863FF2EA3F4B5C6FF90689ADE4D6B5F3A78020" }, { "b" : "7F10F4E36000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x26) [0x10274b6] mongod(_ZN5mongo10logContextEPKc+0xDD) [0xfcaa5d] mongod(_ZN5mongo13fassertFailedEi+0x3F) [0xfb143f] mongod(+0xA270DB) [0xe270db] mongod(+0x107540F) [0x147540f] mongod(__wt_err+0x95) [0x1475695] mongod(__wt_panic+0x24) [0x1475a74] mongod(__wt_lsm_tree_switch+0x165) [0x144e835] mongod(__wt_lsm_work_switch+0x69) [0x1450569] mongod(+0x1051329) [0x1451329] libpthread.so.0(+0x7E9A) [0x7f10f4c20e9a] libc.so.6(clone+0x6D) [0x7f10f3d3031d] ----- END BACKTRACE ----- 2014-11-13T08:38:14.611-0500 I - ***aborting after fassert() failure 2014-11-13T08:38:14.615-0500 F - Got signal: 6 (Aborted). 0x10274b6 0x1027032 0x1027406 0x7f10f3c72150 0x7f10f3c720d5 0x7f10f3c7583b 0xfb1476 0xe270db 0x147540f 0x1475695 0x1475a74 0x144e835 0x1450569 0x1451329 0x7f10f4c20e9a 0x7f10f3d3031d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"C274B6"},{"b":"400000","o":"C27032"},{"b":"400000","o":"C27406"},{"b":"7F10F3C3C000","o":"36150"},{"b":"7F10F3C3C000","o":"360D5"},{"b":"7F10F3C3C000","o":"3983B"},{"b":"400000","o":"BB1476"},{"b":"400000","o":"A270DB"},{"b":"400000","o":"107540F"},{"b":"400000","o":"1075695"},{"b":"400000","o":"1075A74"},{"b":"400000","o":"104E835"},{"b":"400000","o":"1050569"},{"b":"400000","o":"1051329"},{"b":"7F10F4C19000","o":"7E9A"},{"b":"7F10F3C3C000","o":"F431D"}],"processInfo":{ "mongodbVersion" : "2.8.0-rc1-pre-", "gitVersion" : "7bdca162807d6436d469a352a129226252cb451d", "uname" : { "sysname" : "Linux", "release" : "3.5.0-54-generic", "version" : "#81~precise1-Ubuntu SMP Tue Jul 15 04:02:22 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E90FB46BF71A8E0E1A6D807EF9AC021ED6AF1DB6" }, { "b" : "7FFF084FF000", "elfType" : 3, "buildId" : "CAB4773A6DC38342CE3FAEC3A776EF5FC8A277D4" }, { "b" : "7F10F4C19000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "C340AF9DEE97C17C730F7D03693286C5194A46B8" }, { "b" : "7F10F4A11000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "352C5B373A50E6C4AB881A5DB6F5766FDF81EEE0" }, { "b" : "7F10F480D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D181AF551DBBC43E9D55913D532635FDE18E7C4E" }, { "b" : "7F10F450D000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7F10F4211000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "817AA99B3DD02501F8BC04A3E9A9358A08F20D7D" }, { "b" : "7F10F3FFB000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7F10F3C3C000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "E7863FF2EA3F4B5C6FF90689ADE4D6B5F3A78020" }, { "b" : "7F10F4E36000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E25AD1A11CCF57E734116B8EC9C69F643DCA9F18" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x26) [0x10274b6] mongod(+0xC27032) [0x1027032] mongod(+0xC27406) [0x1027406] libc.so.6(+0x36150) [0x7f10f3c72150] libc.so.6(gsignal+0x35) [0x7f10f3c720d5] libc.so.6(abort+0x17B) [0x7f10f3c7583b] mongod(_ZN5mongo13fassertFailedEi+0x76) [0xfb1476] mongod(+0xA270DB) [0xe270db] mongod(+0x107540F) [0x147540f] mongod(__wt_err+0x95) [0x1475695] mongod(__wt_panic+0x24) [0x1475a74] mongod(__wt_lsm_tree_switch+0x165) [0x144e835] mongod(__wt_lsm_work_switch+0x69) [0x1450569] mongod(+0x1051329) [0x1451329] libpthread.so.0(+0x7E9A) [0x7f10f4c20e9a] libc.so.6(clone+0x6D) [0x7f10f3d3031d] ----- END BACKTRACE -----
Using local SSD with journal sym-linked to another directory. Loads of RAM available at the time.