2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] MongoDB starting : pid=40072 port=27017 dbpath=/mnt/nvme0n1/db 64-bit host=linux-txmb 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] db version v3.4.10 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2j-fips 26 Sep 2016 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] allocator: tcmalloc 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] modules: enterprise 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] build environment: 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] distmod: suse12 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] distarch: x86_64 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] target_arch: x86_64 2017-11-30T02:59:28.152-0700 I CONTROL [initandlisten] options: { storage: { dbPath: "/mnt/nvme0n1/db", engine: "wiredTiger", wiredTiger: { collectionConfig: { configString: "block_compressor=zlib" }, engineConfig: { configString: "cache_size=200GB,eviction=(threads_min=4,threads_max=16)" }, indexConfig: { configString: "type=lsm,block_compressor=zlib" } } }, systemLog: { destination: "file", path: "/mnt/nvme0n1/logs/mongo.log" } } 2017-11-30T02:59:28.152-0700 W - [initandlisten] Detected unclean shutdown - /mnt/nvme0n1/db/mongod.lock is not empty. 2017-11-30T02:59:28.178-0700 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2017-11-30T02:59:28.178-0700 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=192298M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),cache_size=200GB,eviction=(threads_min=4,threads_max=16) 2017-11-30T02:59:28.474-0700 I CONTROL [initandlisten] 2017-11-30T02:59:28.474-0700 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-11-30T02:59:28.474-0700 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-11-30T02:59:28.474-0700 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2017-11-30T02:59:28.474-0700 I CONTROL [initandlisten] 2017-11-30T02:59:28.476-0700 I CONTROL [initandlisten] 2017-11-30T02:59:28.476-0700 I CONTROL [initandlisten] ** WARNING: You are running on a NUMA machine. 2017-11-30T02:59:28.476-0700 I CONTROL [initandlisten] ** We suggest launching mongod like this to avoid performance problems: 2017-11-30T02:59:28.476-0700 I CONTROL [initandlisten] ** numactl --interleave=all mongod [other options] 2017-11-30T02:59:28.476-0700 I CONTROL [initandlisten] 2017-11-30T02:59:28.479-0700 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/mnt/nvme0n1/db/diagnostic.data' 2017-11-30T02:59:28.479-0700 I NETWORK [thread1] waiting for connections on port 27017 2017-11-30T02:59:29.014-0700 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK 2017-11-30T02:59:37.841-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38698 #1 (1 connection now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38702 #2 (2 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38703 #3 (3 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38700 #4 (4 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38706 #5 (5 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38708 #6 (6 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38709 #7 (7 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38712 #8 (8 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38716 #9 (9 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38718 #10 (10 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38714 #11 (11 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38720 #12 (12 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38722 #13 (13 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38724 #14 (14 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38728 #15 (15 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38726 #16 (16 connections now open) 2017-11-30T02:59:37.862-0700 I NETWORK [thread1] connection accepted from 127.0.0.1:38727 #17 (17 connections now open) 2017-11-30T02:59:38.238-0700 I WRITE [conn6] update ycsb.usertable query: { _id: "user7420173051200326067" } planSummary: IDHACK update: { $set: { field4: BinData(0, 3E337E235D6D3348652B3C323C24602E5E3727307A3F387227523D213F70305D6B32466F363F2A284133364A6137336C325A293C3F643A216C273B623E3536214B7935557937...) } } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 writeConflicts:1 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 158ms 2017-11-30T02:59:38.238-0700 I COMMAND [conn6] command ycsb.$cmd command: update { update: "usertable", ordered: true, updates: [ { q: { _id: "user7420173051200326067" }, u: { $set: { field4: BinData(0, 3E337E235D6D3348652B3C323C24602E5E3727307A3F387227523D213F70305D6B32466F363F2A284133364A6137336C325A293C3F643A216C273B623E3536214B7935557937...) } } } ] } numYields:0 reslen:59 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } protocol:op_query 158ms 2017-11-30T02:59:40.666-0700 F - [conn14] Invalid access at address: 0x88 2017-11-30T02:59:40.666-0700 I - [conn6] Invariant failure session->cursorsOut() == 0 src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 312 2017-11-30T02:59:40.667-0700 I - [conn6] ***aborting after invariant() failure 2017-11-30T02:59:40.689-0700 F - [conn14] Got signal: 11 (Segmentation fault). 0x558958e8aa11 0x558958e89c29 0x558958e8a296 0x7f841cf46b00 0x5589597d678f 0x5589597a2ed9 0x5589597f3a78 0x558958b8e8be 0x55895844740b 0x5589583fe2e8 0x55895840fa63 0x5589587c5fba 0x5589587c68db 0x558958772008 0x5589584f15df 0x5589580edcfd 0x5589580ee62d 0x558958df2871 0x7f841cf3e734 0x7f841cc7cd3d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"558957810000","o":"167AA11","s":"_ZN5mongo15printStackTraceERSo"},{"b":"558957810000","o":"1679C29"},{"b":"558957810000","o":"167A296"},{"b":"7F841CF36000","o":"10B00"},{"b":"558957810000","o":"1FC678F","s":"__wt_row_search"},{"b":"558957810000","o":"1F92ED9","s":"__wt_btcur_search"},{"b":"558957810000","o":"1FE3A78"},{"b":"558957810000","o":"137E8BE","s":"_ZN5mongo21WiredTigerRecordStore6Cursor9seekExactERKNS_8RecordIdE"},{"b":"558957810000","o":"C3740B","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEE"},{"b":"558957810000","o":"BEE2E8","s":"_ZN5mongo11IDHackStage6doWorkEPm"},{"b":"558957810000","o":"BFFA63","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"558957810000","o":"FB5FBA","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"558957810000","o":"FB68DB","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"558957810000","o":"F62008","s":"_ZN5mongo8runQueryB5cxx11EPNS_16OperationContextERNS_12QueryMessageERKNS_15NamespaceStringERNS_7MessageE"},{"b":"558957810000","o":"CE15DF","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"558957810000","o":"8DDCFD","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"558957810000","o":"8DE62D"},{"b":"558957810000","o":"15E2871"},{"b":"7F841CF36000","o":"8734"},{"b":"7F841CB93000","o":"E9D3D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10", "gitVersion" : "078f28920cb24de0dd479b5ea6c66c644f6326e9", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "4.4.21-69-default", "version" : "#1 SMP Tue Oct 25 10:58:20 UTC 2016 (9464f67)", "machine" : "x86_64" }, "somap" : [ { "b" : "558957810000", "elfType" : 3, "buildId" : "A41E5221524CAEB6E082B08D5A3F6EDBD6978C9D" }, { "b" : "7FFC0AD58000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F0CA78853A9A08401393D8E91C24A829A7B99C92" }, { "b" : "7F841FEB9000", "path" : "/usr/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "0A33828FF613CF92C49320B236899E07910DCCF8" }, { "b" : "7F841FC70000", "path" : "/usr/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0810999A29328DF60A1AC50333263715DD703CF2" }, { "b" : "7F841FA23000", "path" : "/usr/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "B71F9A176FD0367CDC552A47B97E88137A791943" }, { "b" : "7F841F814000", "path" : "/usr/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "AF58222069250DD346F76B01C4A63A8A85A38FDF" }, { "b" : "7F841F35A000", "path" : "/usr/lib64/libnetsnmpmibs.so.30", "elfType" : 3, "buildId" : "4B62B1CC35D9DABD9F1B0C481AD5ECFA9D0D1A5B" }, { "b" : "7F841F14B000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "BC1DE263B13CC1F5A0C8EF21EB24DAAFE7856118" }, { "b" : "7F841EF47000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5EB2F40AD3B0125943ABA8F08DD08609351A2967" }, { "b" : "7F841EB75000", "path" : "/usr/lib64/librpm.so.3", "elfType" : 3, "buildId" : "1945438BF00EE02FEBBAACF23BF801ACFCB65878" }, { "b" : "7F841E924000", "path" : "/usr/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "61CA10DC6458A52C4999250299DD2E17F2CE3847" }, { "b" : "7F841E6B5000", "path" : "/usr/lib64/libnetsnmpagent.so.30", "elfType" : 3, "buildId" : "01C9D6230C537F986B0AC22506F94EF9F71CA34D" }, { "b" : "7F841E4AA000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "CF41E1DEA345BC9D4725626F995851BFD26EC846" }, { "b" : "7F841E1A6000", "path" : "/usr/lib64/libnetsnmp.so.30", "elfType" : 3, "buildId" : "BBC6E0AA1C25C282609253FE4D4C0E0DE16C8B2B" }, { "b" : "7F841DD4C000", "path" : "/lib64/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "55FB54F17E6F64A257C4CD85B707894071CBCBCF" }, { "b" : "7F841DADB000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "4DFB9FADA869C2641F09013268895FA2D751845B" }, { "b" : "7F841D7DE000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "4E96CF37D52B9C2F3648E691878B682DA5ABFA42" }, { "b" : "7F841D572000", "path" : "/lib64/libssl.so.1.0.0", "elfType" : 3, "buildId" : "38B103F675F703D0E414B134B2FA03B278B95112" }, { "b" : "7F841D36A000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "B115BB26E97505A5BD3B56D70D20459AA1206AC9" }, { "b" : "7F841D153000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "871B35872AD91A01084297E715A02069B938AE6A" }, { "b" : "7F841CF36000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "4F3D05F200DB29C6835A48E466E0378A8541FD36" }, { "b" : "7F841CB93000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "93C4DEAC1088EB84FBD01CF2A2C54399F516E9A7" }, { "b" : "7F84200D6000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "BA6B43023ADFD54F0DA1F2732F67D6FA62D79711" }, { "b" : "7F841C8C2000", "path" : "/usr/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "77523DBA55EBDA546A8F67B2939F07384E8EC4B7" }, { "b" : "7F841C692000", "path" : "/usr/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "9824BDAB3C561F2DC4612DA1242FB0B5123EE996" }, { "b" : "7F841C48E000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "72B2D85C2EF0EC8CD1410578E86FAE6A4A61BF7B" }, { "b" : "7F841C281000", "path" : "/usr/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "F593B3387D5AC3DDA5952F996FA7016F69E36BBB" }, { "b" : "7F841C06A000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F6260CED80C400AF648F1016712E54D8EA85673D" }, { "b" : "7F841BE5E000", "path" : "/usr/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "068111A4BCB931F524D0CE0632376379F8201E9E" }, { "b" : "7F841BC38000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "0627D20B9E481E6A9C6FCACAF8E3F5C2318529CF" }, { "b" : "7F841BA33000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "9BF069FD1FE829D295F2854BD3C1E6348FE67537" }, { "b" : "7F841B82A000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "08351BC8F2DE7C4C65EF458BA05A7F6082F075CE" }, { "b" : "7F841B5FA000", "path" : "/usr/lib64/liblua.so.5.1", "elfType" : 3, "buildId" : "6CDB83FBAACACA4E45A9CEA9ABE37F05C9AC83D2" }, { "b" : "7F841B3EB000", "path" : "/usr/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "3309CDA7827653FE2BEAD6ADAEF2B43EE3ABB1C0" }, { "b" : "7F841B1D5000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "C7FEFB0BF5BDD1213743AA7D56972E11D09437D1" }, { "b" : "7F841AFBF000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "70013C32B32E4C30D6A41A0DE27E73D2136585CD" }, { "b" : "7F841AD99000", "path" : "/usr/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "85D85F61C61E23BFEA879FA52DEF274FBDD7DB71" }, { "b" : "7F841AA06000", "path" : "/usr/lib/perl5/5.18.2/x86_64-linux-thread-multi/CORE/libperl.so", "elfType" : 3, "buildId" : "F72FE7C5BAE384A9A90D3295394C54A59F47070F" }, { "b" : "7F841A7D2000", "path" : "/usr/lib64/libidn.so.11", "elfType" : 3, "buildId" : "87E0F137C2858C0C816CA76262C98C58D18333A5" }, { "b" : "7F841A5A6000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "CF09CB24D09BDBA13DAB9B62EB671E2D7DD8C9F8" }, { "b" : "7F841A3A2000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8982882FA17C60FED02166F229335A9D0EAF77C9" }, { "b" : "7F841A13C000", "path" : "/usr/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "19A0D7FE3116C0AADDA368E8F39807A7E1193058" }, { "b" : "7F8419F37000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "D4AD8C49D3B5573D417BB9CD7B87F3E6E9CD6ABF" }, { "b" : "7F8419CFC000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "1421B20F6C867EF617D9DC1D9D9A0770E28B97E2" }, { "b" : "7F8419AF7000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "50899E5DFD594328E133FBD2283E3D389D780EF7" }, { "b" : "7F84198F0000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "883D66554C1AA45179F4CBCA741F19702C596505" }, { "b" : "7F8419573000", "path" : "/usr/lib64/libdb-4.8.so", "elfType" : 3, "buildId" : "439F53DF1F2729F81F156B63C1014CBEE39E3D67" }, { "b" : "7F8419365000", "path" : "/usr/lib64/sasl2/libdigestmd5.so", "elfType" : 3, "buildId" : "BBDCB90131D0EB2D306B1DD8370DB2C0E43BB21C" }, { "b" : "7F841915C000", "path" : "/usr/lib64/sasl2/libgssapiv2.so", "elfType" : 3, "buildId" : "5E7AD405373623CAE0DC2E5120C9395989F49C49" }, { "b" : "7F8418F57000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "27F363A82A09B36EA95601E11D5BC5B765283ADA" }, { "b" : "7F8418D52000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "EDD774CAC0102492D9566355D24070FFBD995EA0" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x558958e8aa11] mongod(+0x1679C29) [0x558958e89c29] mongod(+0x167A296) [0x558958e8a296] libpthread.so.0(+0x10B00) [0x7f841cf46b00] mongod(__wt_row_search+0x3F) [0x5589597d678f] mongod(__wt_btcur_search+0x7C9) [0x5589597a2ed9] mongod(+0x1FE3A78) [0x5589597f3a78] mongod(_ZN5mongo21WiredTigerRecordStore6Cursor9seekExactERKNS_8RecordIdE+0x4E) [0x558958b8e8be] mongod(_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEE+0xAB) [0x55895844740b] mongod(_ZN5mongo11IDHackStage6doWorkEPm+0x1E8) [0x5589583fe2e8] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55895840fa63] mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x5589587c5fba] mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x5589587c68db] mongod(_ZN5mongo8runQueryB5cxx11EPNS_16OperationContextERNS_12QueryMessageERKNS_15NamespaceStringERNS_7MessageE+0x8D8) [0x558958772008] mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xB5F) [0x5589584f15df] mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1FD) [0x5589580edcfd] mongod(+0x8DE62D) [0x5589580ee62d] mongod(+0x15E2871) [0x558958df2871] libpthread.so.0(+0x8734) [0x7f841cf3e734] libc.so.6(clone+0x6D) [0x7f841cc7cd3d] ----- END BACKTRACE -----