Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-15539

Invariant failure keyOffset >= 0 during 16 thread write command remove operation

XMLWordPrintableJSON

    • Fully Compatible
    • Windows
    • Hide

      Use mongod 78a778 from a green MCI build.

      > db.serverBuildInfo()
      {
      "version" : "2.7.7-pre-",
      "gitVersion" : "78a778b23bed9018e56b8573b052ee2117beac46",
      "OpenSSLVersion" : "",
      "sysInfo" : "Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49",
      "loaderFlags" : "-fPIC -pthread -Wl,-z,now -rdynamic",
      "compilerFlags" : "-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Werror -O3 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -fno-builtin-memcmp -std=c99",
      "allocator" : "tcmalloc",
      "versionArray" : [
      2,
      7,
      7,
      -100
      ],
      "javascriptEngine" : "V8",
      "bits" : 64,
      "debug" : false,
      "maxBsonObjectSize" : 16777216,
      "ok" : 1
      }

      Start mongod:
      ./mongod --dbpath /mnt/db2 --smallfiles --fork --logpath /tmp/server.log

      Use a mongo shell with version 2.7.7 (hand compiled).

      Obtain util/utils.js from the mongo-perf project (mongodb/mongo-perf master branch).

      Enter this into mongo a few times until the issue reproduces:

      var tests = []
load('/insert-path-to-mongo-perf-here/util/utils.js')
tests.push( { name: "Remove.v2.IntNonIdNoIndex",
              pre: function( collection ) {
                  collection.drop();
                  for ( var i = 0; i < 1000; i++ ) {
                      collection.insert( { x : i } );
                  }
              },
              ops: [
                  { op: "let", target: "x", value: {"#RAND_INT_PLUS_THREAD": [0,1000]}},
                  { op:  "insert",
                    doc: { x : { "#VARIABLE" : "x" } } },
                  { op:  "remove",
                    query: { x : { "#VARIABLE" : "x" } } }
              ] } );

runTests([16], 4, 5, 100, 'reproIsIntermittent')

      Use the following block to reproduce. The "alternative" is less unwieldy.

      ALTERNATIVELY:

      from the mongo-perf top level directory:
      edit util/benchrun_daemon.sh
      set RUNUSER and MPERFBASE if needed
      set RHOST and RPORT
      uncomment FETCHMCI='TRUE'

      The following will run single db first, then multi db second:

      $ util/benchrun_daemon.sh 78a778b23bed9018e56b8573b052ee2117beac46

      The failure occurred in the second run of the test suite, with multi database (4 databases active).

      Show
      Use mongod 78a778 from a green MCI build. > db.serverBuildInfo() { "version" : "2.7.7-pre-", "gitVersion" : "78a778b23bed9018e56b8573b052ee2117beac46", "OpenSSLVersion" : "", "sysInfo" : "Linux build8.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49", "loaderFlags" : "-fPIC -pthread -Wl,-z,now -rdynamic", "compilerFlags" : "-Wnon-virtual-dtor -Woverloaded-virtual -std=c++11 -fPIC -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -pipe -Werror -O3 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -fno-builtin-memcmp -std=c99", "allocator" : "tcmalloc", "versionArray" : [ 2, 7, 7, -100 ], "javascriptEngine" : "V8", "bits" : 64, "debug" : false, "maxBsonObjectSize" : 16777216, "ok" : 1 } Start mongod: ./mongod --dbpath /mnt/db2 --smallfiles --fork --logpath /tmp/server.log Use a mongo shell with version 2.7.7 (hand compiled). Obtain util/utils.js from the mongo-perf project (mongodb/mongo-perf master branch). Enter this into mongo a few times until the issue reproduces: var tests = [] load( '/insert-path-to-mongo-perf-here/util/utils.js' ) tests.push( { name: "Remove.v2.IntNonIdNoIndex" , pre: function( collection ) { collection.drop(); for ( var i = 0; i < 1000; i++ ) { collection.insert( { x : i } ); } }, ops: [ { op: "let" , target: "x" , value: { "#RAND_INT_PLUS_THREAD" : [0,1000]}}, { op: "insert" , doc: { x : { "#VARIABLE" : "x" } } }, { op: "remove" , query: { x : { "#VARIABLE" : "x" } } } ] } ); runTests([16], 4, 5, 100, 'reproIsIntermittent' ) Use the following block to reproduce. The "alternative" is less unwieldy. ALTERNATIVELY: from the mongo-perf top level directory: edit util/benchrun_daemon.sh set RUNUSER and MPERFBASE if needed set RHOST and RPORT uncomment FETCHMCI='TRUE' The following will run single db first, then multi db second: $ util/benchrun_daemon.sh 78a778b23bed9018e56b8573b052ee2117beac46 The failure occurred in the second run of the test suite, with multi database (4 databases active).

      Hit an Invariant failure in mmap_v1/btree/btree_logic.cpp at line 2059. The operation that encountered the error was a remove, sent via the write command. 16 threads were executing the mongo-perf Remove.v2.IntNonIdIndex test.

      The issue may be intermittent and difficult to reproduce. It may depend on preconditions such as the loading of other data or execution of other tests in the mongo-perf suite.

      In other words, the issue may not reproduce unless the mongo-perf suite is run. I was not able to reproduce with a single invocation of benchRun() or runTests in util/utils.js.

      mongod is commit 78a778b23bed9018e56b8573b052ee2117beac46
      and mongo is r2.7.7

      This occurred while running mongo-perf against the Remove.v2.IntNonIdIndex test. The test ran a few trials before the server died:
      16 28727.915427512453
      16 28967.207063672307
      16 28903.757846425404
      16 28614.076587610252
      16 28621.215665117536

      mongod log file shows the following:

      2014-10-06T19:26:20.571+0000 I -        [conn25082] Invariant failure keyOffset >= 0 src/mongo/db/storage/mmap_v1/btree/btree_logic.cpp 2059
2014-10-06T19:26:20.597+0000 I -        [conn25082] 
 0xeed369 0xe99031 0xe7faa3 0xcd740e 0xccb6ec 0xa615c3 0x9d953f 0x9da268 0x9d0cf7 0x9cdd84 0xbaf77d 0xad2395 0x98850b 0x98a7b5 0x98adeb 0x98d449 0x9a7d84 0x9a8c11 0x9a9680 0xb9d5ea 0xa7fd03 0x7ed730 0xeac3d1 0x7f16a6625e9a 0x7f16a573473d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"AED369"},{"b":"400000","o":"A99031"},{"b":"400000","o":"A7FAA3"},{"b":"400000","o":"8D740E"},{"b":"400000","o":"8CB6EC"},{"b":"400000","o":"6615C3"},{"b":"400000","o":"5D953F"},{"b":"400000","o":"5DA268"},{"b":"400000","o":"5D0CF7"},{"b":"400000","o":"5CDD84"},{"b":"400000","o":"7AF77D"},{"b":"400000","o":"6D2395"},{"b":"400000","o":"58850B"},{"b":"400000","o":"58A7B5"},{"b":"400000","o":"58ADEB"},{"b":"400000","o":"58D449"},{"b":"400000","o":"5A7D84"},{"b":"400000","o":"5A8C11"},{"b":"400000","o":"5A9680"},{"b":"400000","o":"79D5EA"},{"b":"400000","o":"67FD03"},{"b":"400000","o":"3ED730"},{"b":"400000","o":"AAC3D1"},{"b":"7F16A661E000","o":"7E9A"},{"b":"7F16A5640000","o":"F473D"}],"processInfo":{ "mongodbVersion" : "2.7.7-pre-", "gitVersion" : "78a778b23bed9018e56b8573b052ee2117beac46", "uname" : { "sysname" : "Linux", "release" : "3.2.0-67-virtual", "version" : "#101-Ubuntu SMP Tue Jul 15 17:58:37 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000" }, { "b" : "7FFF69D51000", "elfType" : 3 }, { "b" : "7F16A661E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7F16A6416000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7F16A6212000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7F16A5F12000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3 }, { "b" : "7F16A5C16000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7F16A5A00000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3 }, { "b" : "7F16A5640000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7F16A683B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x29) [0xeed369]
 mongod(_ZN5mongo10logContextEPKc+0xE1) [0xe99031]
 mongod(_ZN5mongo15invariantFailedEPKcS1_j+0xD3) [0xe7faa3]
 mongod(_ZNK5mongo10BtreeLogicINS_13BtreeLayoutV1EE6getKeyEPNS_16OperationContextERKNS_7DiskLocEi+0xBE) [0xcd740e]
 mongod(_ZNK5mongo18BtreeInterfaceImplINS_13BtreeLayoutV1EE6Cursor6getKeyEv+0x1C) [0xccb6ec]
 mongod(_ZNK5mongo16BtreeIndexCursor6getKeyEv+0x13) [0xa615c3]
 mongod(_ZN5mongo9IndexScan8checkEndEv+0x7F) [0x9d953f]
 mongod(_ZN5mongo9IndexScan4workEPm+0xA8) [0x9da268]
 mongod(_ZN5mongo10FetchStage4workEPm+0x57) [0x9d0cf7]
 mongod(_ZN5mongo11DeleteStage4workEPm+0x64) [0x9cdd84]
 mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x2D) [0xbaf77d]
 mongod(_ZN5mongo14DeleteExecutor7executeEPNS_8DatabaseE+0x75) [0xad2395]
 mongod(_ZN5mongo18WriteBatchExecutor10execRemoveERKNS_12BatchItemRefEPPNS_16WriteErrorDetailE+0x38B) [0x98850b]
 mongod(_ZN5mongo18WriteBatchExecutor11bulkExecuteERKNS_21BatchedCommandRequestEPSt6vectorIPNS_19BatchedUpsertDetailESaIS6_EEPS4_IPNS_16WriteErrorDetailESaISB_EE+0xF5) [0x98a7b5]
 mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0x38B) [0x98adeb]
 mongod(_ZN5mongo8WriteCmd3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x169) [0x98d449]
 mongod(_ZN5mongo12_execCommandEPNS_16OperationContextEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x34) [0x9a7d84]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_iPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xC51) [0x9a8c11]
 mongod(_ZN5mongo12_runCommandsEPNS_16OperationContextEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x1F0) [0x9a9680]
 mongod(_ZN5mongo11newRunQueryEPNS_16OperationContextERNS_7MessageERNS_12QueryMessageERNS_5CurOpES3_b+0x136A) [0xb9d5ea]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortEb+0xD93) [0xa7fd03]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0xE0) [0x7ed730]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x421) [0xeac3d1]
 libpthread.so.0(+0x7E9A) [0x7f16a6625e9a]
 libc.so.6(clone+0x6D) [0x7f16a573473d]
-----  END BACKTRACE  -----
2014-10-06T19:26:20.597+0000 I -        [conn25082] 

***aborting after invariant() failure


2014-10-06T19:26:20.608+0000 F -        [conn25082] Got signal: 6 (Aborted).

        1. mongod-invariant-btree-20141122a.log.gz
          1.39 MB
          Quentin Conner
        2. mongod-invariant-btree-20141122b.log.gz
          1.36 MB
          Quentin Conner
        3. mongoperf.log
          8.43 MB
          Quentin Conner
        4. utils.js
          13 kB
          Quentin Conner

            Assignee:
            geert.bosch@mongodb.com Geert Bosch
            Reporter:
            quentin.conner Quentin Conner
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: