Uploaded image for project: 'MongoDB Database Tools'
  1. MongoDB Database Tools
  2. TOOLS-1703

mongoimport should not insert docs larger than 16MB

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.5
    • Component/s: mongoimport
    • None

      Updated description:

      mongoimport appears to be able to send BSON documents greater than 16MB to the server, causing a server-side assertion. Instead, it should detect too-large documents and give a useful error message that helps users identify the problem document. Whether the document is skipped or the import halts should continue to be controlled by --stopOnError.

      Original description:

      When importing the attached document using mongoimport, the import fails and the server logs a backtrace. I tried numerous options including batchSize and it does not seem to work. None of the individual documents are over the 16MB limit.

      Below are my commands:

      mongoimport
      dev@dev20:~$ mongoimport --version
      mongoimport version: r3.4.5
      git version: 520b8f3092c48d934f0cd78ab5f40fe594f96863
      Go version: go1.7
         os: linux
         arch: amd64
         compiler: gc
      OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
      dev@dev20:~$ mongoimport --db borg --collection packetlog --batchSize=1 --stopOnError -vv --drop --file requests.json
      2017-06-20T21:13:39.189+0200	using 0 decoding workers
      2017-06-20T21:13:39.189+0200	using 1 insert workers
      2017-06-20T21:13:39.189+0200	filesize: 50414859 bytes
      2017-06-20T21:13:39.189+0200	using fields: 
      2017-06-20T21:13:39.190+0200	will listen for SIGTERM, SIGINT, and SIGKILL
      2017-06-20T21:13:39.191+0200	connected to: localhost
      2017-06-20T21:13:39.191+0200	ns: borg.packetlog
      2017-06-20T21:13:39.191+0200	connected to node type: standalone
      2017-06-20T21:13:39.191+0200	standalone server: setting write concern w to 1
      2017-06-20T21:13:39.191+0200	using write concern: w='1', j=false, fsync=false, wtimeout=0
      2017-06-20T21:13:39.191+0200	dropping: borg.packetlog
      2017-06-20T21:13:39.192+0200	standalone server: setting write concern w to 1
      2017-06-20T21:13:39.194+0200	using write concern: w='1', j=false, fsync=false, wtimeout=0
      2017-06-20T21:13:40.196+0200	Failed: lost connection to server
      2017-06-20T21:13:40.196+0200	imported 188 documents
      

      The backtrace is below:

      mongo log file
      2017-06-20T21:13:40.186+0200 I -        [conn95] Assertion: 10334:BSONObj size: 19805143 (0x12E33D7) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "packetlog" src/mongo/bson/bsonobj.cpp 58
      2017-06-20T21:13:40.195+0200 I CONTROL  [conn95] 
       0x5591527debe1 0x559152769a37 0x559151a79905 0x559151a799e4 0x559151b137b8 0x559151dc2d60 0x559151ee73f9 0x559151ee94c6 0x559151ae842d 0x559151ae8d6d 0x559152744c52 0x7fe2c69926da 0x7fe2c66cc17f
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"559151268000","o":"1576BE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"559151268000","o":"1501A37","s":"_ZN5mongo10logContextEPKc"},{"b":"559151268000","o":"811905","s":"_ZN5mongo23msgassertedWithLocationEiPKcS1_j"},{"b":"559151268000","o":"8119E4","s":"_ZN5mongo30msgassertedNoTraceWithLocationEiPKcS1_j"},{"b":"559151268000","o":"8AB7B8","s":"_ZNK5mongo7BSONObj14_assertInvalidEv"},{"b":"559151268000","o":"B5AD60","s":"_ZN5mongo9DbMessage9nextJsObjEv"},{"b":"559151268000","o":"C7F3F9"},{"b":"559151268000","o":"C814C6","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"559151268000","o":"88042D","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"559151268000","o":"880D6D"},{"b":"559151268000","o":"14DCC52"},{"b":"7FE2C698B000","o":"76DA"},{"b":"7FE2C65C4000","o":"10817F","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.4", "gitVersion" : "888390515874a9debd1b6c5d36559ca86b44babd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.10.0-21-generic", "version" : "#23-Ubuntu SMP Fri Apr 28 16:14:22 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "559151268000", "elfType" : 3, "buildId" : "5754B63E6FC9683BABEC9CDF7BF6764E6939C8A2" }, { "b" : "7FFEC59C0000", "elfType" : 3, "buildId" : "6AD662AE66D2690703876837CAF3524BE77CE78C" }, { "b" : "7FE2C7919000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "4957783883176B8025AF45D8E771B92C46A85A84" }, { "b" : "7FE2C74D5000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "446FE5C583238974E0BD57B54D589591854F664C" }, { "b" : "7FE2C72CD000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "375C37AE53F311E5F68DAD4890D953E88FEB226B" }, { "b" : "7FE2C70C9000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "A021F9F70C343C23E2DEB7DAEA4C1A3FC053DBC7" }, { "b" : "7FE2C6DC0000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "4A736E6C4EFF6FD9BFC51A281F9EABBC7C9CA707" }, { "b" : "7FE2C6BA9000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BD7D7479293DD928CA0A4769701EF66A4B0D0D29" }, { "b" : "7FE2C698B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "236FBBFD91B7DC1EDE76916401A1435A1DA42114" }, { "b" : "7FE2C65C4000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5B72576FF331E93852355123AFECDEC70FD247B5" }, { "b" : "7FE2C7B82000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9BD92DC65FF841703504FBB338CA807C1F15A25C" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5591527debe1]
       mongod(_ZN5mongo10logContextEPKc+0x177) [0x559152769a37]
       mongod(_ZN5mongo23msgassertedWithLocationEiPKcS1_j+0x1C9) [0x559151a79905]
       mongod(_ZN5mongo30msgassertedNoTraceWithLocationEiPKcS1_j+0x0) [0x559151a799e4]
       mongod(_ZNK5mongo7BSONObj14_assertInvalidEv+0x208) [0x559151b137b8]
       mongod(_ZN5mongo9DbMessage9nextJsObjEv+0x110) [0x559151dc2d60]
       mongod(+0xC7F3F9) [0x559151ee73f9]
       mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x6D6) [0x559151ee94c6]
       mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1ED) [0x559151ae842d]
       mongod(+0x880D6D) [0x559151ae8d6d]
       mongod(+0x14DCC52) [0x559152744c52]
       libpthread.so.0(+0x76DA) [0x7fe2c69926da]
       libc.so.6(clone+0x5F) [0x7fe2c66cc17f]
      -----  END BACKTRACE  -----
      2017-06-20T21:13:40.195+0200 I -        [conn95] AssertionException handling request, closing client connection: 10334 BSONObj size: 19805143 (0x12E33D7) is invalid. Size must be between 0 and 16793600(16MB) First element: insert: "packetlog"
      

            Assignee:
            david.golden@mongodb.com David Golden
            Reporter:
            imraanparker Imraan Parker
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: