Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-2566

Assertion: 13548:BufBuilder grow() > 64MB then clean crash (segfault)

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 1.8.2
    • Affects Version/s: 1.7.5
    • Component/s: Stability
    • Environment:
    • Linux

      Possibly related to SERVER-2485

      Wed Feb 16 10:27:40 [conn113104] remove sd.metrics query:

      { _id: ObjectId('4d5707c2160ba00d6a0011da'), accId: 3567, sId: 5 }

      138ms
      Wed Feb 16 10:27:40 [conn113508] query admin.$cmd ntoreturn:1 command:

      { setShardVersion: "sd.metricsLatest", configdb: "config1:27019", version: Timestamp 0|0, serverID: ObjectId('4d596016fe9bde831c61e013'), shard: "shard2", shardHost: "set2/rs2b:27018" }

      reslen:53 116ms
      Wed Feb 16 10:27:40 [conn113255] query admin.$cmd ntoreturn:1 command:

      { setShardVersion: "sd.metricsLatest", configdb: "config1:27019", version: Timestamp 0|0, serverID: ObjectId('4d596016fe9bde831c61e013'), shard: "shard2", shardHost: "set2/rs2b:27018" }

      reslen:53 120ms
      Wed Feb 16 10:27:40 [conn113357] query admin.$cmd ntoreturn:1 command:

      { setShardVersion: "sd.metricsLatest", configdb: "config1:27019", version: Timestamp 0|0, serverID: ObjectId('4d596016fe9bde831c61e013'), shard: "shard2", shardHost: "set2/rs2b:27018" }

      reslen:53 115ms
      Wed Feb 16 10:27:41 [conn113859] query admin.$cmd ntoreturn:1 command:

      { writebacklisten: ObjectId('4d5ba56021d7d00ef5bbe3b7') }

      reslen:60 300053ms
      Wed Feb 16 10:27:46 [ReplSetHealthPollTask] DBClientCursor::init call() failed
      Wed Feb 16 10:27:46 [ReplSetHealthPollTask] replSet info rs2d:27018 is down (or slow to respond): DBClientBase::findOne: transport error: rs2d:27018 query:

      { replSetHeartbeat: "set2", v: 7, pv: 1, checkEmpty: false, from: "rs2a:27018" }

      Wed Feb 16 10:27:53 [conn113104] query sd.metrics ntoreturn:1000 reslen:146469 nscanned:3571817 { $query: { $nor: [ { accId: 2461, tA:

      { $gt: new Date(1295260061132) }

      }, { accId: 2805, tA:

      { $gt: new Date(1297247261132) }

      }, { accId: 2424, tA:

      { $gt: new Date(1295260061135) }

      }, { tA:

      { $gt: new Date(1297592861135) }

      } ] }, $orderby:

      { _id: 1 }

      } nreturned:1000 12267ms
      Wed Feb 16 10:28:05 [conn113377] query admin.$cmd ntoreturn:1 command:

      { setShardVersion: "sd.servers", configdb: "config1:27019", version: Timestamp 0|0, serverID: ObjectId('4d596016fe9bde831c61e013'), shard: "shard2", shardHost: "set2/rs2b:27018" }

      reslen:53 227ms
      Wed Feb 16 10:28:06 [conn113881] Assertion: 13548:BufBuilder grow() > 64MB
      Wed Feb 16 10:28:06 [conn113220] query admin.$cmd ntoreturn:1 command:

      { setShardVersion: "sd.metricsLatest", configdb: "config1:27019", version: Timestamp 0|0, serverID: ObjectId('4d596016fe9bde831c61e013'), shard: "shard2", shardHost: "set2/rs2b:27018" }

      reslen:53 117ms
      0x560819 0x4e87a7 0x58aa7c 0x6d08e5 0x6cbdb4 0x7d518a 0x7d64e1 0x63fdc5 0x645e85 0x74dc0c 0x75399d 0x89a15e 0x8ad550 0x3f7b60673d 0x3f7aad3d1d
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo11msgassertedEiPKc+0x129) [0x560819]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo10BufBuilder15grow_reallocateEv+0x47) [0x4e87a7]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo14BSONObjBuilder6appendERKNS_10StringDataEPKci+0xbc) [0x58aa7c]
      /home/david/mongodb/latest/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0xad5) [0x6d08e5]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x114) [0x6cbdb4]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x58a) [0x7d518a]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x851) [0x7d64e1]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x63fdc5]
      /home/david/mongodb/latest/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x3005) [0x645e85]
      /home/david/mongodb/latest/bin/mongod [0x74dc0c]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5ad) [0x75399d]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x89a15e]
      /home/david/mongodb/latest/bin/mongod(thread_proxy+0x80) [0x8ad550]
      /lib64/libpthread.so.0 [0x3f7b60673d]
      /lib64/libc.so.6(clone+0x6d) [0x3f7aad3d1d]
      Wed Feb 16 10:28:06 Got signal: 6 (Aborted).

      Wed Feb 16 10:28:06 Backtrace:
      0x899789 0x3f7aa302d0 0x3f7aa30265 0x3f7aa31d10 0x3f7aa6a84b 0x3f7aa7230f 0x3f7aa7276b 0x3f81a9db6a 0x6d13f3 0x6cbdb4 0x7d518a 0x7d64e1 0x63fdc5 0x645e85 0x74dc0c 0x75399d 0x89a15e 0x8ad550 0x3f7b60673d 0x3f7aad3d1d
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x899789]
      /lib64/libc.so.6 [0x3f7aa302d0]
      /lib64/libc.so.6(gsignal+0x35) [0x3f7aa30265]
      /lib64/libc.so.6(abort+0x110) [0x3f7aa31d10]
      /lib64/libc.so.6 [0x3f7aa6a84b]
      /lib64/libc.so.6 [0x3f7aa7230f]
      /lib64/libc.so.6(cfree+0x4b) [0x3f7aa7276b]
      /usr/lib64/libstdc++.so.6(_ZNSsD1Ev+0x3a) [0x3f81a9db6a]
      /home/david/mongodb/latest/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x15e3) [0x6d13f3]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x114) [0x6cbdb4]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x58a) [0x7d518a]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x851) [0x7d64e1]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_10BufBuilderERNS_14BSONObjBuilderEbi+0x35) [0x63fdc5]
      /home/david/mongodb/latest/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x3005) [0x645e85]
      /home/david/mongodb/latest/bin/mongod [0x74dc0c]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x5ad) [0x75399d]
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x21e) [0x89a15e]
      /home/david/mongodb/latest/bin/mongod(thread_proxy+0x80) [0x8ad550]
      /lib64/libpthread.so.0 [0x3f7b60673d]
      /lib64/libc.so.6(clone+0x6d) [0x3f7aad3d1d]

      Wed Feb 16 10:28:06 dbexit:
      Wed Feb 16 10:28:06 [conn113881] shutdown: going to close listening sockets...
      Wed Feb 16 10:28:06 [conn113881] closing listening socket: 5
      Wed Feb 16 10:28:06 [conn113881] closing listening socket: 6
      Wed Feb 16 10:28:06 [conn113881] closing listening socket: 8
      Wed Feb 16 10:28:06 [conn113881] closing listening socket: 9
      Wed Feb 16 10:28:06 [conn113881] removing socket file: /tmp/mongodb-27018.sock
      Wed Feb 16 10:28:06 [conn113246] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113671] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113168] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113394] got request after shutdown()
      Wed Feb 16 10:28:06 [conn112133] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113350] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113508] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113723] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113881] removing socket file: /tmp/mongodb-28018.sock
      Wed Feb 16 10:28:06 [conn113881] shutdown: going to flush diaglog...
      Wed Feb 16 10:28:06 [conn113447] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113881] shutdown: going to close sockets...
      Wed Feb 16 10:28:06 [conn113220] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113126] got request after shutdown()
      Wed Feb 16 10:28:06 [conn96111] got request after shutdown()
      Wed Feb 16 10:28:06 ERROR: Client::shutdown not called: slaveTracking
      Wed Feb 16 10:28:06 [conn113212] got request after shutdown()
      Wed Feb 16 10:28:06 [conn112761] got request after shutdown()
      Wed Feb 16 10:28:06 [conn112695] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113140] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113881] shutdown: waiting for fs preallocator...
      Wed Feb 16 10:28:06 [conn113881] shutdown: closing all files...
      Wed Feb 16 10:28:06 [conn112337] got request after shutdown()
      Wed Feb 16 10:28:06 [initandlisten] now exiting
      Wed Feb 16 10:28:06 dbexit: ; exiting immediately
      Wed Feb 16 10:28:06 [conn111690] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113265] got request after shutdown()
      Wed Feb 16 10:28:06 [conn113157] got request after shutdown()
      Wed Feb 16 10:28:06 [conn112451] got request after shutdown()
      Wed Feb 16 10:28:06 [conn112255] got request after shutdown()
      Wed Feb 16 10:28:06 [conn112088] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113785] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113202] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113518] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113267] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113413] got request after shutdown()
      Wed Feb 16 10:28:07 [conn112565] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113884] assertion 11600 interrupted at shutdown ns:local.oplog.rs query:{ ts:

      { $gte: new Date(5574201246161567754) }

      }
      Wed Feb 16 10:28:07 [conn113885] assertion 11600 interrupted at shutdown ns:local.oplog.rs query:{ ts:

      { $gte: new Date(5574201246161567754) }

      }
      Wed Feb 16 10:28:07 [conn112456] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113319] got request after shutdown()
      Wed Feb 16 10:28:07 [conn112713] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113555] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113364] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113248] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113884] query local.oplog.rs exception 10230ms
      Wed Feb 16 10:28:07 [conn113885] query local.oplog.rs exception 10107ms
      Wed Feb 16 10:28:07 [conn111054] got request after shutdown()
      Wed Feb 16 10:28:07 [conn113598] got request after shutdown()
      Wed Feb 16 10:28:07 [conn112527] got request after shutdown()
      Wed Feb 16 10:28:07 [conn112670] got request after shutdown()
      Wed Feb 16 10:28:07 Got signal: 11 (Segmentation fault).

      Wed Feb 16 10:28:07 Backtrace:
      0x899789 0x3f7aa302d0 0x5fa0a70
      /home/david/mongodb/latest/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x899789]
      /lib64/libc.so.6 [0x3f7aa302d0]
      [0x5fa0a70]

            Assignee:
            kristina Kristina Chodorow (Inactive)
            Reporter:
            boxedice David Mytton
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: