Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-2900

mongos crash with "Received signal 6"

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.8.1
    • Component/s: None
    • None
    • Environment:
    • Linux

      Tue Apr 5 09:55:33 [mongosMain] dbexit: received signal 15 rc:0 received signal 15
      Tue Apr 5 09:55:34 /home/david/mongodb/latest/bin/mongos db version v1.8.1-rc1, pdfile version 4.5 starting (--help for usage)
      Tue Apr 5 09:55:34 git version: c340b4882b752b9e9fdae4db2738ee502cd254e3
      Tue Apr 5 09:55:34 build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
      Tue Apr 5 09:55:34 [websvr] web admin interface listening on port 28017
      Tue Apr 5 09:55:34 [websvr] couldn't unlink socket file /tmp/mongodb-28017.sockerrno:1 Operation not permitted skipping
      Tue Apr 5 09:55:34 [mongosMain] waiting for connections on port 27017
      Tue Apr 5 09:55:34 [mongosMain] couldn't unlink socket file /tmp/mongodb-27017.sockerrno:1 Operation not permitted skipping
      Tue Apr 5 09:55:34 [Balancer] about to contact config servers and shards
      Tue Apr 5 09:55:34 [Balancer] updated set (set1) to: set1/rs1a:27018,rs1b:27018
      Tue Apr 5 09:55:34 [ReplicaSetMonitorWatcher] starting
      Tue Apr 5 09:55:34 [Balancer] updated set (set2) to: set2/rs2a:27018,rs2b:27018
      Tue Apr 5 09:55:34 [Balancer] updated set (set3) to: set3/rs3a:27018,rs3b:27018
      Tue Apr 5 09:55:34 [Balancer] config servers and shards contacted successfully
      Tue Apr 5 09:55:34 [Balancer] balancer id: ad1:27017 started at Apr 5 09:55:34
      Tue Apr 5 09:55:34 [LockPinger] creating dist lock ping thread for: config1:27019
      Tue Apr 5 09:55:34 [conn2] creating WriteBackListener for: rs1a:27018
      Tue Apr 5 09:55:34 [conn2] creating WriteBackListener for: rs1b:27018
      Tue Apr 5 09:55:34 [conn2] creating WriteBackListener for: rs2a:27018
      Tue Apr 5 09:55:34 [conn2] creating WriteBackListener for: rs2b:27018
      Tue Apr 5 09:55:34 [conn2] creating WriteBackListener for: rs3a:27018
      Tue Apr 5 09:55:34 [conn2] creating WriteBackListener for: rs3b:27018
      Tue Apr 5 10:00:04 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:05:04 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:10:04 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:15:04 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:20:04 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:25:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:30:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:35:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:40:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:45:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:46:34 [conn69] warning: splitChunk failed - cmd: { splitChunk: "sd.metrics_110405", keyPattern:

      { accId: 1, sId: 1 }

      , min:

      { accId: 2461, sId: 35 }

      , max:

      { accId: 2845, sId: 2 }

      , from: "set2/rs2a:27018,rs2b:27018", splitKeys: [

      { accId: 2596, sId: 11 }

      ], shardId: "sd.metrics_110405-accId_2461sId_35", configdb: "config1:27019" } result: { currMin:

      { accId: 2461, sId: 35 }

      , currMax:

      { accId: 2596, sId: 11 }

      , requestedMin:

      { accId: 2461, sId: 35 }

      , requestedMax:

      { accId: 2845, sId: 2 }

      , errmsg: "chunk boundaries are outdated (likely a split occurred)", ok: 0.0 }
      Tue Apr 5 10:50:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 10:55:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:00:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:05:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:10:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:15:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:20:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:25:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:29:09 [conn124] ns: sd.servers ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 11:29:09 [conn124] ns: sd.metricsLatest ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 11:29:09 [conn124] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 11:30:05 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:35:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:38:40 [conn128] autosplitted sd.metrics_110405 shard: ns:sd.metrics_110405 at: shard2:set2/rs2a:27018,rs2b:27018 lastmod: 1|17 min:

      { accId: 2959, sId: 12 }

      max:

      { accId: 3177, sId: 6 }

      on:

      { accId: 3441, sId: 2 }

      (splitThreshold 209715200)
      Tue Apr 5 11:40:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:45:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:50:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 11:55:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:00:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:05:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:10:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:15:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:20:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:25:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:30:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:35:06 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:40:07 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:45:07 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:50:07 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:55:07 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 12:56:16 [conn124] ns: sd.servers ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 12:56:16 [conn124] ns: sd.metricsLatest ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 12:56:16 [conn124] ns: sd.alertsTriggered ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 12:56:16 [conn124] AssertionException in process: ns: sd.alertsLog doWRite
      Tue Apr 5 12:56:24 [conn124] ns: sd.users ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 12:59:43 [conn124] ns: sd.usersPhones ClusteredCursor::query ShardConnection had to change attempt: 0
      Tue Apr 5 13:00:07 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Tue Apr 5 13:05:07 [LockPinger] dist_lock pinged successfully for: ad1:1301997334:1804289383
      Received signal 6
      Backtrace: 0x52e235 0x3b71e302d0 0x3b71e30265 0x3b71e31d10 0x3b71e296e6 0x697f22 0x5035ab 0x504e64 0x69ec30 0x3b7260673d 0x3b71ed3f6d
      /home/david/mongodb/latest/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
      /lib64/libc.so.6[0x3b71e302d0]
      /lib64/libc.so.6(gsignal+0x35)[0x3b71e30265]
      /lib64/libc.so.6(abort+0x110)[0x3b71e31d10]
      /lib64/libc.so.6(__assert_fail+0xf6)[0x3b71e296e6]
      /home/david/mongodb/latest/bin/mongos(_ZN5mongo17WriteBackListener3runEv+0x19d2)[0x697f22]
      /home/david/mongodb/latest/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b)[0x5035ab]
      /home/david/mongodb/latest/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
      /home/david/mongodb/latest/bin/mongos(thread_proxy+0x80)[0x69ec30]
      /lib64/libpthread.so.0[0x3b7260673d]
      /lib64/libc.so.6(clone+0x6d)[0x3b71ed3f6d]
      ===

        1. log
          117 kB

            Assignee:
            Unassigned Unassigned
            Reporter:
            boxedice David Mytton
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: