Tue Jun 26 12:57:22 [Balancer] could not acquire lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' (another update won)
Tue Jun 26 12:57:22 [Balancer] distributed lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' was not acquired.
Tue Jun 26 12:58:10 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 12:58:09 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 12:58:53 [Balancer] distributed lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' acquired, ts : 4fe9b20d318960e603ecc88e
Tue Jun 26 12:58:57 [Balancer] distributed lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' unlocked. 
Tue Jun 26 12:59:08 [Balancer] could not acquire lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' (another update won)
Tue Jun 26 12:59:08 [Balancer] distributed lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' was not acquired.
Tue Jun 26 12:59:38 [Balancer] distributed lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' acquired, ts : 4fe9b23a318960e603ecc890
Tue Jun 26 12:59:44 [Balancer] distributed lock 'balancer/dalwork3.clarity.net:27017:1339721826:1804289383' unlocked. 
Tue Jun 26 13:00:18 [mongosMain] connection accepted from 127.0.0.1:58755 #3631
Tue Jun 26 13:00:18 [conn3631] end connection 127.0.0.1:58755
Tue Jun 26 13:03:14 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 13:03:14 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 13:05:18 [mongosMain] connection accepted from 127.0.0.1:52082 #3632
Tue Jun 26 13:05:18 [conn3632] end connection 127.0.0.1:52082
Tue Jun 26 13:08:19 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 13:08:18 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 13:10:18 [mongosMain] connection accepted from 127.0.0.1:46310 #3633
Tue Jun 26 13:10:18 [conn3633] end connection 127.0.0.1:46310
Tue Jun 26 13:13:26 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 13:13:25 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 13:15:18 [mongosMain] connection accepted from 127.0.0.1:51249 #3634
Tue Jun 26 13:15:18 [conn3634] end connection 127.0.0.1:51249
Tue Jun 26 13:18:30 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 13:18:29 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 13:20:19 [mongosMain] connection accepted from 127.0.0.1:34479 #3635
Tue Jun 26 13:20:19 [conn3635] end connection 127.0.0.1:34479
Tue Jun 26 13:23:34 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 13:23:34 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 13:25:19 [mongosMain] connection accepted from 127.0.0.1:47254 #3636
Tue Jun 26 13:25:19 [conn3636] end connection 127.0.0.1:47254
Tue Jun 26 13:28:39 [LockPinger] cluster mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019 pinged successfully at Tue Jun 26 13:28:38 2012 by distributed lock pinger 'mcfg1.clarity.net:27019,mcfg2.clarity.net:27019,mcfg3.clarity.net:27019/dalwork3.clarity.net:27017:1339721826:1804289383', sleeping for 30000ms
Tue Jun 26 13:30:19 [mongosMain] connection accepted from 127.0.0.1:46141 #3637
Tue Jun 26 13:30:19 [conn3637] end connection 127.0.0.1:46141
Tue Jun 26 13:30:37 [mongosMain] connection accepted from 127.0.0.1:46322 #3638
Tue Jun 26 13:30:37 [conn3638] getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Tue Jun 26 13:30:37 [conn3638] DBException in process: socket exception
Tue Jun 26 13:30:37 [conn3532] getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Tue Jun 26 13:30:37 [conn3532] warning: could not get last error from a shard dalmdbc2.clarity.net:27018 :: caused by :: socket exception
Received signal 6
Backtrace: 0x54e5b5 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 0x7c89c1 0x5e9747 0x30fb20673d 0x30fa2d44bd 
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e5b5]
/lib64/libc.so.6[0x30fa2302d0]
/lib64/libc.so.6(gsignal+0x35)[0x30fa230265]
/lib64/libc.so.6(abort+0x110)[0x30fa231d10]
/lib64/libc.so.6(__assert_fail+0xf6)[0x30fa2296e6]
/usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec)[0x7b8d5c]
/usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46)[0x77bb16]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894)[0x794f24]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac)[0x777c5c]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x187)[0x7b6467]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)[0x7c89c1]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)[0x5e9747]
/lib64/libpthread.so.0[0x30fb20673d]
/lib64/libc.so.6(clone+0x6d)[0x30fa2d44bd]
===
 CursorCache at shutdown -  sharded: 8 passthrough: 0
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c1173 0x7c033d 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x2b3) [0x7c1173]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0xfd) [0x7c033d]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c1173 0x7c033d 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x2b3) [0x7c1173]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0xfd) [0x7c033d]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c1193 0x7c033d 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x2d3) [0x7c1193]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0xfd) [0x7c033d]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c0366 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 0x7c89c1 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0x126) [0x7c0366]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) [0x7c89c1]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c1193 0x7c033d 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E+0x2d3) [0x7c1193]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0xfd) [0x7c033d]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c0366 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 0x7c89c1 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0x126) [0x7c0366]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) [0x7c89c1]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 getaddrinfo("dalmdbc2.clarity.net") failed: Name or service not known
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37 Assertion: 13548:BufBuilder grow() > 64MB
0x5355f2 0x4b6436 0x562bd4 0x592f09 0x5eb002 0x5edbff 0x7bdea1 0x7c0366 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 0x7b6467 0x7c89c1 
 /usr/bin/mongos(_ZN5mongo11msgassertedEiPKc+0x112) [0x5355f2]
 /usr/bin/mongos [0x4b6436]
 /usr/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSsd+0x1c4) [0x562bd4]
 /usr/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x8a9) [0x592f09]
 /usr/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x62) [0x5eb002]
 /usr/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev+0x4f) [0x5edbff]
 /usr/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21) [0x7bdea1]
 /usr/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0x126) [0x7c0366]
 /lib64/libc.so.6(exit+0xe5) [0x30fa2334f5]
 /usr/bin/mongos [0x54e61b]
 /lib64/libc.so.6 [0x30fa2302d0]
 /lib64/libc.so.6(gsignal+0x35) [0x30fa230265]
 /lib64/libc.so.6(abort+0x110) [0x30fa231d10]
 /lib64/libc.so.6(__assert_fail+0xf6) [0x30fa2296e6]
 /usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec) [0x7b8d5c]
 /usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46) [0x77bb16]
 /usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894) [0x794f24]
 /usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac) [0x777c5c]
 /usr/bin/mongos(_ZN5mongo7Request7processEi+0x187) [0x7b6467]
 /usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) [0x7c89c1]
Logstream::get called in uninitialized state
Tue Jun 26 13:30:37   caught exception (BufBuilder grow() > 64MB) in destructor (~DBClientCursor)
*** glibc detected *** /usr/bin/mongos: corrupted double-linked list: 0x00000000117888c0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x30fa270793]
/lib64/libc.so.6[0x30fa27239c]
/lib64/libc.so.6(cfree+0x4b)[0x30fa2728bb]
/usr/bin/mongos(_ZNSt8_Rb_treeIN5mongo17WriteBackListener15ConnectionIdentESt4pairIKS2_NS1_8WBStatusEESt10_Select1stIS6_ESt4lessIS2_ESaIS6_EE8_M_eraseEPSt13_Rb_tree_nodeIS6_E+0x4dc)[0x7f399c]
/usr/bin/mongos[0x7ecbf0]
/lib64/libc.so.6(exit+0xe5)[0x30fa2334f5]
/usr/bin/mongos[0x54e61b]
/lib64/libc.so.6[0x30fa2302d0]
/lib64/libc.so.6(gsignal+0x35)[0x30fa230265]
/lib64/libc.so.6(abort+0x110)[0x30fa231d10]
/lib64/libc.so.6(__assert_fail+0xf6)[0x30fa2296e6]
/usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec)[0x7b8d5c]
/usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46)[0x77bb16]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894)[0x794f24]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac)[0x777c5c]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x187)[0x7b6467]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71)[0x7c89c1]
/usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)[0x5e9747]
/lib64/libpthread.so.0[0x30fb20673d]
/lib64/libc.so.6(clone+0x6d)[0x30fa2d44bd]
======= Memory map: ========
00400000-008fe000 r-xp 00000000 fd:00 7737333                            /usr/bin/mongos
00afe000-00b16000 rw-p 004fe000 fd:00 7737333                            /usr/bin/mongos
00b16000-00b19000 rw-p 00b16000 00:00 0 
00d15000-00d83000 rw-p 00515000 fd:00 7737333                            /usr/bin/mongos
11045000-11cad000 rw-p 11045000 00:00 0                                  [heap]
4021f000-40220000 ---p 4021f000 00:00 0 
40220000-40320000 rw-p 40220000 00:00 0 
40320000-40321000 ---p 40320000 00:00 0 
40321000-40421000 rw-p 40321000 00:00 0 
40955000-40956000 ---p 40955000 00:00 0 
40956000-40a56000 rw-p 40956000 00:00 0 
40b35000-40b36000 ---p 40b35000 00:00 0 
40b36000-41536000 rw-p 40b36000 00:00 0 
41536000-41537000 ---p 41536000 00:00 0 
41537000-41f37000 rw-p 41537000 00:00 0 
41f37000-41f38000 ---p 41f37000 00:00 0 
41f38000-42938000 rw-p 41f38000 00:00 0 
42938000-42939000 ---p 42938000 00:00 0 
42939000-43339000 rw-p 42939000 00:00 0 
43339000-4333a000 ---p 43339000 00:00 0 
4333a000-43d3a000 rw-p 4333a000 00:00 0 
43d3a000-43d3b000 ---p 43d3a000 00:00 0 
43d3b000-4473b000 rw-p 43d3b000 00:00 0 
4473b000-4473c000 ---p 4473b000 00:00 0 
4473c000-4513c000 rw-p 4473c000 00:00 0 
4523d000-4523e000 ---p 4523d000 00:00 0 
4523e000-45c3e000 rw-p 4523e000 00:00 0 
45c3e000-45c3f000 ---p 45c3e000 00:00 0 
45c3f000-45d3f000 rw-p 45c3f000 00:00 0 
45d3f000-45d40000 ---p 45d3f000 00:00 0 
45d40000-45e40000 rw-p 45d40000 00:00 0 
45e40000-45e41000 ---p 45e40000 00:00 0 
45e41000-45f41000 rw-p 45e41000 00:00 0 
45f41000-45f42000 ---p 45f41000 00:00 0 
45f42000-46042000 rw-p 45f42000 00:00 0 
46143000-46144000 ---p 46143000 00:00 0 
46144000-46244000 rw-p 46144000 00:00 0 
46244000-46245000 ---p 46244000 00:00 0 
46245000-46345000 rw-p 46245000 00:00 0 
46446000-46447000 ---p 46446000 00:00 0 
46447000-46547000 rw-p 46447000 00:00 0 
46547000-46548000 ---p 46547000 00:00 0 
46548000-46648000 rw-p 46548000 00:00 0 
46648000-46649000 ---p 46648000 00:00 0 
46649000-46749000 rw-p 46649000 00:00 0 
46749000-4674a000 ---p 46749000 00:00 0 
4674a000-4684a000 rw-p 4674a000 00:00 0 
4684a000-4684b000 ---p 4684a000 00:00 0 
4684b000-4694b000 rw-p 4684b000 00:00 0 
4694b000-4694c000 ---p 4694b000 00:00 0 
4694c000-46a4c000 rw-p 4694c000 00:00 0 
46b4d000-46b4e000 ---p 46b4d000 00:00 0 
46b4e000-46c4e000 rw-p 46b4e000 00:00 0 
46c4e000-46c4f000 ---p 46c4e000 00:00 0 
46c4f000-46d4f000 rw-p 46c4f000 00:00 0 
46d4f000-46d50000 ---p 46d4f000 00:00 0 
46d50000-46e50000 rw-p 46d50000 00:00 0 
46e50000-46e51000 ---p 46e50000 00:00 0 
46e51000-46f51000 rw-p 46e51000 00:00 0 
46f51000-46f52000 ---p 46f51000 00:00 0 
46f52000-47052000 rw-p 46f52000 00:00 0 
47254000-47255000 ---p 47254000 00:00 0 
47255000-47355000 rw-p 47255000 00:00 0 
4785a000-4785b000 ---p 4785a000 00:00 0 
4785b000-4825b000 rw-p 4785b000 00:00 0 
4825b000-4825c000 ---p 4825b000 00:00 0 
4825c000-48c5c000 rw-p 4825c000 00:00 0 
4b460000-4b461000 ---p 4b460000 00:00 0 
4b461000-4be61000 rw-p 4b461000 00:00 0 
4be61000-4be62000 ---p 4be61000 00:00 0 
4be62000-4c862000 rw-p 4be62000 00:00 0 
4d263000-4d264000 ---p 4d263000 00:00 0 
4d264000-4dc64000 rw-p 4d264000 00:00 0 
55e71000-55e72000 ---p 55e71000 00:00 0 
55e72000-56872000 rw-p 55e72000 00:00 0 
56872000-56873000 ---p 56872000 00:00 0 
56873000-57273000 rw-p 56873000 00:00 0 
57273000-57274000 ---p 57273000 00:00 0 
57274000-57c74000 rw-p 57274000 00:00 0 
5a478000-5a479000 ---p 5a478000 00:00 0 
5a479000-5ae79000 rw-p 5a479000 00:00 0 
5ae79000-5ae7a000 ---p 5ae79000 00:00 0 
5ae7a000-5b87a000 rw-p 5ae7a000 00:00 0 
5c27b000-5c27c000 ---p 5c27b000 00:00 0 
5c27c000-5cc7c000 rw-p 5c27c000 00:00 0 
30f9e00000-30f9e1c000 r-xp 00000000 fd:00 5668868                        /lib64/ld-2.5.so
30fa01c000-30fa01d000 r--p 0001c000 fd:00 5668868                        /lib64/ld-2.5.so
30fa01d000-30fa01e000 rw-p 0001d000 fd:00 5668868                        /lib64/ld-2.5.so
30fa200000-30fa34e000 r-xp 00000000 fd:00 5668875                        /lib64/libc-2.5.so
30fa34e000-30fa54e000 ---p 0014e000 fd:00 5668875                        /lib64/libc-2.5.so
30fa54e000-30fa552000 r--p 0014e000 fd:00 5668875                        /lib64/libc-2.5.so
30fa552000-30fa553000 rw-p 00152000 fd:00 5668875                        /lib64/libc-2.5.so
30fa553000-30fa558000 rw-p 30fa553000 00:00 0 
30fb200000-30fb216000 r-xp 00000000 fd:00 5668891                        /lib64/libpthread-2.5.so
30fb216000-30fb415000 ---p 00016000 fd:00 5668891                        /lib64/libpthread-2.5.so
30fb415000-30fb416000 r--p 00015000 fd:00 5668891                        /lib64/libpthread-2.5.so
30fb416000-30fb417000 rw-p 00016000 fd:00 5668891                        /lib64/libpthread-2.5.so
30fb417000-30fb41b000 rw-p 30fb417000 00:00 0 
30fb600000-30fb682000 r-xp 00000000 fd:00 5668899                        /lib64/libm-2.5.so
30fb682000-30fb881000 ---p 00082000 fd:00 5668899                        /lib64/libm-2.5.so
30fb881000-30fb882000 r--p 00081000 fd:00 5668899                        /lib64/libm-2.5.so
30fb882000-30fb883000 rw-p 00082000 fd:00 5668899                        /lib64/libm-2.5.so
30fd600000-30fd60d000 r-xp 00000000 fd:00 5668903                        /lib64/libgcc_s-4.1.2-20080825.so.1
30fd60d000-30fd80d000 ---p 0000d000 fd:00 5668903                        /lib64/libgcc_s-4.1.2-20080825.so.1
30fd80d000-30fd80e000 rw-p 0000d000 fd:00 5668903                        /lib64/libgcc_s-4.1.2-20080825.so.1
322c600000-322c6e6000 r-xp 00000000 fd:00 7748049                        /usr/lib64/libstdc++.so.6.0.8
322c6e6000-322c8e5000 ---p 000e6000 fd:00 7748049                        /usr/lib64/libstdc++.so.6.0.8
322c8e5000-322c8eb000 r--p 000e5000 fd:00 7748049                        /usr/lib64/libstdc++.so.6.0.8
322c8eb000-322c8ee000 rw-p 000eb000 fd:00 7748049                        /usr/lib64/libstdc++.so.6.0.8
322c8ee000-322c900000 rw-p 322c8ee000 00:00 0 
2aaaaaae5000-2aaaaaae6000 rw-p 2aaaaaae5000 00:00 0 
2aaaac000000-2aaaac958000 rw-p 2aaaac000000 00:00 0 
2aaaac958000-2aaab0000000 ---p 2aaaac958000 00:00 0 
2aaab0000000-2aaab087e000 rw-p 2aaab0000000 00:00 0 
2aaab087e000-2aaab4000000 ---p 2aaab087e000 00:00 0 
2aaab4000000-2aaab7e0b000 rw-p 2aaab4000000 00:00 0 
2aaab7e0b000-2aaab8000000 ---p 2aaab7e0b000 00:00 0 
2aeac4197000-2aeac419a000 rw-p 2aeac4197000 00:00 0 
2aeac41a1000-2aeac41a4000 rw-p 2aeac41a1000 00:00 0 
2aeac41a4000-2aeac41d9000 r--s 00000000 fd:00 3309657                    /var/db/nscd/hosts
7fff6d59f000-7fff6d5b4000 rw-p 7ffffffe9000 00:00 0                      [stack]
7fff6d5fc000-7fff6d600000 r-xp 7fff6d5fc000 00:00 0                      [vdso]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  [vsyscall]
Received signal 6
Backtrace: 0x54e5b5 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa26a99b 0x30fa270793 0x30fa27239c 0x30fa2728bb 0x7f399c 0x7ecbf0 0x30fa2334f5 0x54e61b 0x30fa2302d0 0x30fa230265 0x30fa231d10 0x30fa2296e6 0x7b8d5c 0x77bb16 0x794f24 0x777c5c 
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e5b5]
/lib64/libc.so.6[0x30fa2302d0]
/lib64/libc.so.6(gsignal+0x35)[0x30fa230265]
/lib64/libc.so.6(abort+0x110)[0x30fa231d10]
/lib64/libc.so.6[0x30fa26a99b]
/lib64/libc.so.6[0x30fa270793]
/lib64/libc.so.6[0x30fa27239c]
/lib64/libc.so.6(cfree+0x4b)[0x30fa2728bb]
/usr/bin/mongos(_ZNSt8_Rb_treeIN5mongo17WriteBackListener15ConnectionIdentESt4pairIKS2_NS1_8WBStatusEESt10_Select1stIS6_ESt4lessIS2_ESaIS6_EE8_M_eraseEPSt13_Rb_tree_nodeIS6_E+0x4dc)[0x7f399c]
/usr/bin/mongos[0x7ecbf0]
/lib64/libc.so.6(exit+0xe5)[0x30fa2334f5]
/usr/bin/mongos[0x54e61b]
/lib64/libc.so.6[0x30fa2302d0]
/lib64/libc.so.6(gsignal+0x35)[0x30fa230265]
/lib64/libc.so.6(abort+0x110)[0x30fa231d10]
/lib64/libc.so.6(__assert_fail+0xf6)[0x30fa2296e6]
/usr/bin/mongos(_ZN5mongo10ClientInfo12getLastErrorERKNS_7BSONObjERNS_14BSONObjBuilderEb+0x13ec)[0x7b8d5c]
/usr/bin/mongos(_ZN5mongo11dbgrid_cmds23CmdShardingGetLastError3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x46)[0x77bb16]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x894)[0x794f24]
/usr/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x5ac)[0x777c5c]
===
Received signal 11
Backtrace: 0x54e5b5 0x30fa2302d0 0x11049df0 
/usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e5b5]
/lib64/libc.so.6[0x30fa2302d0]
[0x11049df0]
===


***** SERVER RESTARTED *****


Tue Jun 26 13:37:28 /usr/bin/mongos db version v2.0.6, pdfile version 4.5 starting (--help for usage)
Tue Jun 26 13:37:28 git version: e1c0cbc25863f6356aa4e31375add7bb49fb05bc
Tue Jun 26 13:37:28 build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Tue Jun 26 13:37:29 SyncClusterConnection connecting to [mcfg1.clarity.net:27019]
Tue Jun 26 13:37:29 SyncClusterConnection connecting to [mcfg2.clarity.net:27019]
Tue Jun 26 13:37:29 SyncClusterConnection connecting to [mcfg3.clarity.net:27019]
Tue Jun 26 13:37:29 [mongosMain] waiting for connections on port 27017
Tue Jun 26 13:37:29 [websvr] admin web console waiting for connections on port 28017
Tue Jun 26 13:37:29 [Balancer] about to contact config servers and shards
Tue Jun 26 13:37:29 [Balancer] SyncClusterConnection connecting to [mcfg1.clarity.net:27019]
Tue Jun 26 13:37:29 [Balancer] SyncClusterConnection connecting to [mcfg2.clarity.net:27019]
Tue Jun 26 13:37:29 [Balancer] SyncClusterConnection connecting to [mcfg3.clarity.net:27019]

...