Whenever I attempt to shut down mongos via db.shutdownServer() after running it with active clients, shutdown fails with the following error:
Wed May 11 09:18:58 /opt/mongodb/bin/mongos db version v1.8.1, pdfile version 4.5 starting (--help for usage)
Wed May 11 09:18:58 git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04
Wed May 11 09:18:58 build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
...
Wed May 11 09:26:25 [conn2] terminating, shutdown command received
Wed May 11 09:26:25 [conn2] dbexit: shutdown called rc:0 shutdown called
Wed May 11 09:26:25 [mongosMain] dbexit: rc:0
Received signal 11
Backtrace: 0x52e235 0x3fb5832730 0x3fb5835c35 0x6700a9 0x5b2819 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(exit+0x35)[0x3fb5835c35]
/opt/mongodb/bin/mongos[0x6700a9]
/opt/mongodb/bin/mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x89)[0x5b2819]
/opt/mongodb/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
/opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
/opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
/opt/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
/opt/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===
Received signal 6
Backtrace: 0x52e235 0x3fb5832730 0x3fb58326b5 0x3fb5833e95 0x3fb586fb1b 0x3fb5875436 0x50364e 0x504e64 0x69ec30 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
/lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
/lib64/libc.so.6[0x3fb586fb1b]
/lib64/libc.so.6[0x3fb5875436]
/opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1ce)[0x50364e]
/opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===
Additionally, this entry appears in /var/log/messages:
May 11 09:26:25 ip-10-243-3-246 mongos: *** glibc detected *** /opt/mongodb/bin/mongos: free(): invalid pointer: 0x0000000000ac9020 ***
Shutdown also fails when using the "kill -2 PID" approach:
Wed May 11 13:17:39 [mongosMain] dbexit: received signal 2 rc:0 received signal 2
Wed May 11 13:17:40 [WriteBackListener] DBClientCursor::init call() failed
Wed May 11 13:17:40 [WriteBackListener] DBClientCursor::init call() failed
Received signal 6
Backtrace: 0x52e235 0x3fb5832730 0x3fb58326b5 0x3fb5833e95 0x3fb586fb1b 0x3fb5875436 0x50364e 0x504e64 0x69ec30 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
/lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
/lib64/libc.so.6[0x3fb586fb1b]
/lib64/libc.so.6[0x3fb5875436]
/opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1ce)[0x50364e]
/opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===