Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3082

mongos crash on shutdown

    • Type: Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.7.3, 1.8.1
    • Component/s: Sharding
    • Environment:
      EC2 m2.xlarge instance running Amazon linux 2.6.34.7-56.40.amzn1.x86_64
    • Linux

      Whenever I attempt to shut down mongos via db.shutdownServer() after running it with active clients, shutdown fails with the following error:

      Wed May 11 09:18:58 /opt/mongodb/bin/mongos db version v1.8.1, pdfile version 4.5 starting (--help for usage)
      Wed May 11 09:18:58 git version: a429cd4f535b2499cc4130b06ff7c26f41c00f04
      Wed May 11 09:18:58 build sys info: Linux bs-linux64.10gen.cc 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
      ...
      Wed May 11 09:26:25 [conn2] terminating, shutdown command received
      Wed May 11 09:26:25 [conn2] dbexit: shutdown called rc:0 shutdown called
      Wed May 11 09:26:25 [mongosMain] dbexit: rc:0
      Received signal 11
      Backtrace: 0x52e235 0x3fb5832730 0x3fb5835c35 0x6700a9 0x5b2819 0x57bdcc 0x631062 0x66432c 0x6761c7 0x57ea3c 0x69ec30 0x3fb6407761 0x3fb58e0e7d
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
      /lib64/libc.so.6[0x3fb5832730]
      /lib64/libc.so.6(exit+0x35)[0x3fb5835c35]
      /opt/mongodb/bin/mongos[0x6700a9]
      /opt/mongodb/bin/mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb+0x89)[0x5b2819]
      /opt/mongodb/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x67c)[0x57bdcc]
      /opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x262)[0x631062]
      /opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x29c)[0x66432c]
      /opt/mongodb/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x6761c7]
      /opt/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x57ea3c]
      /opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
      /lib64/libpthread.so.0[0x3fb6407761]
      /lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
      ===
      Received signal 6
      Backtrace: 0x52e235 0x3fb5832730 0x3fb58326b5 0x3fb5833e95 0x3fb586fb1b 0x3fb5875436 0x50364e 0x504e64 0x69ec30 0x3fb6407761 0x3fb58e0e7d
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
      /lib64/libc.so.6[0x3fb5832730]
      /lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
      /lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
      /lib64/libc.so.6[0x3fb586fb1b]
      /lib64/libc.so.6[0x3fb5875436]
      /opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1ce)[0x50364e]
      /opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
      /opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
      /lib64/libpthread.so.0[0x3fb6407761]
      /lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
      ===

      Additionally, this entry appears in /var/log/messages:
      May 11 09:26:25 ip-10-243-3-246 mongos: *** glibc detected *** /opt/mongodb/bin/mongos: free(): invalid pointer: 0x0000000000ac9020 ***

      Shutdown also fails when using the "kill -2 PID" approach:

      Wed May 11 13:17:39 [mongosMain] dbexit: received signal 2 rc:0 received signal 2
      Wed May 11 13:17:40 [WriteBackListener] DBClientCursor::init call() failed
      Wed May 11 13:17:40 [WriteBackListener] DBClientCursor::init call() failed
      Received signal 6
      Backtrace: 0x52e235 0x3fb5832730 0x3fb58326b5 0x3fb5833e95 0x3fb586fb1b 0x3fb5875436 0x50364e 0x504e64 0x69ec30 0x3fb6407761 0x3fb58e0e7d
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52e235]
      /lib64/libc.so.6[0x3fb5832730]
      /lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
      /lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
      /lib64/libc.so.6[0x3fb586fb1b]
      /lib64/libc.so.6[0x3fb5875436]
      /opt/mongodb/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1ce)[0x50364e]
      /opt/mongodb/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x504e64]
      /opt/mongodb/bin/mongos(thread_proxy+0x80)[0x69ec30]
      /lib64/libpthread.so.0[0x3fb6407761]
      /lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
      ===

            Assignee:
            Unassigned Unassigned
            Reporter:
            gilby Allen Gilbert
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: