Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-44490

The mongo command connecting mongodb instance hangs forever.

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Fully Compatible
    • ALL
    • Hide

      [^diagnostic.data.tar.gz]

      Show
      [^diagnostic.data.tar.gz]

      We are now facing a issue, which trying to connect to the mongod through mongo client hangs forever. Nothing response, until Ctr+C stop the attempt. And the output of the stop is,

      MongoDB shell version v3.6.8
      connecting to: mongodb://127.0.0.1:28000/admin
      MongoDB server version: 3.6.8
      ^C2019-11-08T10:09:59.607+0800 E - [main] Error saving history file: FileOpenFailed: Unable to open() file : No such file or directory
      2019-11-08T10:09:59.607+0800 I CONTROL [main] shutting down with code:0

       

      Another strange thing is that the dbpath of the node is pretty larger than other replica set member. The Primary node's data dir is about 20GB , and the node is about 181GB. When I list all the files in the dbpath, the WiredTigerLAS.wt file is much large, 171GB. It is wired, because we cannot connect to the node any more, so why the WT cache is ALWAYS growing.

       

      We also try to dump the stack info while trying to connect, and here is the log:

      _Thread 3 (Thread 0x7fab3e99d700 (LWP 20357)):_
       _#0 0x00007fab40196d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0_
       _#1 0x000055f1e0133407 in mongo::DeadlineMonitor<mongo::mozjs::MozJSImplScope>::deadlineMonitorThread() ()_
       _#2 0x000055f1e0e3c2b0 in execute_native_thread_routine ()_
       _#3 0x00007fab40192dd5 in start_thread () from /lib64/libpthread.so.0_
       _#4 0x00007fab3febbead in clone () from /lib64/libc.so.6_
       _Thread 2 (Thread 0x7fab3e19c700 (LWP 20358)):_
       _#0 0x00007fab40199a8b in recv () from /lib64/libpthread.so.0_
       _#1 0x000055f1e0be90b4 in mongo::Socket::_recv(char*, int) ()_
       _#2 0x000055f1e0be9137 in mongo::Socket::recv(char*, int) ()_
       _#3 0x000055f1e0be0c78 in mongo::MessagingPort::recv(mongo::Message&) ()_
       _#4 0x000055f1e0be1f45 in mongo::MessagingPort::call(mongo::Message const&, mongo::Message&) ()_
       _#5 0x000055f1e0754b2f in mongo::DBClientConnection::call(mongo::Message&, mongo::Message&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) ()_
       _#6 0x000055f1e074998c in mongo::DBClientBase::runCommandWithTarget(mongo::OpMsgRequest) ()_
       _#7 0x000055f1e074e781 in mongo::DBClientConnection::runCommandWithTarget(mongo::OpMsgRequest) ()_
       _#8 0x000055f1e074c36a in std::_Function_handler<void (mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>), mongo::DBClientBase::_auth(mongo::BSONObj const&)::\{lambda(mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>)#1}>::_M_invoke(std::_Any_data const&, mongo::executor::RemoteCommandRequest&&, std::function<void (mongo::executor::RemoteCommandResponse)>&&) ()_
       _#9 0x000055f1e080f79d in mongo::(anonymous namespace)::asyncSaslConversation(std::function<void (mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>)>, std::shared_ptr<mongo::SaslClientSession> const&, mongo::BSONObj const&, mongo::BSONObj const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::function<void (mongo::executor::RemoteCommandResponse)>) [clone .constprop.74] ()_
       _#10 0x000055f1e0813c29 in mongo::(anonymous namespace)::saslClientAuthenticateImpl(std::function<void (mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>)>, mongo::HostAndPort const&, mongo::BSONObj const&, std::function<void (mongo::executor::RemoteCommandResponse)>) ()_
       _#11 0x000055f1e080903e in mongo::auth::(anonymous namespace)::auth(std::function<void (mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>)>, mongo::BSONObj const&, mongo::HostAndPort const&, mongo::StringData, std::function<void (mongo::executor::RemoteCommandResponse)>) [clone .constprop.88] ()_
       _#12 0x000055f1e0809f1c in mongo::auth::(anonymous namespace)::asyncAuth(std::function<void (mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>)>, mongo::BSONObj const&, mongo::HostAndPort const&, mongo::StringData, std::function<void (mongo::executor::RemoteCommandResponse)>) ()_
       _#13 0x000055f1e080a11c in mongo::auth::authenticateClient(mongo::BSONObj const&, mongo::HostAndPort const&, mongo::StringData, std::function<void (mongo::executor::RemoteCommandRequest, std::function<void (mongo::executor::RemoteCommandResponse)>)>, std::function<void (mongo::executor::RemoteCommandResponse)>) ()_
       _#14 0x000055f1e07483f2 in mongo::DBClientBase::_auth(mongo::BSONObj const&) ()_
       _#15 0x000055f1e0753d5a in mongo::DBClientConnection::_auth(mongo::BSONObj const&) ()_
       _#16 0x000055f1e014ecf3 in mongo::mozjs::MongoBase::Functions::auth::call(JSContext*, JS::CallArgs) ()_
       _#17 0x000055f1e015711f in bool mongo::mozjs::smUtils::wrapConstrainedMethod<mongo::mozjs::MongoBase::Functions::auth, true, mongo::mozjs::MongoExternalInfo>(JSContext*, unsigned int, JS::Value*) ()_
       _#18 0x000055f1e05d37cb in js::Invoke(JSContext*, JS::CallArgs const&, js::MaybeConstruct) ()_
       _#19 0x000055f1e05cf171 in Interpret(JSContext*, js::RunState&) ()_
       _#20 0x000055f1e05d3543 in js::RunScript(JSContext*, js::RunState&) ()_
       _#21 0x000055f1e05d9f3f in js::ExecuteKernel(JSContext*, JS::Handle<JSScript*>, JSObject&, JS::Value const&, js::ExecuteType, js::AbstractFramePtr, JS::Value*) ()_
       _#22 0x000055f1e05da0a9 in js::Execute(JSContext*, JS::Handle<JSScript*>, JSObject&, JS::Value*) ()_
       _#23 0x000055f1e04800ca in ExecuteScript(JSContext*, JS::Handle<JSObject*>, JS::Handle<JSScript*>, JS::Value*) ()_
       _#24 0x000055f1e0480170 in JS_ExecuteScript(JSContext*, JS::Handle<JSScript*>, JS::MutableHandle<JS::Value>) ()_
       _#25 0x000055f1e0140351 in mongo::mozjs::MozJSImplScope::exec(mongo::StringData, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, int) ()_
       _#26 0x000055f1e00fdc10 in mongo::shell_utils::initScope(mongo::Scope&) ()_
       _#27 0x000055f1e01435d5 in mongo::mozjs::MozJSImplScope::MozJSImplScope(mongo::mozjs::MozJSScriptEngine*) ()_
       _#28 0x000055f1e01727d5 in mongo::mozjs::MozJSProxyScope::implThread(void*) ()_
       _#29 0x000055f1e016e37c in nspr::Thread::ThreadRoutine(void*) ()_
       _#30 0x000055f1e0e3c2b0 in execute_native_thread_routine ()_
       _#31 0x00007fab40192dd5 in start_thread () from /lib64/libpthread.so.0_
       _#32 0x00007fab3febbead in clone () from /lib64/libc.so.6_
       _Thread 1 (Thread 0x7fab417c8a80 (LWP 20356)):_
       _#0 0x00007fab40196965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0_
       _#1 0x000055f1e0e3969c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()_
       _#2 0x000055f1e016fed3 in mongo::mozjs::MozJSProxyScope::runOnImplThread(std::function<void ()>) ()_
       _#3 0x000055f1e01700c8 in mongo::mozjs::MozJSProxyScope::MozJSProxyScope(mongo::mozjs::MozJSScriptEngine*) ()_
       _#4 0x000055f1e0130332 in mongo::mozjs::MozJSScriptEngine::createScope() ()_
       _#5 0x000055f1e00d2280 in _main(int, char**, char**) ()_
       _#6 0x000055f1e009da8c in main ()_
      

       

      We searched but not found any clue.

      The MongoDB is running in the docker container, and the version of mongo is 3.6.8. 

      We also attached the diagnostic.data.

       

       

        1. gdb_output.txt
          5 kB
          Chengcheng Ma
        2. mongod.log
          74 kB
          Chengcheng Ma
        3. node-3.diagnostic.data.tar.gz
          1.72 MB
          Chengcheng Ma

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            cora_ma Chengcheng Ma
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: