Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-52688

db.serverStatus() hang on SECONDARY server

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.2.10
    • Component/s: None
    • None
    • ALL
    • Hide

      I don't know. We just found server in such state 3 days ago.

      Show
      I don't know. We just found server in such state 3 days ago.

      Whenever I run `mongo --host mongo3:27017 --eval 'db.serverStatus()'` this command hangs for a very long time (never managed to get reply).
      mongo3 is a SECONDARY server in mongodb replica set. On PRIMARY and other SECONDARIES everything works as expected.

      Output of the following command in attachment (gdb_2020-11-09_15-58-04.txt.xz):

      gdb p $(pidof mongod) -batch -ex 'thread apply all bt' > gdb_`date +"%Y%m-%d_%H-%M-%S"`.txt

       

      This is Gentoo Linux with 4.2.10 mongodb installed.

      Another observation. I've tried to stop mongodb and it hanged with the following in logs:

      2020-11-09T16:32:51.302+0300 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
      2020-11-09T16:32:51.304+0300 I REPL [signalProcessingThread] Stepping down the ReplicationCoordinator for shutdown, waitTime: 10000ms
      2020-11-09T16:32:51.309+0300 I SHARDING [signalProcessingThread] Shutting down the WaitForMajorityService
      2020-11-09T16:32:51.313+0300 I CONTROL [signalProcessingThread] Shutting down the LogicalSessionCache
      2020-11-09T16:32:51.316+0300 I CONTROL [LogicalSessionCacheRefresh] Failed to refresh session cache: ClientMarkedKilled: client has been killed
      2020-11-09T16:32:51.320+0300 I CONTROL [LogicalSessionCacheReap] Failed to reap transaction table: ClientMarkedKilled: client has been killed
      2020-11-09T16:32:51.321+0300 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets...
      2020-11-09T16:32:51.321+0300 I NETWORK [listener] removing socket file: /tmp/mongodb-27017.sock
      2020-11-09T16:32:51.321+0300 I NETWORK [signalProcessingThread] Shutting down the global connection pool
      2020-11-09T16:32:51.321+0300 I STORAGE [signalProcessingThread] Shutting down the FlowControlTicketholder
      2020-11-09T16:32:51.321+0300 I - [signalProcessingThread] Stopping further Flow Control ticket acquisitions.
      2020-11-09T16:32:51.321+0300 I STORAGE [signalProcessingThread] Shutting down the PeriodicThreadToAbortExpiredTransactions
      2020-11-09T16:32:51.321+0300 I STORAGE [signalProcessingThread] Shutting down the PeriodicThreadToDecreaseSnapshotHistoryIfNotNeeded
      2020-11-09T16:32:51.322+0300 I REPL [signalProcessingThread] Shutting down the ReplicationCoordinator
      2020-11-09T16:32:51.322+0300 I REPL [signalProcessingThread] shutting down replication subsystems
      2020-11-09T16:32:51.322+0300 I REPL [signalProcessingThread] Stopping replication reporter thread
      2020-11-09T16:32:51.322+0300 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to mongo1:27017: CallbackCanceled: Reporter no longer valid
      2020-11-09T16:32:51.323+0300 I REPL [signalProcessingThread] Stopping replication fetcher thread
      2020-11-09T16:32:51.324+0300 I REPL [signalProcessingThread] Stopping replication applier thread
      2020-11-09T16:32:51.507+0300 I REPL [rsBackgroundSync] Replication producer stopped after oplog fetcher finished returning a batch from our sync source. Abandoning this batch of oplog entries and re-evaluating our sync source.
      2020-11-09T16:32:51.508+0300 I REPL [rsBackgroundSync] Stopping replication producer

       

      I've run gdb command (mentioned above) again and output is in gdb_2020-11-09_16-34-39.txt.xz.

      After I've killed server with -9 and started again db.serverStatus() started to work.

       

        1. gdb_2020-11-09_15-58-04.txt.xz
          26 kB
          Peter Volkov
        2. gdb_2020-11-09_16-34-39.txt.xz
          16 kB
          Peter Volkov
        3. gdb_202109-23_11-07-08.txt.xz
          11 kB
          Peter Volkov
        4. mongodb.log-20201111.xz
          5.37 MB
          Peter Volkov

            Assignee:
            edwin.zhou@mongodb.com Edwin Zhou
            Reporter:
            peter.volkov@gmail.com Peter Volkov
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: