Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-2238

Connection and thread leakage in DefaultServerMonitor

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 3.3.0
    • Affects Version/s: 3.0.0
    • Component/s: Connection Management
    • None

      When experiencing network issues between mongos and application server, we found that there is massive connection and thread leakage. In jstack this looks like this:

      "cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5957 daemon prio=5 os_prio=0 tid=0x00007f75e039c000 nid=0x18185 waiting on condition [0x00007f6b00d4c000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x000000038c39b358> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)
      
      "cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5956 daemon prio=5 os_prio=0 tid=0x00007f6be819a800 nid=0x18184 waiting on condition [0x00007f6b00a49000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x0000000396151050> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)
      
      "cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5948 daemon prio=5 os_prio=0 tid=0x00007f7620083800 nid=0x1817c waiting on condition [0x00007f6b00b4a000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
              at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
              - locked <0x0000000396151230> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
              at java.lang.Thread.run(Thread.java:745)
      

      And up to 6-7 thousands of these useless threads per application server. At that point our mongos was killed by "too many opened files" error.

            Assignee:
            jeff.yemin@mongodb.com Jeffrey Yemin
            Reporter:
            alexbool Alexander Bulaev
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: