-
Type:
Bug
-
Resolution: Done
-
Priority:
Critical - P2
-
Affects Version/s: 3.0.0
-
Component/s: Connection Management
-
None
When experiencing network issues between mongos and application server, we found that there is massive connection and thread leakage. In jstack this looks like this:
"cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5957 daemon prio=5 os_prio=0 tid=0x00007f75e039c000 nid=0x18185 waiting on condition [0x00007f6b00d4c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168) - locked <0x000000038c39b358> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable) at java.lang.Thread.run(Thread.java:745) "cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5956 daemon prio=5 os_prio=0 tid=0x00007f6be819a800 nid=0x18184 waiting on condition [0x00007f6b00a49000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168) - locked <0x0000000396151050> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable) at java.lang.Thread.run(Thread.java:745) "cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5948 daemon prio=5 os_prio=0 tid=0x00007f7620083800 nid=0x1817c waiting on condition [0x00007f6b00b4a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219) at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168) - locked <0x0000000396151230> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable) at java.lang.Thread.run(Thread.java:745)
And up to 6-7 thousands of these useless threads per application server. At that point our mongos was killed by "too many opened files" error.
- is duplicated by
-
-
- Closed
-