-
Type: Bug
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Reactive Streams
-
Fully Compatible
-
Not Needed
We are observing a memory leak related to com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup
Driver version is 4.3.4
Server is Azure CosmosDB using MongoDB Server version 4.0, server-side retries enabled, SSL enabled.
Application is using spring-boot 2.4.5 and reactive stack.
The leak starts after the following exception is thrown:
{{java.nio.channels.ClosedChannelException
at java.base/java.nio.channels.spi.AbstractSelectableChannel.register(Unknown Source)
at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.registerPendingSockets(AsynchronousTlsChannelGroup.java:612)
at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.loop(AsynchronousTlsChannelGroup.java:414)
at java.base/java.lang.Thread.run(Unknown Source)}}
The leak is in com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup#pendingRegistrations
Pending registrations are accumulating and eventually causing OOM.
It seems that com.mongodb.connection.TlsChannelStreamFactoryFactory.TlsChannelStream#openAsync will continue to create AsynchronousTlsChannel instances addig new pending registrations even after the com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup#loop was shut down.
Not exactly clear what is triggering this behavior as we only observed it on one of our environments. It also needs decent time before it happens. Last time it took 2 days before it started.
There are other exceptions thrown prior to this one but hours or days before so not necessarily related:
java.lang.NullPointerException: Cannot invoke "com.mongodb.internal.connection.tlschannel.impl.BufferHolder.prepare()" because "this.outEncrypted" is null at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.wrapAndWrite(TlsChannelImpl.java:393) at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.write(TlsChannelImpl.java:384) at com.mongodb.internal.connection.tlschannel.ClientTlsChannel.write(ClientTlsChannel.java:184) at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.writeHandlingTasks(AsynchronousTlsChannelGroup.java:540) at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doWrite(AsynchronousTlsChannelGroup.java:498) at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processWrite$4(AsynchronousTlsChannelGroup.java:459) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)
Another one is:
error in operation java.lang.NullPointerException
Most probably thrown from{{ com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup#processWrite}}
- links to