Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-4122

Getting Exceptions when CPU starving the driver

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Unknown Unknown
    • None
    • Affects Version/s: 4.2.3
    • Component/s: Scala
    • None

      Using the Scala driver (4.2.3) when we switched over to the use of an Atlas cluster we started noticing some instabilities in our application. 

      We where for instance getting

       com.mongodb.MongoInternalException: Unexpected runtime exception
      	at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:638)
      	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:591)
      	at com.mongodb.internal.connection.InternalStreamConnection.access$1100(InternalStreamConnection.java:78)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:701)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:686)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:581)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:578)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:250)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:233)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$4(AsynchronousTlsChannel.java:122)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$5(AsynchronousTlsChannel.java:122)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doRead(AsynchronousTlsChannelGroup.java:569)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processRead$5(AsynchronousTlsChannelGroup.java:475)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.processRead(AsynchronousTlsChannelGroup.java:472)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.loop(AsynchronousTlsChannelGroup.java:407)
      	at java.base/java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.IllegalArgumentException: capacity < 0: (-16 < 0)
      	at java.base/java.nio.Buffer.createCapacityException(Unknown Source)
      	at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
      	at com.mongodb.internal.connection.PowerOfTwoBufferPool.createNew(PowerOfTwoBufferPool.java:90)
      	at com.mongodb.internal.connection.PowerOfTwoBufferPool.getByteBuffer(PowerOfTwoBufferPool.java:82)
      	at com.mongodb.internal.connection.PowerOfTwoBufferPool.getBuffer(PowerOfTwoBufferPool.java:77)
      	at com.mongodb.internal.connection.AsynchronousChannelStream.readAsync(AsynchronousChannelStream.java:111)
      	at com.mongodb.internal.connection.AsynchronousChannelStream.readAsync(AsynchronousChannelStream.java:107)
      	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:578)
      	... 23 common frames omitted

      and

      com.mongodb.MongoInternalException: The reply message length 680237824 is greater than the maximum message length 48000000
      	at com.mongodb.internal.connection.MessageHeader.<init>(MessageHeader.java:41)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:700)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:686)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:581)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:578)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:250)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:233)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$4(AsynchronousTlsChannel.java:122)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$5(AsynchronousTlsChannel.java:122)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doRead(AsynchronousTlsChannelGroup.java:569)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processRead$5(AsynchronousTlsChannelGroup.java:475)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.processRead(AsynchronousTlsChannelGroup.java:472)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.loop(AsynchronousTlsChannelGroup.java:407)
      	at java.base/java.lang.Thread.run(Unknown Source) 

      It seems that the driver is throwing these exceptions when it's starved for CPU. We are running our application on K8s and have specific resources set. After doing some load testing the exception seems to disappear after we double the cpu resources to be able to handle 50% of the same traffic. 

      Can it be something is not thread safe?

      Why we need at least double the cpu resources I don't know yet.

            Assignee:
            jeff.yemin@mongodb.com Jeffrey Yemin
            Reporter:
            jork.zijlstra@nielsen.com Jork Zijlstra
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: