Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-4387

MongoDb returns message length larger than the max message length

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Not Needed

      Hi guys,

      We are facing an issue that happened several times a day.  Our java driver version is 4.0.5 and Mongo DB version is 4.0.9. We have 2 routers and several nodes. And we are using reactive stream.  This issue only happens in our production env. It usually first occurred while MongodbHealthIndicator checking the connections with mongo db. Please see the stacktrace as below:

      org.springframework.dao.InvalidDataAccessResourceUsageException: The reply message length 1013281644 is greater than the maximum message length 48000000; nested exception is com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000org.springframework.dao.InvalidDataAccessResourceUsageException: The reply message length 1013281644 is greater than the maximum message length 48000000; nested exception is com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000 at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): |_ checkpoint ⇢ com.active.platform.modules.pos.service.config.ContextFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ HTTP GET "/agencies/f953b9de-2260-4833-bf32-d7be2a9ae945/catalogs" [ExceptionHandlingWebHandler]Stack trace: at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92) at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814) at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797) at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3286) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) at com.mongodb.reactivestreams.client.internal.AbstractSubscription.onError(AbstractSubscription.java:142) at com.mongodb.reactivestreams.client.internal.SingleResultCallbackSubscription.lambda$requestInitialData$0(SingleResultCallbackSubscription.java:41) at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:188) at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:184) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.async.client.OperationExecutorImpl$1$1$1.onResult(OperationExecutorImpl.java:92) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.operation.FindOperation$3.onResult(FindOperation.java:724) at com.mongodb.internal.operation.OperationHelper$ReferenceCountedReleasingWrappedCallback.onResult(OperationHelper.java:411) at com.mongodb.internal.operation.CommandOperationHelper$10.checkRetryableException(CommandOperationHelper.java:493) at com.mongodb.internal.operation.CommandOperationHelper$10.onResult(CommandOperationHelper.java:478) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor$2.onResult(DefaultServer.java:248) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.CommandProtocolImpl$1.onResult(CommandProtocolImpl.java:82) at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:517) at com.mongodb.internal.connection.UsageTrackingInternalConnection$2.onResult(UsageTrackingInternalConnection.java:111) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:380) at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:635) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510) at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232) at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510) at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)Caused by: com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000 at com.mongodb.internal.connection.MessageHeader.<init>(MessageHeader.java:41) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:632) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510) at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232) at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510) at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

       
      After about 20+ times of this error(the reply message length is different every time). The error will become

      
      The responseTo (2342443) in the response does not match the requestId (2342522) in the request
      

      *The responseTo is always smaller than the request id *

      The full stack trace is

      org.springframework.dao.InvalidDataAccessResourceUsageException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request; nested exception is com.mongodb.MongoInternalException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request
      	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92)
      	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
      Error has been observed at the following site(s):
      	|_ checkpoint ⇢ com.active.platform.modules.pos.service.config.ContextFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
      	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
      	|_ checkpoint ⇢ HTTP POST "/agencies/3c91bc7d-2581-46b8-ac39-9b46cdbb7bc4/catalogs" [ExceptionHandlingWebHandler]
      Stack trace:
      		at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92)
      		at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814)
      		at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797)
      		at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3286)
      		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88)
      		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185)
      		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251)
      		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165)
      		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87)
      		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87)
      		at com.mongodb.reactivestreams.client.internal.AbstractSubscription.onError(AbstractSubscription.java:142)
      		at com.mongodb.reactivestreams.client.internal.SingleResultCallbackSubscription.lambda$requestInitialData$0(SingleResultCallbackSubscription.java:41)
      		at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:188)
      		at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:184)
      		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      		at com.mongodb.internal.async.client.OperationExecutorImpl$1$1$1.onResult(OperationExecutorImpl.java:92)
      		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      		at com.mongodb.internal.operation.FindOperation$3.onResult(FindOperation.java:724)
      		at com.mongodb.internal.operation.OperationHelper$ReferenceCountedReleasingWrappedCallback.onResult(OperationHelper.java:411)
      		at com.mongodb.internal.operation.CommandOperationHelper$10.checkRetryableException(CommandOperationHelper.java:493)
      		at com.mongodb.internal.operation.CommandOperationHelper$10.onResult(CommandOperationHelper.java:478)
      		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      		at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor$2.onResult(DefaultServer.java:248)
      		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      		at com.mongodb.internal.connection.CommandProtocolImpl$1.onResult(CommandProtocolImpl.java:82)
      		at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:517)
      		at com.mongodb.internal.connection.UsageTrackingInternalConnection$2.onResult(UsageTrackingInternalConnection.java:111)
      		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      		at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:400)
      		at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375)
      		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:676)
      		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:643)
      		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
      		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
      		at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
      		at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
      		at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
      		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:633)
      		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618)
      		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
      		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
      		at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
      		at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
      		at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
      		at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375)
      		at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364)
      		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      		at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467)
      		at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464)
      		at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187)
      		at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181)
      		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
      		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
      		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
      		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
      		at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
      		at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
      		at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
      		at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
      		at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
      		at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
      		at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414)
      		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
      		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
      		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898)
      		at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
      		at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
      		at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
      		at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
      		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
      		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
      		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
      		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      		at java.lang.Thread.run(Thread.java:748)
      Caused by: com.mongodb.MongoInternalException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request
      	at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:65)
      	at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:43)
      	at com.mongodb.internal.connection.ResponseBuffers.getResponseDocument(ResponseBuffers.java:47)
      	at com.mongodb.internal.connection.LoggingCommandEventSender.sendSucceededEvent(LoggingCommandEventSender.java:141)
      	at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:395)
      	at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:676)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:643)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
      	at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
      	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
      	at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:633)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
      	at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
      	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
      	at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
      	at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375)
      	at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364)
      	at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
      	at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467)
      	at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464)
      	at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187)
      	at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181)
      	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
      	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
      	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
      	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
      	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
      	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
      	at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
      	at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
      	at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
      	at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
      	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414)
      	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
      	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
      	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
      	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
      	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.lang.Thread.run(Thread.java:748)
      

      I investigated the related source code, I guess that when the first error happened. It seems that mongo has replied an unknown message to the driver. When the driver decode the message follow the wire protocol, it found that the message length is too big(reading the first 16 bytes from the stream) and throw an exception then ignored the rest bytes. The following requests will keep doing the same until meeting the first correct reply message but the responseTo in the reply message will be smaller than the current request id.

      I can't reproduce this on my laptop. I'm wondering why the driver needs to check the message size or the responseTo is matching the request id. Is there a concern regarding this? Why not check if the reply message is matching the wire protocol?
      I guess this issue is more likely a network issue or protocol issue. Could you advise me on how to find the root cause?

        1. service_2.log
          14.02 MB
        2. service_1.log
          2.05 MB
        3. POS-vin11c9a-eth0-2021_1020_0622_27.pcap.zip
          16.27 MB
        4. image-2021-11-03-14-46-40-340.png
          image-2021-11-03-14-46-40-340.png
          903 kB

            Assignee:
            valentin.kovalenko@mongodb.com Valentin Kavalenka
            Reporter:
            gloojiang86@gmail.com 运军 蒋
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: