Loading...

Type: Bug
Resolution: Won't Do
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
- external-user

Case:

Documentation Changes:
Not Needed

Hi guys,

We are facing an issue that happened several times a day. Our java driver version is 4.0.5 and Mongo DB version is 4.0.9. We have 2 routers and several nodes. And we are using reactive stream. This issue only happens in our production env. It usually first occurred while MongodbHealthIndicator checking the connections with mongo db. Please see the stacktrace as below:

org.springframework.dao.InvalidDataAccessResourceUsageException: The reply message length 1013281644 is greater than the maximum message length 48000000; nested exception is com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000org.springframework.dao.InvalidDataAccessResourceUsageException: The reply message length 1013281644 is greater than the maximum message length 48000000; nested exception is com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000 at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s): |_ checkpoint ⇢ com.active.platform.modules.pos.service.config.ContextFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain] |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain] |_ checkpoint ⇢ HTTP GET "/agencies/f953b9de-2260-4833-bf32-d7be2a9ae945/catalogs" [ExceptionHandlingWebHandler]Stack trace: at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92) at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814) at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797) at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3286) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185) at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87) at com.mongodb.reactivestreams.client.internal.AbstractSubscription.onError(AbstractSubscription.java:142) at com.mongodb.reactivestreams.client.internal.SingleResultCallbackSubscription.lambda$requestInitialData$0(SingleResultCallbackSubscription.java:41) at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:188) at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:184) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.async.client.OperationExecutorImpl$1$1$1.onResult(OperationExecutorImpl.java:92) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.operation.FindOperation$3.onResult(FindOperation.java:724) at com.mongodb.internal.operation.OperationHelper$ReferenceCountedReleasingWrappedCallback.onResult(OperationHelper.java:411) at com.mongodb.internal.operation.CommandOperationHelper$10.checkRetryableException(CommandOperationHelper.java:493) at com.mongodb.internal.operation.CommandOperationHelper$10.onResult(CommandOperationHelper.java:478) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor$2.onResult(DefaultServer.java:248) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.CommandProtocolImpl$1.onResult(CommandProtocolImpl.java:82) at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:517) at com.mongodb.internal.connection.UsageTrackingInternalConnection$2.onResult(UsageTrackingInternalConnection.java:111) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:380) at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:635) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510) at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232) at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510) at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)Caused by: com.mongodb.MongoInternalException: The reply message length 1013281644 is greater than the maximum message length 48000000 at com.mongodb.internal.connection.MessageHeader.<init>(MessageHeader.java:41) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:632) at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513) at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510) at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232) at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510) at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375) at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364) at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467) at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187) at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48) at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717) at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272) at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352) at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354) at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898) at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372) at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

After about 20+ times of this error(the reply message length is different every time). The error will become


The responseTo (2342443) in the response does not match the requestId (2342522) in the request

*The responseTo is always smaller than the request id *

The full stack trace is

org.springframework.dao.InvalidDataAccessResourceUsageException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request; nested exception is com.mongodb.MongoInternalException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request
	at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ com.active.platform.modules.pos.service.config.ContextFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.authentication.AuthenticationWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ HTTP POST "/agencies/3c91bc7d-2581-46b8-ac39-9b46cdbb7bc4/catalogs" [ExceptionHandlingWebHandler]
Stack trace:
		at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:92)
		at org.springframework.data.mongodb.core.ReactiveMongoTemplate.potentiallyConvertRuntimeException(ReactiveMongoTemplate.java:2814)
		at org.springframework.data.mongodb.core.ReactiveMongoTemplate.lambda$translateException$90(ReactiveMongoTemplate.java:2797)
		at reactor.core.publisher.Mono.lambda$onErrorMap$30(Mono.java:3286)
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:185)
		at reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:251)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87)
		at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:87)
		at com.mongodb.reactivestreams.client.internal.AbstractSubscription.onError(AbstractSubscription.java:142)
		at com.mongodb.reactivestreams.client.internal.SingleResultCallbackSubscription.lambda$requestInitialData$0(SingleResultCallbackSubscription.java:41)
		at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:188)
		at com.mongodb.internal.async.client.AsyncFindIterableImpl$1.onResult(AsyncFindIterableImpl.java:184)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.async.client.OperationExecutorImpl$1$1$1.onResult(OperationExecutorImpl.java:92)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.operation.FindOperation$3.onResult(FindOperation.java:724)
		at com.mongodb.internal.operation.OperationHelper$ReferenceCountedReleasingWrappedCallback.onResult(OperationHelper.java:411)
		at com.mongodb.internal.operation.CommandOperationHelper$10.checkRetryableException(CommandOperationHelper.java:493)
		at com.mongodb.internal.operation.CommandOperationHelper$10.onResult(CommandOperationHelper.java:478)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor$2.onResult(DefaultServer.java:248)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.CommandProtocolImpl$1.onResult(CommandProtocolImpl.java:82)
		at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection$2.onResult(DefaultConnectionPool.java:517)
		at com.mongodb.internal.connection.UsageTrackingInternalConnection$2.onResult(UsageTrackingInternalConnection.java:111)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:400)
		at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:676)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:643)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
		at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
		at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
		at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:633)
		at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
		at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
		at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
		at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
		at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
		at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375)
		at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364)
		at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
		at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467)
		at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464)
		at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187)
		at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
		at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
		at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
		at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
		at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
		at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
		at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
		at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
		at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
		at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
		at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
		at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
		at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
		at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoInternalException: The responseTo (2342443) in the response does not match the requestId (2342522) in the request
	at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:65)
	at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:43)
	at com.mongodb.internal.connection.ResponseBuffers.getResponseDocument(ResponseBuffers.java:47)
	at com.mongodb.internal.connection.LoggingCommandEventSender.sendSucceededEvent(LoggingCommandEventSender.java:141)
	at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:395)
	at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:375)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:676)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:643)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
	at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
	at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:633)
	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:618)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:513)
	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:510)
	at com.mongodb.connection.netty.NettyStream.readAsync(NettyStream.java:232)
	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:510)
	at com.mongodb.internal.connection.InternalStreamConnection.access$1000(InternalStreamConnection.java:75)
	at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:375)
	at com.mongodb.internal.connection.InternalStreamConnection$2.onResult(InternalStreamConnection.java:364)
	at com.mongodb.internal.async.ErrorHandlingResultCallback.onResult(ErrorHandlingResultCallback.java:48)
	at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:467)
	at com.mongodb.internal.connection.InternalStreamConnection$3.completed(InternalStreamConnection.java:464)
	at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:187)
	at com.mongodb.connection.netty.NettyStream$2.operationComplete(NettyStream.java:181)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
	at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
	at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
	at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:414)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:898)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
	at io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1071)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

I investigated the related source code, I guess that when the first error happened. It seems that mongo has replied an unknown message to the driver. When the driver decode the message follow the wire protocol, it found that the message length is too big(reading the first 16 bytes from the stream) and throw an exception then ignored the rest bytes. The following requests will keep doing the same until meeting the first correct reply message but the responseTo in the reply message will be smaller than the current request id.

I can't reproduce this on my laptop. I'm wondering why the driver needs to check the message size or the responseTo is matching the request id. Is there a concern regarding this? Why not check if the reply message is matching the wire protocol?
I guess this issue is more likely a network issue or protocol issue. Could you advise me on how to find the root cause?

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

error_0.log
9 kB
Aug 20 2021 08:01:22 AM UTC
error_1.log
31 kB
Aug 20 2021 08:01:23 AM UTC
error_2.log
42 kB
Aug 20 2021 08:01:23 AM UTC
first_error.log
15 kB
Aug 31 2021 11:49:06 AM UTC
mongodb-driver-core-4.0.5-dddab108d.jar
1.58 MB
Aug 27 2021 11:21:07 PM UTC
mongodb-driver-core-4.0.5-JAVA-4210.jar
1.58 MB
Aug 06 2021 10:43:18 PM UTC

is cloned by

JAVA-4387 MongoDb returns message length larger than the max message length

Closed

related to

JAVA-4122 Getting Exceptions when CPU starving the driver

Closed

JAVA-3061 Re-enable check for requestId matching responseTo in reply header

Closed

split to

JAVA-4294 InternalStreamConnection must close its stream if Stream.readAsync completes abruptly

Closed

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates