Summary

The aggregation query is executed through the Java Web application server.(Java WAS)

When the $merge stage is executed at the end of the mongo aggregation, sometimes the “BSONObjectTooLarge” error occurs.

The “BSONObjectTooLarge” occurs in the the various size of the result set, which size is from 20k docs to 300k docs.

a hours after the error occurs, if I run the errored query through Java WAS, it again shows the BSONObjectTooLarge error, but right after the error shows, if I run the query again, the query executed well with no errors.

In addition, when I run that query through the mongo client, such as mongo shell(mongo) or ‘NoSQLBooster for MongoDB’, it does NOT shows any error.

I uses the 5 or more different aggregation queries to compute the big size of documents, the one of the common part is the $merge stage is at the end of the aggregation queries.

In addition, big size result set may causes the error but not always shows the error.

com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:198) com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:517) com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:503) com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:826) com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:790) com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:650) com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:647) com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:250) com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:233) java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127) java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:282) java.base/sun.nio.ch.WindowsAsynchronousSocketChannelImpl$ReadTask.completed(WindowsAsynchronousSocketChannelImpl.java:581) java.base/sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:387) java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base/java.lang.Thread.run(Thread.java:829), 

{{}}

messag:

{{}}

Command failed with error 10334 (BSONObjectTooLarge): 'PlanExecutor error during aggregation :: caused by :: BSONObj size: 29719999 (0x16C57DBF) is invalid. Size must be between 0 and 16793600(16MB) First element: update: "indicator_by_flow_pt10m"' on server myserver.com:27617. the full response is {\"ok\": 0.0, \"errormsg\":\"PlanExecutor error during aggregation :: caused by :: BSONObj size: 29719999 (0x16C57DBF) is invalid. Size must be between 0 and 16793600(16MB) First element: update: \\\"indicator_by_flow_pt10m\\\"\", \"code\": 10334, \"codName: \"BSONObjectTooLarge\", \"$clusterTime\": {\"clusterTime\", {\"$timestamp\": {\"t\": 1659918034, \"i\": 2031}}, \"signature\": {\"hash\": {\"$binary\": {\"base64\": \"JqcdMT7Jg3PTSajRBz+J082pjPs=\", \"subType\": \"00\"}}, \"keyId\": 7097070755241787396}}, \"operationTime\": {\"$timestamp\": {\"t\": 1659918034, \"i\": 1990}}}

{{}}

Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

• mongo version : v5.5, replica set
• driver version: org.mongodb:mongodb-driver-reactivestreams:4.5.1

How to Reproduce

the below is the query i used, but the there are several different aggregation queries like the below, the aggregation queries are different but the common thing is the $merge stage at the end of the queries.
// mydata collection document{
  "_id" : ObjectId("6188c90f6c7455e39997e209"),
  "sessiontime" : ISODate("2021-11-08T15:40:00.000+09:00"),
  "appid" : "fdjksjldglakdgladglkla",
  "appver" : "3.5.5",
  "devicdid" : "jkdggsjal;hjglkjdks;ajglkdgjsa;",
  "devver" : "9",
  "devno" : "my-device-name1",
  "sessiontime2" : ISODate("2021-11-09T00:40:00.000+09:00"),
  "stzd" : ISODate("2021-11-08T09:00:00.000+09:00"),
  "sstime" : ISODate("2021-11-08T15:40:00.000+09:00"),
  "devicecount" : 1
}
 
db.mydata.aggregate([
  {
    "$match": {
      "sessiontime":

    }
  },
  {
    "$group": {
      "_id":

    }
  },
  {
    "$replaceWith": {
      "$mergeObjects": [
       

      ]
    }
  },
  {
    "$lookup": {
      "from": "mycoll1",
      "let":

,
      "pipeline": [
        {
          "$match": {
            "$expr": {
              "$and": [
               

,
                {
                  "$eq": [
                    "$appid",
                    "$$appid"
                  ]
                },
                {
                  "$eq": [
                    "$appver",
                    "$$appver"
                  ]
                },
                {
                  "$eq": [
                    "$devicdid",
                    "$$devicdid"
                  ]
                },
                {
                  "$eq": [
                    "$devver",
                    "$$devver"
                  ]
                },
                {
                  "$eq": [
                    "$devno",
                    "$$devno"
                  ]
                }
              ]
            }
          }
        }
      ],
      "as": "data"
    }
  },
  {
    "$unwind": "$data"
  },
  {
    "$group": {
      "_id":

,
      "devicecount": {
        "$sum": "$data.devicecount"
      }
    }
  },
  {
    "$replaceWith": {
      "$mergeObjects": [
       

      ]
    }
  },
  {
    "$merge":

  }
])
 

Additional Background