Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-3700

Java driver Out Of Memory, with aggregate to merge

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.0.2
    • Component/s: Command Operations
    • None

      Hi, today we are sustaining an OOM (Out Of Memory) problem when using the driver in version 3.11, I performed an upgrade for testing in version 4.0.2 and I will have the same behavior as OOM.

      Whenever we perform an aggregation operation with merge into, according to the query below we have as a return the entire destination collection, that is, we do not have the behavior as exemplified in this link, where it is compared to an SQL statement
      https://docs.mongodb.com/manual/reference/operator/aggregation/merge/index.html#comparison-with-out

      This is the query that when executed by the following code snippet returns

       

      {"$match" : { 
       status:
      { $in:["SUCESSO","REENVIADO"]}
      }},
       {$limit : 1 },
       { $merge :
      { into : "my-collection-backup" }
      }
      

       

       

      MongoClient client = MongoClients.create("mongodb://localhost:27017");
       MongoDatabase database = client.getDatabase("my-collection");
       MongoCollection<Document> model = database.getCollection(collectionName);
       String query = " ... my query ... ";
       bsonArray = BsonArray.parse(query);
       List<BsonDocument> bsonList = convertBsonArrayToBsonDocumentList(bsonArray);
       AggregateIterable<Document> aggregateResult = model.aggregate(bsonList);
      ...
       //Exemplified here only the conversion method Bson Array To Bson
       private List<BsonDocument> convertBsonArrayToBsonDocumentList(BsonArray bsonArray) {
       Iterator<BsonValue> it = bsonArray.iterator();
       List<BsonDocument> bsonDocuments = new ArrayList<>();
       while (it.hasNext())
      { BsonValue next = it.next(); BsonDocument bsonDocument = next.asDocument(); bsonDocuments.add(bsonDocument); }
      return bsonDocuments;
       }
      

       

      I see that the correct execution of an aggregate with merge should be the same as the execution via mongoshell, which from within the bank, empty or code return, via command line the result is omitted as shown below.

      MongoDB server version: 4.2.0

      ...
      > use my-db;
       switched to db my-db
       > db.getCollection("my-collection").aggregate([{"$match" : { status:{$in:["SUCESSO","REENVIADO"]}}}, {$limit : 1 },{ $merge :
      { into : "my-collection-backup" }
      }]);
       >
       >
      

       

      Therefore, the fact that we suffer from OOM is due to the improper return of the entire destination collection.

      More information: my collections are large, more than 35 thousand records, this is due to the particularity of our customers, and the execution of these queries are in Pod's within the kubernetes.

      I emphasize that the theme here is not the lack of resources in our containers, but rather a harmful and incorrect behavior that differs from that performed natively in the Mongo.

       

            Assignee:
            jeff.yemin@mongodb.com Jeffrey Yemin
            Reporter:
            diego.maia@digibee.com.br Diego Maia
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: