-
Type: Bug
-
Resolution: Fixed
-
Priority: Blocker - P1
-
Affects Version/s: None
-
Component/s: None
-
None
-
Fully Compatible
-
QE 2022-04-04, QE 2022-04-18
$lookup and $group stages may produce an object which exceeds BSON size limit. In case of $lookup, this happens when sufficient number of documents from the foreign collection is pushed into the result array. In case of $group, $push and $addToSet accumulators can create large arrays as well.
Instead of returning BSON, these stages return Document, which does not have a size limitation. Current implementation of $lookup and $group in SBE use mkbson stage, which produces BSON. This means that some queries will successfully produce results in classic engine, but fail with "BSON size limit exceeded" exceptions in SBE.
Here are examples of such queries:
$lookup
for(let i = 0; i < 8; ++i) { db.b.insert({foreign: 1, bigField: 'x'.repeat(10 * 1024 * 1024)}) } db.a.insert({local: 1}) db.a.aggregate([{$lookup: {from: 'b', localField: 'local', foreignField: 'foreign', as: 'out'}}, {$project: {foo: {$add: ["$local", 2]}}}])
$group
for(let i = 0; i < 8; ++i) { db.b.insert({foreign: 1, bigField: 'x'.repeat(10 * 1024 * 1024)}) } db.b.aggregate([{$group: {_id: "$foreign", out: {$push: "$bigField"}}}, {$project: {a: {$add: [1, "$foreign"]}}}])
To mitigate this issue, we should:
- Make SBE plans for $lookup and $group produce SBE Object instead of BSON
- Implement translation from SBE Object to Document in PlanExecutor. This is to make sure that even if only $lookup or $group are pushed into SBE and the rest of the plan is executed by classic engine, we can still return larger-than-BSON-limit objects from SBE.
Separate ticket SERVER-65265 is created to ensure that arrays produced by $lookup and $group do not exceed maximum size limit specified by corresponding query knobs. This ticket is only about lifting BSON size limitation.
- is duplicated by
-
SERVER-64877 $addToSet and $push fail in pushed down $group if accumulated values exceed 64MB
- Closed
- is related to
-
SERVER-65265 Implement memory tracking for accumulators in SBE
- Closed