Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-53074

[SBE] Add support for $map aggregation pipeline operator

    • Type: Icon: Task Task
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Querying

      When I tried to run the "word_count.js" benchmark (from https://github.com/10gen/workloads) and with SBE mode enabled, mongod crashed due to an invariant failure at line 1539 at "src/mongo/db/query/sbe_stage_builder_expression.cpp": 

        invariant(it != _context->environment.end());

       

      I attached gdb to mongod (debug build) and then I ran the "work_count.js" benchmark with SBE mode enabled to reproduce the crash, and here was the stack trace I got: 

      #0 raise (sig=<optimized out>) at ../sysdeps/unix/sysv/linux/raise.c:51
      ..
      #3 0x0000564379faf11d in mongo::invariantWithLocation<bool> (testOK=@0x7fca449670e8: false,
       expr=0x56437e6eedc8 "it != _context->environment.end()",
       file=0x56437e6ee668 "src/mongo/db/query/sbe_stage_builder_expression.cpp", line=1539)
       at src/mongo/util/invariant.h:69
      #4 0x000056437b83ecea in mongo::stage_builder::(anonymous namespace)::ExpressionPostVisitor::visit (
       this=0x7fca44967370, expr=0x7fca15da4f20)
       at src/mongo/db/query/sbe_stage_builder_expression.cpp:1539
      #5 0x000056437c328da0 in mongo::ExpressionFieldPath::acceptVisitor (this=0x7fca15da4f20,
       visitor=0x7fca44967370) at src/mongo/db/pipeline/expression.h:1441
      #6 0x000056437b84a45e in mongo::stage_builder::(anonymous namespace)::ExpressionWalker::postVisit (
       this=0x7fca449673a0, expr=0x7fca15da4f20)
       at src/mongo/db/query/sbe_stage_builder_expression.cpp:2728
      #7 0x000056437b84bd84 in mongo::expression_walker::walk<mongo::stage_builder::(anonymous namespace)::ExpressionWalker> (walker=0x7fca449673a0, expression=0x7fca15da4f20)
       at src/mongo/db/pipeline/expression_walker.h:62
      #8 0x000056437b84bd63 in mongo::expression_walker::walk<mongo::stage_builder::(anonymous namespace)::ExpressionWalker> (walker=0x7fca449673a0, expression=0x7fca3d11dd80)
       at src/mongo/db/pipeline/expression_walker.h:59
      #9 0x000056437b84bd63 in mongo::expression_walker::walk<mongo::stage_builder::(anonymous namespace)::ExpressionWalker> (walker=0x7fca449673a0, expression=0x7fca15da5520)
       at src/mongo/db/pipeline/expression_walker.h:59
      #10 0x000056437b84aaf6 in mongo::stage_builder::generateExpression (opCtx=0x7fca2586f720,
       expr=0x7fca15da5520, stage=..., slotIdGenerator=0x7fc9d50b6f48, frameIdGenerator=0x7fc9d50b6f58,
       rootSlot=2, env=0x7fc9fe7a9120, planNodeId=2, relevantSlots=0x7fca44967848)
       at src/mongo/db/query/sbe_stage_builder_expression.cpp:2797
      #11 0x000056437b87d8f3 in mongo::stage_builder::(anonymous namespace)::ProjectionTraversalPostVisitor::visit (this=0x7fca44967750, node=0x7fc9cc5010e0)
       at src/mongo/db/query/sbe_stage_builder_projection.cpp:243
      #12 0x000056437c0feb18 in mongo::projection_ast::ExpressionASTNode::acceptVisitor (this=0x7fc9cc5010e0,
       visitor=0x7fca44967750) at src/mongo/db/query/projection_ast.h:293
      #13 0x000056437b87f298 in mongo::stage_builder::(anonymous namespace)::ProjectionTraversalWalker::postVisit (this=0x7fca44967760, node=0x7fc9cc5010e0) at src/mongo/db/query/sbe_stage_builder_projection.cpp:382
      #14 0x000056437b87fb20 in mongo::tree_walker::walk<true, mongo::projection_ast::ASTNode, mongo::stage_builder::(anonymous namespace)::ProjectionTraversalWalker> (node=0x7fc9cc5010e0, walker=0x7fca44967760)
       at src/mongo/db/query/tree_walker.h:70
      #15 0x000056437b87faff in mongo::tree_walker::walk<true, mongo::projection_ast::ASTNode, mongo::stage_builder::(anonymous namespace)::ProjectionTraversalWalker> (node=0x7fc9eb38d358, walker=0x7fca44967760)
       at src/mongo/db/query/tree_walker.h:67
      #16 0x000056437b87f4c7 in mongo::stage_builder::generateProjection (opCtx=0x7fca2586f720,
       projection=0x7fc9eb38d358, stage=..., slotIdGenerator=0x7fc9d50b6f48,
       frameIdGenerator=0x7fc9d50b6f58, inputVar=2, env=0x7fc9fe7a9120, planNodeId=2)
       at src/mongo/db/query/sbe_stage_builder_projection.cpp:413
      #17 0x000056437b8138ef in mongo::stage_builder::SlotBasedStageBuilder::buildProjectionDefault (
       this=0x7fc9d50b6f20, root=0x7fc9eb38d2e0, reqs=...) at src/mongo/db/query/sbe_stage_builder.cpp:708
      ..
      #25 0x000056437b816af6 in mongo::stage_builder::SlotBasedStageBuilder::build (this=0x7fc9d50b6f20,
       root=0x7fc9eb38d2e0, reqs=...) at src/mongo/db/query/sbe_stage_builder.cpp:1052
      #26 0x000056437b80ea57 in mongo::stage_builder::SlotBasedStageBuilder::build (this=0x7fc9d50b6f20,
       root=0x7fc9eb38d2e0) at src/mongo/db/query/sbe_stage_builder.cpp:159
      #27 0x000056437b88a4c3 in mongo::stage_builder::buildSlotBasedExecutableTree (opCtx=0x7fca2586f720,
       collection=..., cq=..., solution=..., yieldPolicy=0x7fca2445f160,
       needsTrialRunProgressTracker=false) at src/mongo/db/query/stage_builder_util.cpp:74
      #28 0x000056437b7b8421 in mongo::(anonymous namespace)::SlotBasedPrepareExecutionHelper::buildExecutableTree (this=0x7fca449688f0, solution=..., needsTrialRunProgressTracker=false)
       at src/mongo/db/query/get_executor.cpp:956
      #29 0x000056437b7b7d86 in mongo::(anonymous namespace)::SlotBasedPrepareExecutionHelper::buildExecutableTree (this=0x7fca449688f0, solution=...) at src/mongo/db/query/get_executor.cpp:904
      #30 0x000056437b7c4e87 in mongo::(anonymous namespace)::PrepareExecutionHelper<std::pair<std::unique_ptr<mongo::sbe::PlanStage, std::default_delete<mongo::sbe::PlanStage> >, mongo::stage_builder::PlanStageData>, mongo::(anonymous namespace)::SlotBasedPrepareExecutionResult>::prepare (this=0x7fca449688f0)
       at src/mongo/db/query/get_executor.cpp:672
      #31 0x000056437b7b8ebb in mongo::(anonymous namespace)::getSlotBasedExecutor (opCtx=0x7fca2586f720,
       collection=0x7fca44969938, cq=...,
       requestedYieldPolicy=mongo::PlanYieldPolicy::YieldPolicy::YIELD_AUTO, plannerOptions=0)
       at src/mongo/db/query/get_executor.cpp:1068
      #32 0x000056437b7b9660 in mongo::getExecutor (opCtx=0x7fca2586f720, collection=0x7fca44969938,
       canonicalQuery=..., yieldPolicy=mongo::PlanYieldPolicy::YieldPolicy::YIELD_AUTO, plannerOptions=0)
       at src/mongo/db/query/get_executor.cpp:1116
      ..
      #36 0x000056437b791def in mongo::PipelineD::prepareExecutor (expCtx=..., collection=..., nss=...,
       pipeline=0x7fc9f55427c0, sortStage=..., rewrittenGroupStage=..., unavailableMetadata=...,
       queryObj=..., skipThenLimit=..., aggRequest=0x7fc96b543058, matcherFeatures=@0x56437e6a8198: 57,
       hasNoRequirements=0x7fca44969042) at src/mongo/db/pipeline/pipeline_d.cpp:788
      ..

      I came up with a small repro for this crash - see the "Steps To Reproduce" section.

      I also tried out running a modified version of the small repro where I replaced "$$word" with 1, like so:

      db.adminCommand({setParameter:1, internalQueryEnableSlotBasedExecutionEngine:true});
      db.foo.insert({ "t" : "blah" });
      db.foo.aggregate([{$project: {emits: {$map: {input: {$split: ["$t", " "]}, as: "word", in: {k: 1, v: 1}}}}}]);

      When I ran the commands above, mongod did not crash, but I got the following error message: 

      "Expression is not supported in SBE: $map"

      Thus it looks like even if the crash were fixed, the "word_count.js" benchmark would still fail because SBE doesn't support the $map aggregation pipeline operator yet.

      The goal of this task is: (1) to implement SBE support for the $map aggregation pipeline operator; and (2) to fix the "invariant(it != _context->environment.end())" crash.

            Assignee:
            andrew.paroski@mongodb.com Drew Paroski
            Reporter:
            andrew.paroski@mongodb.com Drew Paroski
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: