Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-72651

$match filter is erroneously pushed past $project into COLLSCAN

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 6.0.3
    • Component/s: None
    • None
    • Fully Compatible
    • ALL
    • Hide

      On v6.0:

      const c = db.bf27387;
      c.drop();
      c.save({_id: 0, array: ["abc"]});
      
      const res = c.aggregate(
          [{$project: {"obj.obj.array": 1}},
          {$match: {$expr: {$getField: {$literal: "array"}}}}]);
      
      // The bug caused the query above to return { "_id" : 0 } instead of no documents.
      assert.eq(0, res.toArray().length);
      

      Mongo v6.0 will erroneously generate the following plan for the aggregation:

                      "winningPlan" : {
                              "stage" : "PROJECTION_DEFAULT",
                              "transformBy" : {
                                      "_id" : true,
                                      "obj" : {
                                              "obj" : {
                                                      "array" : true
                                              }
                                      }
                              },
                              "inputStage" : {
                                      "stage" : "COLLSCAN",
                                      "filter" : {
                                              "$expr" : {
                                                      "$getField" : {
                                                              "field" : {
                                                                      "$const" : "array"
                                                              },
                                                              "input" : "$$CURRENT"
                                                      }
                                              }
                                      },
                                      "direction" : "forward"
                              }
                      },
      

      The filter is merged into the collscan, putting the filter before the projection. This is problematic because the semantics of the filter are different when operating on the full (unprojected) document.

      The correct plan, produced by v6.1, is:

      {
      	"explainVersion" : "1",
      	"stages" : [
      		{
      			"$cursor" : {
      				"queryPlanner" : {
      					"namespace" : "test.bf27387",
      					"indexFilterSet" : false,
      					"parsedQuery" : {
      						
      					},
      					"queryHash" : "92148A8F",
      					"planCacheKey" : "92148A8F",
      					"maxIndexedOrSolutionsReached" : false,
      					"maxIndexedAndSolutionsReached" : false,
      					"maxScansToExplodeReached" : false,
      					"winningPlan" : {
      						"stage" : "PROJECTION_DEFAULT",
      						"transformBy" : {
      							"_id" : true,
      							"obj" : {
      								"obj" : {
      									"array" : true
      								}
      							}
      						},
      						"inputStage" : {
      							"stage" : "COLLSCAN",
      							"direction" : "forward"
      						}
      					},
      					"rejectedPlans" : [ ]
      				}
      			}
      		},
      		{
      			"$match" : {
      				"$expr" : {
      					"$getField" : {
      						"field" : {
      							"$const" : "array"
      						},
      						"input" : "$$CURRENT"
      					}
      				}
      			}
      		}
      	],...} 
      Show
      On v6.0: const c = db.bf27387; c.drop(); c.save({_id: 0, array: [ "abc" ]}); const res = c.aggregate( [{$project: { "obj.obj.array" : 1}}, {$match: {$expr: {$getField: {$literal: "array" }}}}]); // The bug caused the query above to return { "_id" : 0 } instead of no documents. assert.eq(0, res.toArray().length); Mongo v6.0 will erroneously generate the following plan for the aggregation: "winningPlan" : { "stage" : "PROJECTION_DEFAULT" , "transformBy" : { "_id" : true , "obj" : { "obj" : { "array" : true } } }, "inputStage" : { "stage" : "COLLSCAN" , "filter" : { "$expr" : { "$getField" : { "field" : { "$ const " : "array" }, "input" : "$$CURRENT" } } }, "direction" : "forward" } }, The filter is merged into the collscan, putting the filter before the projection. This is problematic because the semantics of the filter are different when operating on the full (unprojected) document. The correct plan, produced by v6.1, is: { "explainVersion" : "1" , "stages" : [ { "$cursor" : { "queryPlanner" : { "namespace" : "test.bf27387" , "indexFilterSet" : false , "parsedQuery" : { }, "queryHash" : "92148A8F" , "planCacheKey" : "92148A8F" , "maxIndexedOrSolutionsReached" : false , "maxIndexedAndSolutionsReached" : false , "maxScansToExplodeReached" : false , "winningPlan" : { "stage" : "PROJECTION_DEFAULT" , "transformBy" : { "_id" : true , "obj" : { "obj" : { "array" : true } } }, "inputStage" : { "stage" : "COLLSCAN" , "direction" : "forward" } }, "rejectedPlans" : [ ] } } }, { "$match" : { "$expr" : { "$getField" : { "field" : { "$ const " : "array" }, "input" : "$$CURRENT" } } } } ],...}
    • QE 2023-01-23
    • 120

      A $match filter can be pushed past a $project into a COLLSCAN only if the $project has no impact on the filter results.  For this to be true, the $match filters' paths must be a subset of the $project paths.  This check seems to be broken in v6.0.

      This ticket came out of rui.liu@mongodb.com's investigation of BF-27387.

      This issue is only seen in v6.0 because it was fixed for 6.1.0-rc0 by a change to isIndependentOf in SERVER-67416.  To fix this bug in v6.0 we can try to backport SERVER-67416.  This ticket may also track work to add a js test for this bug.

      v5.0 does not exhibit this bug.

            Assignee:
            steve.tarzia@mongodb.com Steve Tarzia
            Reporter:
            steve.tarzia@mongodb.com Steve Tarzia
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: