Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-39166

$graphLookup should force a pipeline to split in sharded cluster

    • Fully Compatible
    • ALL
    • v4.0, v3.6, v3.4
    • Query 2019-02-11

      The $graphLookup stage does not inherit from NeedsMergerDocumentSource, which means that it is eligible to execute in parallel on all shards. This doesn't work because it will only ever do the searching locally. Instead, it should force itself to split the pipeline and execute in the merging half of the pipeline so that it can always run on the primary shard for the database.

      The $graphLookup StageConstraints are correct in saying that it must run on the primary shard, but this would only apply if some other stage before $graphLookup forced the pipeline to split.

      For example, see the following running against a mongos:

      mongos> db.adminCommand({enableSharding: "test"})
      ...
      mongos> db.adminCommand({shardCollection: "test.foo", key: {_id: "hashed"}})
      ...
      mongos> db.foo.insert([{}, {}, {}, {}])
      ...
      mongos> db.bar.insert({_id: 1, x: 1})
      WriteResult({ "nInserted" : 1 })
      mongos> db.foo.aggregate([{$project: {l_x: {$literal: 1}}}, {$graphLookup: {from: "bar", startWith: "$l_x", connectFromField: "x", connectToField: "_id", as: "res"}}])
      { "_id" : ObjectId("5c48f159a3c842122a8dec28"), "l_x" : 1, "res" : [ ] }  // Uh oh, these should have the same results as below.
      { "_id" : ObjectId("5c48f159a3c842122a8dec29"), "l_x" : 1, "res" : [ ] }
      { "_id" : ObjectId("5c48f159a3c842122a8dec26"), "l_x" : 1, "res" : [ { "_id" : 1, "x" : 1 } ] }
      { "_id" : ObjectId("5c48f159a3c842122a8dec27"), "l_x" : 1, "res" : [ { "_id" : 1, "x" : 1 } ] }
      

            Assignee:
            martin.neupauer@mongodb.com Martin Neupauer
            Reporter:
            charlie.swanson@mongodb.com Charlie Swanson
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: