Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-58728

mqlrun does not support spilling to disk for $setWindowFields

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • ALL
    • Hide

      Build or download an mqlrun binary (version 5.0.0). Prepare a large (at least 1 million docs) BSON file to use as an input (I can provide the one I used to reproduce this issue).

      Start mqlrun from the command line (where ./tmp is a directory for spilling-to-disk): 

      ./mqlrun -f -j -t ./tmp -e '[{$_internalSetWindowFields: {output: {}}}]' big_1000000.bson > output.json

       This always causes mqlrun to crash with a SIGABRT:

      [1] 78171 abort ./mqlrun -f -j -t ./tmp -e big_1000000.bson > output.json
      

      The mqlrun process also writes the following to the tail end of the JSON output file:

      {"t":{"$date":"2021-07-20T17:33:45.472Z"},"s":"F", "c":"-", "id":23079, "ctx":"thread1","msg":"Invariant failure","attr":{"expr":"Hit a MONGO_UNREACHABLE!","file":"src/mongo/db/pipeline/process_interface/stub_mongo_process_interface.h","line":271}}
      {"t":{"$date":"2021-07-20T17:33:45.473Z"},"s":"F", "c":"-", "id":23080, "ctx":"thread1","msg":"\n\n***aborting after invariant() failure\n\n"}
      

      However, when I run mqlrun with a $limit stage preceding the $_internalSetWindowFields stage:

      ./mqlrun -f -j -t ./tmp -e '[{$limit: 100000}, {$_internalSetWindowFields: {output: {}}}]' big_1000000.bson > output.json 

      then mqlrun runs successfully for limits as high as 100,000. I've tried this with several large files and there's always some point after which mqlrun consistently fails with this same error. I've been unable to reproduce this with $sort or $group. I imported the contents of this file into MongoDB using mongorestore and am able to run a $setWindowFields aggregations without any errors.

      Show
      Build or download an mqlrun binary (version 5.0.0). Prepare a large (at least 1 million docs) BSON file to use as an input (I can provide the one I used to reproduce this issue). Start  mqlrun  from the command line (where  ./tmp  is a directory for spilling-to-disk):  ./mqlrun -f -j -t ./tmp -e '[{$_internalSetWindowFields: {output: {}}}]' big_1000000.bson > output.json  This always causes  mqlrun  to crash with a  SIGABRT : [1] 78171 abort ./mqlrun -f -j -t ./tmp -e big_1000000.bson > output.json The  mqlrun  process also writes the following to the tail end of the JSON output file: { "t" :{ "$date" : "2021-07-20T17:33:45.472Z" }, "s" : "F" , "c" : "-" , "id" :23079, "ctx" : "thread1" , "msg" : "Invariant failure" , "attr" :{ "expr" : "Hit a MONGO_UNREACHABLE!" , "file" : "src/mongo/db/pipeline/process_interface/stub_mongo_process_interface.h" , "line" :271}} { "t" :{ "$date" : "2021-07-20T17:33:45.473Z" }, "s" : "F" , "c" : "-" , "id" :23080, "ctx" : "thread1" , "msg" : "\n\n***aborting after invariant() failure\n\n" } However, when I run  mqlrun  with a  $limit  stage preceding the  $_internalSetWindowFields  stage: ./mqlrun -f -j -t ./tmp -e '[{$limit: 100000}, {$_internalSetWindowFields: {output: {}}}]' big_1000000.bson > output.json  then  mqlrun  runs successfully for limits as high as 100,000. I've tried this with several large files and there's always some point after which  mqlrun  consistently fails with this same error. I've been unable to reproduce this with  $sort  or  $group . I imported the contents of this file into MongoDB using  mongorestore  and am able to run a  $setWindowFields  aggregations without any errors.
    • QO 2021-09-20

      Running mqlrun (version 5.0.0) with $setWindowFields or $_internalSetWindowFields consistently triggers the following invariant on input data sets over a certain size (but never fails on smaller input sizes):

      {"t":{"$date":"2021-07-20T17:33:45.472Z"},"s":"F", "c":"-", "id":23079, "ctx":"thread1","msg":"Invariant failure","attr":{"expr":"Hit a MONGO_UNREACHABLE!","file":"src/mongo/db/pipeline/process_interface/stub_mongo_process_interface.h","line":271}} {"t":{"$date":"2021-07-20T17:33:45.473Z"},"s":"F", "c":"-", "id":23080, "ctx":"thread1","msg":"\n\n***aborting after invariant() failure\n\n"}

            Assignee:
            ted.tuckman@mongodb.com Ted Tuckman
            Reporter:
            thomas.zembowicz@mongodb.com Thomas Zembowicz
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: