Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-395

ReadConfig.AllowDiskUse should also be taken into account during sort

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 10.1.1
    • Affects Version/s: 10.0.3
    • Component/s: Reads
    • None
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Hello,

      We noticed that during the sort of chunks used in ShardedPartitioner, we had issues such as

      Executor error during find command :: caused by :: Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting
      
      

      We saw an issue was raised in SPARK-355 that fixed some aggregation queries used in other partitioner.
      Starting from Mongo 4.4, it's possible to use the allowDiskUse on cursor, so we could change that piece of code from ShardedPartitioner to use the allowDiskUse option from read config

                      client
                          .getDatabase(CONFIG_DATABASE)
                          .getCollection(CONFIG_CHUNKS, BsonDocument.class)
                          .find(chunksMatchPredicate)
                          .projection(CHUNKS_PROJECTIONS)
                          .sort(SORTS)
                          .into(new ArrayList<>()));
      
      

       

      Thanks for your work!

      See also https://www.mongodb.com/community/forums/t/spark-structure-streaming-with-mongodb-change-stream-allowdiskuse-doesnt-work/175727

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            mehdi.elhajami.pro@gmail.com Mehdi El Hajami
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: