with mongodb connector, spark stuck at the last task.

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 2.3.2
    • Component/s: Reads
    • None
    • Environment:
      linux, mongodb3.x, spark2.3.1, scala2.11.11
    • None
    • None
    • None
    • None
    • None
    • None

      //
      val rdd = Global.sparkContext.loadFromMongoDB(...).withPipeline(...)
      
      rdd.count()

      it always stuck at the last task. it may take 30 minutes to finish this last task, or maybe hange foreaver. No exception or error is found.

       

      If it just reads few records, for example, 2000 records, it could finish the last task quickly. If it reads above 100000 records, it will hange there.

      I have set 

      System.setProperty("spark.mongodb.keep_alive_ms", "1000000"), but it does not help.

       

            Assignee:
            Ross Lawley
            Reporter:
            Feng Zhang
            None
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: