Add multithreaded support to mongoimport

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Done
    • Priority: Major - P3
    • 2.8.0-rc0
    • Affects Version/s: None
    • Component/s: mongoimport
    • None
    • MCI 2.7.8

      There are a couple possible approaches to take here:

      1. add a user configurable knob (pros: easy and easily tunable; cons: easy to misuse)
      2. do an adaptive solution (pros: just works; cons: tricky to get right)

      Other considerations:

      1. It will be easy to overload the server
      2. Need to change how users throttle writes
      3. If it is a requirement that the import is sequential according to the source file, parallel import is not possible

      Because mongoimport works against a single collection, it is the easiest tool to parallelize. Simply read-ahead from the source into memory and use >1 connection to the server.

            Assignee:
            Adinoyi Omuya
            Reporter:
            Ian Whalen (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: