Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22049

mongoimport with upsert is very slow on 3.x with writeConcern majority

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.8, 3.2.0
    • Component/s: Tools
    • None
    • ALL
    • Hide

      All following tests are done with the primer dataset as mentioned in our documentation.


      In 2.6.9, it works as expected.

      2.6.9 mongoimport

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop
      connected to: 127.0.0.1
      2016-01-04T11:23:04.064-0500 dropping: test.restaurants
      2016-01-04T11:23:06.012-0500 check 9 25359
      2016-01-04T11:23:06.013-0500 imported 25359 objects
      

      2.6.9 mongoimport with upsert

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert
      connected to: 127.0.0.1
      2016-01-04T11:23:13.559-0500 dropping: test.restaurants
      2016-01-04T11:23:15.452-0500 check 9 25359
      2016-01-04T11:23:15.455-0500 imported 25359 objects
      


      However, in 3.x the upserted import takes a long time to run. Tested on multiple versions including 3.0.8 and 3.2.0.

      3.0.8 mongoimport

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop
      2016-01-04T11:19:22.904-0500	connected to: localhost
      2016-01-04T11:19:22.904-0500	dropping: test.restaurants
      2016-01-04T11:19:24.406-0500	imported 25359 documents
      

      3.0.8 mongoimport with upset

      mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert
      2016-01-04T11:19:29.025-0500	connected to: localhost
      2016-01-04T11:19:29.025-0500	dropping: test.restaurants
      2016-01-04T11:19:32.022-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:35.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:38.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:41.027-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:44.025-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:47.026-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:50.024-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:53.026-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:56.023-0500	[###########.............] test.restaurants	5.3 MB/11.3 MB (46.7%)
      2016-01-04T11:19:59.022-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:02.025-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:05.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:08.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:11.025-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:14.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:17.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:20.023-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:23.026-0500	[####################....] test.restaurants	9.9 MB/11.3 MB (87.1%)
      2016-01-04T11:20:26.022-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:29.024-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:32.027-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:35.022-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:38.026-0500	[########################] test.restaurants	11.3 MB/11.3 MB (100.0%)
      2016-01-04T11:20:39.418-0500	imported 25359 documents
      
      Show
      All following tests are done with the primer dataset as mentioned in our documentation . In 2.6.9, it works as expected. 2.6.9 mongoimport mongoimport --db test --collection restaurants --file primer-dataset.json --drop connected to: 127.0.0.1 2016-01-04T11:23:04.064-0500 dropping: test.restaurants 2016-01-04T11:23:06.012-0500 check 9 25359 2016-01-04T11:23:06.013-0500 imported 25359 objects 2.6.9 mongoimport with upsert mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert connected to: 127.0.0.1 2016-01-04T11:23:13.559-0500 dropping: test.restaurants 2016-01-04T11:23:15.452-0500 check 9 25359 2016-01-04T11:23:15.455-0500 imported 25359 objects However, in 3.x the upserted import takes a long time to run. Tested on multiple versions including 3.0.8 and 3.2.0. 3.0.8 mongoimport mongoimport --db test --collection restaurants --file primer-dataset.json --drop 2016-01-04T11:19:22.904-0500 connected to: localhost 2016-01-04T11:19:22.904-0500 dropping: test.restaurants 2016-01-04T11:19:24.406-0500 imported 25359 documents 3.0.8 mongoimport with upset mongoimport --db test --collection restaurants --file primer-dataset.json --drop --upsert 2016-01-04T11:19:29.025-0500 connected to: localhost 2016-01-04T11:19:29.025-0500 dropping: test.restaurants 2016-01-04T11:19:32.022-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:35.023-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:38.023-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:41.027-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:44.025-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:47.026-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:50.024-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:53.026-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:56.023-0500 [###########.............] test.restaurants 5.3 MB/11.3 MB (46.7%) 2016-01-04T11:19:59.022-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:02.025-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:05.023-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:08.026-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:11.025-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:14.023-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:17.026-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:20.023-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:23.026-0500 [####################....] test.restaurants 9.9 MB/11.3 MB (87.1%) 2016-01-04T11:20:26.022-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:29.024-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:32.027-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:35.022-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:38.026-0500 [########################] test.restaurants 11.3 MB/11.3 MB (100.0%) 2016-01-04T11:20:39.418-0500 imported 25359 documents

      mongoimport with the --upsert flag is much slower than expected on 3.x. When running mongoimport with the --upsert flag on an empty collection, performance should be about equal to mongoimport without the flag. This was the case in 2.6.9 but is no longer true in 3.x.

            Assignee:
            mikeo@mongodb.com Michael O'Brien
            Reporter:
            daniel.hatcher@mongodb.com Danny Hatcher (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: