Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-14056

Moving large collection across databases with renameCollection triggers fatal assertion

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.6.2, 2.7.2
    • Affects Version/s: 2.6.1, 2.7.1
    • Component/s: Storage
    • None
    • ALL

      Issue Status as of Jun 6, 2014

      ISSUE SUMMARY
      When moving a collection across databases, the renameCollection command does not issue a journal commit until after the data copy has finished. As a result, moving a collection of size ~0.5GB or larger across databases will trigger a fatal assertion when the journal commit buffer exceeds its maximum size.

      USER IMPACT
      Users running with journaling enabled can't move large collections across databases.

      WORKAROUNDS
      If the destination database is empty, a possible workaround is a follows:

      • use the copydb command to copy all collections to the target database
      • use the drop command to remove unneeded collections
      • use the renameCollection command to give the collection the desired name in the target database.

      If the destination database is not empty there are no workarounds.

      AFFECTED VERSIONS
      MongoDB 2.6.0 and 2.6.1 are affected by this issue.

      FIX VERSION
      The fix is included in the 2.6.2 production release.

      RESOLUTION DETAILS
      The renameCollection command issues a journal commit after each data insertion in the target collection.

      Original description

      When moving a collection across databases, the renameCollection command does not issue a journal commit until after the data copy has finished. As a result, moving a collection of size ~0.5GB or larger across databases will trigger a fatal assertion when the journal commit buffer exceeds its maximum size.

      Regression since 2.4 (introduced in 0abf27ae); confirmed against 2.6.1 and 2.7.1.

      Reproduce with the following:

      var s = '';
      for (var i=0; i<1024; i++) {
        s += 'x';
      }
      for (var i=0; i<1000*1000; i++) {
        db.foo.insert({a: s});
      }
      db.adminCommand({renameCollection: "test.foo", to: "test2.foo"})
      

      The above generates the following output in the mongod log:

      2014-05-27T12:13:25.641-0400 [conn2] test2.foo Assertion failure a <= 512*1024*1024 src/mongo/util/alignedbuilder.cpp 104
      2014-05-27T12:13:25.656-0400 [conn2] test2.foo 0x1006ac7ab 0x1006642c2 0x100654d52 0x1006533f9 0x1001e1dd0 0x1001e2081 0x1001d84f0 0x1001d65a1 0x1001d8159 0x1001d6628 0x1000fb54b 0x1000fc26d 0x1000f6bd5 0x10016eea1 0x1001b7605 0x1001b8375 0x1001b907c 0x1003cfc7f 0x1002a1050 0x100006854
       0   mongod261                           0x00000001006ac7ab _ZN5mongo15printStackTraceERSo + 43
       1   mongod261                           0x00000001006642c2 _ZN5mongo10logContextEPKc + 114
       2   mongod261                           0x0000000100654d52 _ZN5mongo12verifyFailedEPKcS1_j + 274
       3   mongod261                           0x00000001006533f9 _ZN5mongo14AlignedBuilder14growReallocateEj + 191
       4   mongod261                           0x00000001001e1dd0 _ZN5mongo3durL21prepBasicWrite_inlockERNS_14AlignedBuilderEPKNS0_11WriteIntentERNS_12RelativePathE + 880
       5   mongod261                           0x00000001001e2081 _ZN5mongo3dur13PREPLOGBUFFERERNS0_11JSectHeaderERNS_14AlignedBuilderE + 401
       6   mongod261                           0x00000001001d84f0 _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 272
       7   mongod261                           0x00000001001d65a1 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
       8   mongod261                           0x00000001001d8159 _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 113
       9   mongod261                           0x00000001001d6628 _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 76
       10  mongod261                           0x00000001000fb54b _ZN5mongo18addExistingToIndexEPNS_10CollectionEPKNS_15IndexDescriptorEPNS_17IndexAccessMethodEb + 1275
       11  mongod261                           0x00000001000fc26d _ZN5mongo12buildAnIndexEPNS_10CollectionEPNS_17IndexCatalogEntryEb + 2253
       12  mongod261                           0x00000001000f6bd5 _ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE + 1231
       13  mongod261                           0x000000010016eea1 _ZN5mongo19CmdRenameCollection3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 7681
       14  mongod261                           0x00000001001b7605 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
       15  mongod261                           0x00000001001b8375 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2873
       16  mongod261                           0x00000001001b907c _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1388
       17  mongod261                           0x00000001003cfc7f _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 1615
       18  mongod261                           0x00000001002a1050 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1968
       19  mongod261                           0x0000000100006854 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 308
      2014-05-27T12:13:25.694-0400 [conn2] dbexception in groupCommit causing immediate shutdown: 0 assertion src/mongo/util/alignedbuilder.cpp:104
      2014-05-27T12:13:25.746-0400 [conn2] SEVERE: gc1
      2014-05-27T12:13:25.756-0400 [conn2] SEVERE: Got signal: 6 (Abort trap: 6).
      Backtrace:0x1006ac7ab 0x1006ac50f 0x101cfb92a 0x1d12866 0x101d52dfa 0x100298b98 0x1001d873b 0x1001d65a1 0x1001d8159 0x1001d6628 0x1000fb54b 0x1000fc26d 0x1000f6bd5 0x10016eea1 0x1001b7605 0x1001b8375 0x1001b907c 0x1003cfc7f 0x1002a1050 0x100006854
       0   mongod261                           0x00000001006ac7ab _ZN5mongo15printStackTraceERSo + 43
       1   mongod261                           0x00000001006ac50f _ZN5mongo12_GLOBAL__N_110abruptQuitEi + 191
       2   libsystem_c.dylib                   0x0000000101cfb92a _sigtramp + 26
       3   ???                                 0x0000000001d12866 0x0 + 30484582
       4   libsystem_c.dylib                   0x0000000101d52dfa abort + 143
       5   mongod261                           0x0000000100298b98 _ZN5mongo10mongoAbortEPKc + 104
       6   mongod261                           0x00000001001d873b _ZN5mongo3durL11groupCommitEPNS_4Lock11GlobalWriteE + 859
       7   mongod261                           0x00000001001d65a1 _ZN5mongo3dur11DurableImpl9commitNowEv + 21
       8   mongod261                           0x00000001001d8159 _ZN5mongo3dur11DurableImpl16_aCommitIsNeededEv + 113
       9   mongod261                           0x00000001001d6628 _ZN5mongo3dur11DurableImpl14commitIfNeededEb + 76
       10  mongod261                           0x00000001000fb54b _ZN5mongo18addExistingToIndexEPNS_10CollectionEPKNS_15IndexDescriptorEPNS_17IndexAccessMethodEb + 1275
       11  mongod261                           0x00000001000fc26d _ZN5mongo12buildAnIndexEPNS_10CollectionEPNS_17IndexCatalogEntryEb + 2253
       12  mongod261                           0x00000001000f6bd5 _ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE + 1231
       13  mongod261                           0x000000010016eea1 _ZN5mongo19CmdRenameCollection3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 7681
       14  mongod261                           0x00000001001b7605 _ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb + 37
       15  mongod261                           0x00000001001b8375 _ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb + 2873
       16  mongod261                           0x00000001001b907c _ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi + 1388
       17  mongod261                           0x00000001003cfc7f _ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_ + 1615
       18  mongod261                           0x00000001002a1050 _ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE + 1968
       19  mongod261                           0x0000000100006854 _ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE + 308
      

      Original description:

      Hi,

      We have a large data load that loads data from other systems in to MongoDB. At the end of the load it switched the data live by renaming the collection.

      db.runCommand({renameCollection:"contacts_load.contacts_new", to:"contacts.contacts",dropTarget:true})
      

      In version 2.4.5 this has been working without a problem.

      However, in version 2.6.1 this crashes the MongoD.

      Please help.

      Cheers

      Simon

            Assignee:
            rassi J Rassi
            Reporter:
            Simon C Simon Cater [X]
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: