Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9904

Secondaries crash when too many indexes

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.4.4
    • Component/s: None
    • None
    • ALL

      I upgraded the primary and one secondary on my four node + two arbiter cluster (one node nonvoting) yesterday, from 2.2.3 to 2.4.4

      This morning all of my secondaries crashed with this error:

      Tue Jun 11 14:26:20 [repl writer worker 15] ERROR: add index fails, too
      many indexes for
      appdata49.app_b054cda3-7bd2-4f42-8bd9-31ca03987684:DisneyRides key:{
      toyStory: 1 }
      writer worker caught exception: add index fails, too many indexes for
      appdata49.app_b054cda3-7bd2-4f42-8bd9-31ca03987684:DisneyRides key:{
      toyStory: 1 } on: { ts: Timestamp 1370960780000|22, h:
      2282539247784756123, v: 2, op: "i", ns: "appdata49.system.indexes", o: {
      name: "toyStory_1", ns:
      "appdata49.app_b054cda3-7bd2-4f42-8bd9-31ca03987684:DisneyRides", key: {
      toyStory: 1 }, background: true } }
      Tue Jun 11 14:26:20 [repl writer worker 15]   Fatal Assertion 16360
      0xb07561 0xacc8b3 0x9abaf6 0xadab5d 0xb4d3d9 0x7fa1bc2cee9a 0x7fa1bb5e1cbd
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xacc8b3]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjE
      SaIS2_EEPNS0_8SyncTailE+0x156) [0x9abaf6]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
       /usr/bin/mongod() [0xb4d3d9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fa1bc2cee9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fa1bb5e1cbd]
      Tue Jun 11 14:26:20 [repl writer worker 15]
      
      ***aborting after fassert() failure
      
      
      Tue Jun 11 14:26:20 Got signal: 6 (Aborted).
      
      Tue Jun 11 14:26:20 Backtrace:
      0xb07561 0x5598c9 0x7fa1bb5244a0 0x7fa1bb524425 0x7fa1bb527b8b 0xacc8ee
      0x9abaf6 0xadab5d 0xb4d3d9 0x7fa1bc2cee9a 0x7fa1bb5e1cbd
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
       /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
       /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fa1bb5244a0]
       /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fa1bb524425]
       /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7fa1bb527b8b]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xacc8ee]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjE
      SaIS2_EEPNS0_8SyncTailE+0x156) [0x9abaf6]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
       /usr/bin/mongod() [0xb4d3d9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fa1bc2cee9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fa1bb5e1cbd]
      

      We've seen the "add index fails, too many indexes" error before many times on our 2.2 primaries, and it's never caused a problem. Is this a bug that was introduced in 2.4?

      We dynamically generate indexes, so this will make it effectively impossible for us to upgrade to 2.4 until it's fixed.

            Assignee:
            scotthernandez Scott Hernandez (Inactive)
            Reporter:
            charity@parse.com charity majors
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: