Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3961

Segmentation Fault when creating new indexes

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.0.1, 2.1.0
    • Affects Version/s: 2.0.0
    • Component/s: Index Maintenance
    • Environment:
      Red Hat Enterprise Linux 5, 64 bit
    • Linux

      Using a sharded cluster consisting of 5 shards. The collection started off empty and using a batch process to load the data from MySQL into Mongo using Mongoid.

      During the run we decided to create multiple indexes to reduce CPU utilization. When creating the indexes the primary server in the first shard crashed with a Segmentation fault. After restarting the mongod we then re-ran the steps to create the indexes, but this time included the background: true option. The second time around it took about a minute and only then crashed with a Segmentation fault. The key is that these collections were queried and inserted into while the indexes were being created. Did not see this problem when running 1.8.3.

      Below are both stack traces from the log file. The first was without the background option and mongod crashed almost immediately:

      Tue Sep 27 14:23:03 Invalid access at address: 0x10

      Tue Sep 27 14:23:03 Got signal: 11 (Segmentation fault).

      Tue Sep 27 14:23:03 Backtrace:
      0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
      /lib64/libpthread.so.0 [0x3cc340eb10]
      /usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
      /usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
      /usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
      /usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
      /usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
      /usr/bin/mongod [0x87e037]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
      /lib64/libpthread.so.0 [0x3cc340673d]

      pure virtual method called
      Tue Sep 27 14:23:03 terminate() called, printing stack:
      0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
      Logstream::get called in uninitialized state
      Tue Sep 27 14:23:03 ERROR: Client::~Client _context should be null but is not; client:conn
      Logstream::get called in uninitialized state
      Tue Sep 27 14:23:03 ERROR: Client::shutdown not called: conn
      pure virtual method called
      Tue Sep 27 14:23:03 terminate() called, printing stack:
      0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
      pure virtual method called
      Tue Sep 27 14:23:03 terminate() called, printing stack:
      0xa834ed 0x3cc2cbce16 0x3cc2cbce43 0x3cc2cbd34f 0x973ad7 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d 0x3cc24d44bd
      Logstream::get called in uninitialized state
      Tue Sep 27 14:23:03 [snapshotthread] ERROR in SnapshotThread: St9bad_alloc
      Tue Sep 27 14:23:03 Got signal: 6 (Aborted).

      /usr/bin/mongod(_ZN5mongo11myterminateEv+0x4d) [0xa834ed]
      /usr/lib64/libstdc++.so.6 [0x3cc2cbce16]
      /usr/lib64/libstdc++.so.6 [0x3cc2cbce43]
      /usr/lib64/libstdc++.so.6 [0x3cc2cbd34f]
      /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x637) [0x973ad7]
      /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97512f]
      /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x95d725]
      /usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xee4) [0x9607b4]
      /usr/bin/mongod [0x87e037]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
      /lib64/libpthread.so.0 [0x3cc340673d]
      /lib64/libc.so.6(clone+0x6d) [0x3cc24d44bd]
      Tue Sep 27 14:23:03 Got signal: 6 (Aborted).

      Building indexes using background: true
      ========================

      Tue Sep 27 14:31:05 [conn63] build index done 38913 records 9.878 secs
      Tue Sep 27 14:31:05 [conn63] insert clarity_production.system.indexes 9879ms
      Tue Sep 27 14:31:05 Invalid access at address: 0

      Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

      Tue Sep 27 14:31:05 Invalid access at address: 0

      Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

      Tue Sep 27 14:31:05 Invalid access at address: 0

      Tue Sep 27 14:31:05 Got signal: 11 (Segmentation fault).

      Tue Sep 27 14:31:05 Backtrace:
      0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
      /lib64/libpthread.so.0 [0x3cc340eb10]
      /usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
      /usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
      /usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
      /usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
      /usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
      /usr/bin/mongod [0x87e037]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
      /lib64/libpthread.so.0 [0x3cc340673d]

      Logstream::get called in uninitialized state
      Tue Sep 27 14:31:05 [conn43] query clarity_production.clarity_raw_xmls ntoreturn:1 nscanned:39157 reslen:20 400ms
      Logstream::get called in uninitialized state
      Tue Sep 27 14:31:05 ERROR: Client::~Client _context should be null but is not; client:conn
      Logstream::get called in uninitialized state
      Tue Sep 27 14:31:05 ERROR: Client::shutdown not called: conn
      Tue Sep 27 14:31:05 Backtrace:
      0xa83fc9 0xa845a0 0x3cc340eb10 0x54d7dc 0x83d551 0x83d756 0x83d8b0 0x96ef38 0x8c0b66 0x8cd4b0 0x8cdb06 0x8d21ce 0x8d3be5 0x8d50b7 0x960d4a 0x87e037 0x88485c 0xa96a46 0x635dd7 0x3cc340673d
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9]
      /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0]
      /lib64/libpthread.so.0 [0x3cc340eb10]
      /usr/bin/mongod(_ZN5mongo24FieldRangeVectorIterator7advanceERKNS_7BSONObjE+0x4c) [0x54d7dc]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor29skipOutOfRangeKeysAndCheckEndEv+0x81) [0x83d551]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor12skipAndCheckEv+0x26) [0x83d756]
      /usr/bin/mongod(_ZN5mongo11BtreeCursor7advanceEv+0x100) [0x83d8b0]
      /usr/bin/mongod(_ZN5mongo11UserQueryOp4nextEv+0x2b8) [0x96ef38]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner6nextOpERNS_7QueryOpE+0x56) [0x8c0b66]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner4nextEv+0x110) [0x8cd4b0]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet6Runner22runUntilFirstCompletesEv+0x56) [0x8cdb06]
      /usr/bin/mongod(_ZN5mongo12QueryPlanSet5runOpERNS_7QueryOpE+0x11e) [0x8d21ce]
      /usr/bin/mongod(_ZN5mongo16MultiPlanScanner9runOpOnceERNS_7QueryOpE+0x525) [0x8d3be5]
      /usr/bin/mongod(_ZN5mongo16MultiPlanScanner5runOpERNS_7QueryOpE+0x17) [0x8d50b7]
      /usr/bin/mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0x147a) [0x960d4a]
      /usr/bin/mongod [0x87e037]
      /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c]
      /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46]
      /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7]
      /lib64/libpthread.so.0 [0x3cc340673d]

            Assignee:
            eliot Eliot Horowitz (Inactive)
            Reporter:
            reidmo@gmail.com Reid Morrison
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: