Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7361

segfault in mongod after failed moveChunk

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.2.2
    • Affects Version/s: 2.2.0
    • Component/s: None
    • Environment:
      Ubuntu 11.10, Amazon EC2 with EBS RAID 10 db volume
    • ALL

      I woke up this morning to see one of my mongod shards had a segfault over night.

      Mon Oct 15 13:25:10 [conn46] waiting till out of critical section
      Mon Oct 15 13:25:10 [conn235] waiting till out of critical section
      Mon Oct 15 13:25:10 [conn46] waiting till out of critical section
      Mon Oct 15 13:25:10 [conn235] waiting till out of critical section
      Mon Oct 15 13:25:10 [conn234] waiting till out of critical section
      Mon Oct 15 13:25:10 [conn42] waiting till out of critical section
      Mon Oct 15 13:25:10 [conn234] waiting till out of critical section
      Mon Oct 15 13:25:16 [conn235] waiting till out of critical section
      Mon Oct 15 13:25:16 [conn47] moveChunk migrate commit not accepted by TO-shard: { active: true, ns: "pb3.hourly_stats", from: "mongo3.foobar.com:27018", min: { _id: "123.123.123.123-2012-10-02_02:52:40.121883_123.123.123.123.log.gz-a1034eed7679ecb8c8add69d8167afe6" }, max: { _id: "123.123.123.123-2012-10-02_02:57:40.457075_184.169.161.72.log.gz-5b0d8915c0f603a15102ab19f6fc01bc" }, shardKeyPattern: { _id: 1 }, state: "fail", errmsg: "", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, errmsg: "", ok: 0.0 } resetting shard version to: 4699|65||5069e0edf896a39bd4a02ee3
      Mon Oct 15 13:25:16 [conn16] serverStatus was very slow: { after basic: 0, middle of mem: 0, after mem: 0, after connections: 0, after extra info: 0, after counters: 0, after repl: 0, after asserts: 0, after dur: 4690, at end: 4690 }
      Mon Oct 15 13:25:16 Invalid access at address: 0 from thread: conn237
      
      Mon Oct 15 13:25:16 Got signal: 11 (Segmentation fault).
      
      
      Mon Oct 15 13:25:16 Backtrace:
      0xade6e1 0x5582d9 0x558862 0x7f659a678060 0x842976 0xa422ae 0xa42789 0x6571f1 0x658d5d 0x659c1e 0x6afa08 0x6b3241 0xb5ba7d 0xb5d052 0x56fa52 0x5dbf11 0x7f659a66fefc 0x7f6599a0159d
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
       /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
       /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x558862]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7f659a678060]
       /usr/bin/mongod(_ZN5mongo16BSONArrayBuilder6appendINS_7BSONObjEEERS0_RKT_+0x156) [0x842976]
       /usr/bin/mongod(_ZN5mongo17MigrateFromStatus4xferEPSt4listINS_7BSONObjESaIS2_EERNS_14BSONObjBuilderEPKcRxb+0x52e) [0xa422ae]
       /usr/bin/mongod(_ZN5mongo19TransferModsCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xb9) [0xa42789]
       /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6571f1]
       /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcfd) [0x658d5d]
       /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ae) [0x659c1e]
       /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x6afa08]
       /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbc1) [0x6b3241]
       /usr/bin/mongod() [0xb5ba7d]
       /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0xb5d052]
       /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fa52]
       /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0x5dbf11]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f659a66fefc]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6599a0159d]
      
      
      
      ubuntu@mongo3:~$ mongod --version
      db version v2.2.0, pdfile version 4.5
      Mon Oct 15 17:40:29 git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
      

            Assignee:
            greg_10gen Greg Studer
            Reporter:
            zacwitte Zac Witte
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: