Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19128

Fatal assertion during secondary index build

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.9, 3.1.9
    • Affects Version/s: 3.1.5
    • Component/s: Index Maintenance
    • None
    • Fully Compatible
    • ALL
    • QuInt A (10/12/15)
    • 0

      During an evergreen run, the FSM replication+sharding suite failed with the following fatal assertion:

      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.558 0000  m31101| 2015-06-25T05:37:56.558 0000 I INDEX    [repl index builder 283] build index on: db66.reindex_background_0 properties: { v: 1, key: { integer: 1.0 }, name: "integer_1", ns: "db66.reindex_background_0", background: true }
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.577 0000  m31102| 2015-06-25T05:37:56.576 0000 E INDEX    [repl index builder 278] IndexBuilder could not build index: Location28550 Unable to complete index build as the collection is no longer readable
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.577 0000  m31102| 2015-06-25T05:37:56.576 0000 I -        [repl index builder 278] Fatal Assertion 28555
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.602 0000  m31102| 2015-06-25T05:37:56.601 0000 I CONTROL  [repl index builder 278]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.602 0000  m31102|  0x1060af6 0x1018068 0x100571d 0xb5d2f6 0x100a09a 0x7fcf4d05fbf0 0x7fcf4c87c182 0x7fcf4c5a947d
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.602 0000  m31102| ----- BEGIN BACKTRACE -----
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102| {"backtrace":[{"b":"400000","o":"C60AF6"},{"b":"400000","o":"C18068"},{"b":"400000","o":"C0571D"},{"b":"400000","o":"75D2F6"},{"b":"400000","o":"C0A09A"},{"b":"7FCF4CFAE000","o":"B1BF0"},{"b":"7FCF4C874000","o":"8182"},{"b":"7FCF4C4AF000","o":"FA47D"}],"processInfo":{ "mongodbVersion" : "3.1.5-pre-", "gitVersion" : "2e7eb5ed6f33fe179f6a58d2ab8d5b12e11030aa", "compiledModules" : [ "rocksdb" ], "uname" : { "sysname" : "Linux", "release" : "3.13.0-24-generic", "version" : "#46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "0B8F75910268F6A1873E9D9D538A66C0C5566E91" }, { "b" : "7FFF5CAC7000", "elfType" : 3, "buildId" : "6755FAD2CADACDF1667E5B57FF1EDFC28DD1C976" }, { "b" : "7FCF4D8CE000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "7EE9BC42787A0F9A793BDBFCB0671FD12C52BC2A" }, { "b" : "7FCF4D6BE000", "path" : "/lib/x86_64-linux-gnu/libbz2.so.1.0", "elfType" : 3, "buildId" : "E1031DDBFFE20367E874B7093EEC0C8D9F3B43F6" }, { "b" : "7FCF4D4B6000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "92FCF41EFE012D6186E31A59AD05BDBB487769AB" }, { "b" : "7FCF4D2B2000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "C1AE4CB7195D337A77A3C689051DABAA3980CA0C" }, { "b" : "7FCF4CFAE000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc  .so.6", "elfType" : 3, "buildId" : "19EFDDAB11B3BF5C71570078C59F91CF6592CE9E" }, { "b" : "7FCF4CCA8000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1D76B71E905CB867B27CEF230FCB20F01A3178F5" }, { "b" : "7FCF4CA92000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "8D0AA71411580EE6C08809695C3984769F25725B" }, { "b" : "7FCF4C874000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9318E8AF0BFBE444731BB0461202EF57F7C39542" }, { "b" : "7FCF4C4AF000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "30C94DC66A1FE95180C3D68D2B89E576D5AE213C" }, { "b" : "7FCF4DAE7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9F00581AB3C73E3AEA35995A0C50D24D59A01D47" } ] }}
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  mongod(mongo::printStackTrace(std::ostream&) 0x26) [0x1060af6]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  mongod(mongo::logContext(char const*) 0x138) [0x1018068]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  mongod(mongo::fassertFailed(int) 0x4D) [0x100571d]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  mongod(mongo::IndexBuilder::run() 0x616) [0xb5d2f6]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  mongod(mongo::BackgroundJob::jobBody() 0x14A) [0x100a09a]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  libstdc  .so.6( 0xB1BF0) [0x7fcf4d05fbf0]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  libpthread.so.0( 0x8182) [0x7fcf4c87c182]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.604 0000  m31102|  libc.so.6(clone 0x6D) [0x7fcf4c5a947d]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.605 0000  m31102| -----  END BACKTRACE  -----
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.605 0000  m31102| 2015-06-25T05:37:56.601 0000 I -        [repl index builder 278]
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.605 0000  m31102|
      [js_test:fsm_all_sharded_replication] 2015-06-25T05:37:56.605 0000  m31102| ***aborting after fassert() failure
      

      The failing run is here and this is the actual run log.

        1. fassert_bk_index
          3 kB
          Michael Kania

            Assignee:
            mathias@mongodb.com Mathias Stearn
            Reporter:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Votes:
            1 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: