Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3275

Fix a race condition in LSM between eviction and cursor operations

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • WT2.9.3, 3.5.9
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage 2017-04-17, Storage 2017-05-08

      Build failed in Jenkins: wiredtiger-test-format-stress-sanitizer #13458 origin/develop
      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/13458

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      auto_throttle=1
      backups=1
      bitcnt=8
      bloom=1
      bloom_bit_count=59
      bloom_hash_count=26
      bloom_oldest=0
      cache=30
      checkpoints=1
      checksum=uncompressed
      chunk_size=1
      compaction=0
      compression=snappy
      data_extend=0
      data_source=lsm
      delete_pct=6
      dictionary=0
      direct_io=0
      encryption=rotn-7
      evict_max=5
      file_type=row-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      in_memory=0
      insert_pct=90
      internal_key_truncation=1
      internal_page_max=14
      isolation=read-uncommitted
      key_gap=6
      key_max=89
      key_min=10
      leaf_page_max=11
      leak_memory=0
      logging=0
      logging_archive=1
      logging_compression=none
      logging_prealloc=0
      long_running_txn=0
      lsm_worker_threads=4
      merge_max=19
      mmap=1
      ops=100000
      prefix_compression=1
      prefix_compression_min=6
      quiet=1
      read_pct=0
      rebalance=1
      repeat_data_pct=81
      reverse=0
      rows=100000
      runs=1
      salvage=1
      split_pct=85
      statistics=0
      statistics_server=0
      threads=20
      timer=20
      transaction-frequency=43
      value_max=2451
      value_min=17
      verify=1
      wiredtiger_config=
      write_pct=4
      ############################################
      
      ==26664==ERROR: AddressSanitizer: heap-use-after-free on address 0x61b002240e8d at pc 0xd983d0 bp 0x7f1335f81f70 sp 0x7f1335f81f68
      READ of size 1 at 0x61b002240e8d thread T32
          #0 0xd983cf in __wt_lex_compare_skip /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/btree_cmp.i:167
          #1 0xd89634 in __wt_compare_skip /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/btree_cmp.i:185:11
          #2 0xd884e1 in __wt_search_insert /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/row_srch.c:102:34
          #3 0xd915a7 in __wt_row_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/row_srch.c:619
          #4 0x10e9acf in __cursor_row_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:358
          #5 0x10fc3a8 in __btcur_update /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:1064:20
          #6 0x10fa078 in __wt_btcur_reserve /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:1143
          #7 0xe4a46e in __curfile_reserve /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/cursor/cur_file.c:353
          #8 0xf880b5 in __clsm_put /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/lsm/lsm_cursor.c:1482
          #9 0xf704e5 in __clsm_reserve /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/lsm/lsm_cursor.c:1667:9
          #10 0x4a77ad in row_reserve /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:1126
          #11 0x49e5cf in ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:647
          #12 0x7f1350e18dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
          #13 0x7f134fffc73c in __clone (/lib64/libc.so.6+0xf773c)
      
      0x61b002240e8d is located 269 bytes inside of 1482-byte region [0x61b002240d80,0x61b00224134a)
      freed by thread T1 here:
          #0 0x467f49 in __interceptor_free (/mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x467f49)
          #1 0x753680 in __wt_free_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:309
          #2 0xb3d435 in __page_out_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:156
          #3 0xb3bb2f in __wt_ref_out_int /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:60
          #4 0xb3d8f6 in __wt_ref_out /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_discard.c:70
          #5 0x6472aa in __evict_page_clean_update /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:255
          #6 0x6423cf in __wt_evict /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_page.c:164
          #7 0x60ef80 in __evict_page /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:2147
          #8 0x6021bd in __evict_lru_pages /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:1151
          #9 0x625d19 in __evict_pass /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:708
          #10 0x600054 in __evict_server /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:408
          #11 0x5fec6c in __wt_evict_thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:313
          #12 0xa23fcd in __thread_run /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:31
          #13 0x7f1350e18dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
      
      previously allocated by thread T45 here:
          #0 0x4682c3 in realloc (/mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x4682c3)
          #1 0x750d0e in __realloc_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:130
          #2 0x7516e7 in __wt_realloc_noclear /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_common/os_alloc.c:171
          #3 0x9e0c84 in __wt_buf_grow_worker /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/scratch.c:48
          #4 0xb866ce in __wt_buf_grow /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/buf.i:18
          #5 0xb86a78 in __wt_buf_init /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/buf.i:52:10
          #6 0xb7e949 in __wt_buf_initsize /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/buf.i:62:32
          #7 0xb7be70 in __wt_bt_read /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_io.c:92:20
          #8 0xbd507e in __page_read /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_read.c:414
          #9 0xbd0671 in __wt_page_in_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_read.c:524
          #10 0xd99793 in __wt_page_swap_func /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/btree.i:1454
          #11 0xd8d524 in __wt_row_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/row_srch.c:445:14
          #12 0x10e9acf in __cursor_row_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:358
          #13 0x10e6d7c in __wt_btcur_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/btree/bt_cursor.c:457:47
          #14 0xe40a34 in __curfile_search /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/cursor/cur_file.c:187
          #15 0xf85214 in __clsm_lookup /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/lsm/lsm_cursor.c:1225
          #16 0xf70386 in __clsm_reserve /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/lsm/lsm_cursor.c:1666
          #17 0x4a77ad in row_reserve /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:1126
          #18 0x49e5cf in ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:647
          #19 0x7f1350e18dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
      
      Thread T32 created by T0 here:
          #0 0x457582 in __interceptor_pthread_create (/mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x457582)
          #1 0x497607 in wts_ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:120
          #2 0x4b973d in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:209
          #3 0x7f134ff26b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
      
      Thread T1 created by T0 here:
          #0 0x457582 in __interceptor_pthread_create (/mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x457582)
          #1 0x79ac1a in __wt_thread_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/os_posix/os_thread.c:29
          #2 0xa1c5fc in __thread_group_resize /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:214
          #3 0xa1e059 in __wt_thread_group_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/support/thread_group.c:291
          #4 0x605523 in __wt_evict_create /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/evict/evict_lru.c:494
          #5 0x587354 in __wt_connection_workers /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_open.c:261
          #6 0x4f577f in wiredtiger_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/conn/conn_api.c:2457
          #7 0x4cab58 in wts_open /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/wts.c:254
          #8 0x4b95c8 in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:190
          #9 0x7f134ff26b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
      
      Thread T45 created by T0 here:
          #0 0x457582 in __interceptor_pthread_create (/mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/t+0x457582)
          #1 0x497607 in wts_ops /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/ops.c:120
          #2 0x4b973d in main /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/test/format/../../../test/format/t.c:209
          #3 0x7f134ff26b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
      
      SUMMARY: AddressSanitizer: heap-use-after-free /mnt/data0/jenkins/workspace/wiredtiger-test-format-stress-sanitizer/build_posix/../src/include/btree_cmp.i:167 __wt_lex_compare_skip
      Shadow bytes around the buggy address:
        0x0c3680440180: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa
        0x0c3680440190: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c36804401a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c36804401b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c36804401c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      =>0x0c36804401d0: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c36804401e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c36804401f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3680440200: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3680440210: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3680440220: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07
        Heap left redzone:     fa
        Heap right redzone:    fb
        Freed heap region:     fd
        Stack left redzone:    f1
        Stack mid redzone:     f2
        Stack right redzone:   f3
        Stack partial redzone: f4
        Stack after return:    f5
        Stack use after scope: f8
        Global redzone:        f9
        Global init order:     f6
        Poisoned by user:      f7
        ASan internal:         fe
      ==26664==ABORTING
      

            Assignee:
            alexander.gorrod@mongodb.com Alexander Gorrod
            Reporter:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: